ASP.NET Core MVC，CPUs Become Higher Slowly

[goldeneyezhang]

We deploy Asp.net Core MVC on Linux,CPUs become higher slowly
From Zabbix,we can see

.NET Core SDK (reflecting any global.json):
Version: 2.1.402
Commit: 3599f217f4

Runtime Environment:
OS Name: centos
OS Version: 7
OS Platform: Linux
RID: centos.7-x64
Base Path: /usr/share/dotnet/sdk/2.1.402/

Host (useful for support):
Version: 2.1.4
Commit: 85255dde3e

.NET Core SDKs installed:
2.1.402 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

We use perf to collect info ,but have no idea.
sampleTrace.trace.zip
Any ideas on a possible cause?

[Tratcher]

@sebastienros

[sebastienros]

Any idea?

My first thought is that you have a memory leak, and the GC has to work more, or page to disk.
How does the memory profile look like compared to the CPU usage?

Also maybe try to get two traces, one when it's low CPU, and one when CPU is much higher. We can then to a diff and see what changed.

[goldeneyezhang]

Hi,@sebastienros, memory is always stable.

[sebastienros]

Are you using HttpClient in your code? If so are you reusing the instances or creating/disposing them every time?

If you can please share the double traces I suggested you to take.

[goldeneyezhang]

Hi,@sebastienros
CPU high -- sampleTraceHight.trade.zip
CPU low -- sampleTraceLow.trace.zip
sampleTraceHigh.trace.zip

sampleTraceLow.trace.zip

We use HttpClient,
private static readonly HttpClient HttpClient;
private readonly IHttpClientFactory _httpClientFactory;

[marfenij]

@goldeneyezhang what transport do you use Sockets or Libuv ?

[gzpbx]

@marfenij Sockets

[marfenij]

@gzpbx thanks, so we stay with libuv

[goldeneyezhang]

I see a lot of threads is blocked ,use pstack,show epoll_wait

pstack 80514
Thread 1 (process 80514):
#0 0x00007f65475a1183 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f653e7278e6 in SystemNative_WaitForSocketEvents () from /usr/share/dotnet/s hared/Microsoft.NETCore.App/2.1.4/System.Native.so
#2 0x00007f64d233bfd8 in ?? ()
#3 0x000000007e2b72a0 in ?? ()
#4 0x00007f6546ed4a28 in vtable for InlinedCallFrame () from /usr/share/dotnet/shared /Microsoft.NETCore.App/2.1.4/libcoreclr.so
#5 0x00007f5fef7fdc70 in ?? ()
#6 0x00007f64d28e6910 in ?? ()
#7 0x00007f64d28e6910 in ?? ()
#8 0x00007f5fef7fd810 in ?? ()
#9 0x00007f64d233bfd8 in ?? ()
#10 0x00007f5fef7fd880 in ?? ()
#11 0x00007f64d28e6910 in ?? ()
#12 0x00007f604036ddc8 in ?? ()
#13 0x0000000000006bdc in ?? ()
#14 0x0000000000000000 in ?? ()
@marfenij

[davidfowl]

Your high CPU trace shows the swapper taking up most of the time spent (if I'm reading this correctly)

cc @vancem

[vancem]

@davidfowl the swapper is actually just the Idle loop. If you look at the functions where the CPU is being spent in swapper you will see

|  ||||+ kernel.kallsyms!arch_cpu_idle
|  |||| + kernel.kallsyms!__cpuidle_text_start
|  |||| |+ kernel.kallsyms!native_safe_halt

Which I am assuming is where the OS lets CPUS run when it has nothing better.

If you filter to just what is going on in the dotnet process, you see that 3/4 of the time is spend in System.private.Corelib (the main framework DLL), but we don't have symbols for methods inside.

Including @brianrob who can help with understanding why that did now work. However we should make sure you are following the instructions https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md. In particular you have to set

export COMPlus_PerfMapEnabled=1
export COMPlus_EnableEventLog=1

so that these environment variables get inherited into the dotnet process. If you DIDN'T do this it will have the symptoms that you describe. Case of the variables matters (as always on Linux).

Finally there is a potential work-around using a very new (not even documented yet), feature of .NET 2.1 called EventPipe.

You use it by setting some the environment variable.

export COMPlus_EnableEventPipe=1

Then start the process, then let it run (it is OK that it runs a long time). Then you have to shut the process down cleanly (if you are and ASP.NET app, they have a Ctrl-C hander that does this. You shoud have a *.netperf file generated.

[davidfowl]

Ah good to know 😄

[gzpbx]

@vancem we know why we can't get the symbols. Use the export ComPlus_XXXX=1 and then restart supervisor that can't get the symbols , we must use the same shell to execute export and dotnet run.
We are on holiday when we go back to work we will provide the .netperf file with symbols.

For now , we have to restart the program every day... yes on production ...