Performance stuttering occurs in .NET Core 2.2 + 3.0, some background thread exiting by the runtime and any GC aggravates the problem

[WChrisK]

I'm using C# for a rendering project. On .NET Core 3.0, every 10-15 seconds I'd start getting this small stutter while moving the camera around. It is a big enough of a negative that it significantly destroys the user experience of the application. However when I change to .NET Framework 4.8 or .NET Core 2.1 the performance issues (this small microstutter that that lasts for maybe 50-150ms) completely disappears.

Long story short after a lot of digging, I found out the following:

• Any GC at all causes this 'microstutter', even gen0, and even for very small amounts of garbage

• Lots of GC at gen0/gen1 is perfectly fine on Framework 4.8/Core 2.1

• In Core 2.2 there's a random pause when some thread exits and it prints to the output console in VS2019. This occurs less in Core 2.2 since there's only one one thread that exits, but in Core 3.0 this happens a lot more frequently (every 10-15 seconds or so) and causes a lot more problems

There is some 'background thread' that appears to be exiting which is generated beyond my control (I assume this is the runtime according to the debugging it in the threads window). Whatever is generating this thread, I notice in the VS2019 console when it says "... thread has exited" I get a stutter. I am not doing any threading, the application is purely single threaded (not using TPL or PLINQ if that helps, no async or await either). Regardless, even in a simple 'hello world' console app I still notice the message appearing in the console.

I'm not sure why the presence of any kind of GC invocation causes pauses, or what this background thread that keeps getting spawned is (especially since it happens in a program that just prints hello world and then goes into a while(true) { } to prevent exiting). After some discussion online, other people seem to think it might be tiered compilation threads since it starts out causing microstutters every 10-15 seconds and then the gap gets wider and wider as time goes on, but I'm not sure what I could do to definitively prove or disprove this.

So what do I do? I could git bisect this repo until I find out the commits that make it worse. Is there anything else I could do to isolate what is causing it? Or am I writing an issue on the wrong repo (if so, where do I go)?

[jespersh]

Just two links that might help you since it seems you're allocating more and more objects that GC has to figure out if it can GC:

• https://github.com/Microsoft/perfview
• https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/performance

[WChrisK]

@jespersh Much appreciated, though I've already been using perfview to attempt to diagnose the problem and I haven't been able to find much to help me in finding out what is wrong. I also get it happening in a completely-zero garbage generation environment (the GC doesn't run at all and it still happens), I just happened to notice that the GC running seems to also cause issues when it does run. I'm at a loss for where to go from here.

[jkotas]

other people seem to think it might be tiered compilation threads

You can set COMPlus_TieredCompilation=0 in your environment to quickly confirm or deny this hypothesis. This setting disables tiered compilation.

[Suchiman]

Perhaps related to #12882?

[PathogenDavid]

I notice in the VS2019 console when it says "... thread has exited" I get a stutter.

Maybe worth checking: Does the microstutter happen if you turn off the thread exit messages (right-click the output window) or run without a debugger attached?

[WChrisK]

@jkotas As of the current .NET Core 3.0 preview, it appears that disabling tiered compilation works. Another driver bug masked it (meaning turning it off with the driver bug appeared to make no difference), but when I worked around the driver bug and then turned off tiered compilations the microstutter went away.

I don't know if this is what is supposed to happen however. Should the JIT be spawning a thread and/or doing its business every 5-10 seconds? I tested this by running back and forth in the rendered area so there's no new code that is being run or loaded. The only thing that changes is the data structure may be recursed on a different side (but since I run back and forth around the same distance, it shouldn't be taking any new branches). I've been googling around to try and get a better handle on the tiered compilation but from what I've read, I don't know why I'd be running into this - but this is almost certainly due to my lack of knowledge.

dotTrace also showed gaps in the timeline where a Native Thread appears to completely take over for ~50ms, which is not present when I was running on .NET Core 2.1. While 50ms is a pretty small timeframe, real time rendering ends up being visually painful to see.

Do you think this warrants more investigation? Or is this the kind of behavior we'd expect from tiered compilation running over the same code again and again (as in completely halt all other threads for <= 50ms).

If it is expected and seems reasonable, all is good and this issue can be closed. I have the solution by disabling tiered compilation.

If you think that something is amiss and doesn't seem quite right (as in having it consistently halt everything for 50ms), I could try some more stuff or maybe try and build the coreclr myself and play around with some stuff.

Let me know what you think.

[jkotas]

Should the JIT be spawning a thread and/or doing its business every 5-10 seconds?

it is not expected. Once the process reaches steady state, the tiered compilation should not be doing any work.

[sergiy-k]

/cc: @kouvel

I've had the same issue. I have a main thread, and extra threads. They can be System.Time, BackgroundWorkers or simple Threads.

They all call Console.WriteLine("something"), and you can see the stutter if you do that. It's like the execution of the whole program stops for ~1 second every once in a while. Changing the target framework from 3.5 to 4.0 solved it in my case.

[kouvel]

Another driver bug masked it (meaning turning it off with the driver bug appeared to make no difference), but when I worked around the driver bug and then turned off tiered compilations the microstutter went away.

@WChrisK, just to confirm, after working around the driver bug, did you still see the micro stutter with tiered compilation enabled?

Should the JIT be spawning a thread and/or doing its business every 5-10 seconds?
dotTrace also showed gaps in the timeline where a Native Thread appears to completely take over for ~50ms, which is not present when I was running on .NET Core 2.1. While 50ms is a pretty small timeframe, real time rendering ends up being visually painful to see.

Tiered compilation uses a thread pool thread to do some jitting in the background. It rejits methods that were previously called a number of times, with full optimizations. The thread pool thread is released every ~50+ ms. However, it's not supposed to interfere heavily with other threads performing work. There are a few ways in which it could interfere:

• Using a thread pool thread means one thread is taken up for background jitting every so often, and if the app is relying heavily on the thread pool, the background jitting could interfere. Does your app make heavy usage of the thread pool?
• If the app is affinitized to fewer processors or if there are fewer processors, the background jitting could take up a larger portion time away from the app. If affinitized to one processor, it would take away time from the app for the whole duration. Is the process affinitized, and how many processors/cores does the system have?
• Some other miscellaneous interactions during background jitting could slow down other threads, not likely to be the cause

GCs shouldn't be affected by background jitting.

So I wonder what is actually causing app threads to show a stutter. Tiered compilation also does some method call counting, which currently has a fair bit of overhead and can directly affect app threads. Could you please try the following:

• Enable tiered compilation. Note that if you're using a property in the .csproj to configure this, changing the property will need a clean rebuild to reflect the change in the .runtimeconfig.json file. Or you may change the .runtimeconfig.json file directly, or use the environment variable.
• set COMPlus_TC_CallCountThreshold=1 in a terminal and start the process from the same terminal
• Look for micro stutters

That env var would eliminate call counting and skip directly to background jitting for methods that are called. It may end up jitting more methods, but there shouldn't be any call counting overhead. Curious if the micro stutters still happen, and if their duration changes.

[WChrisK]

@kouvel

@WChrisK, just to confirm, after working around the driver bug, did you still see the micro stutter with tiered compilation enabled?

Yes, still there as of .NET Core 3.0 preview 9 when tiered compilation is on.

I made a discovery just now while testing out the env vars: It does not happen when in debug mode. The small stuttering only happens when in release mode when neither tiered compilation is disabled or call count threshold is changed.

Using a thread pool thread means one thread is taken up for background jitting every so often, and if the app is relying heavily on the thread pool, the background jitting could interfere. Does your app make heavy usage of the thread pool?

Did some checking: I avoid thread pools, but to make sure I asked the OpenTK devs resulted in them telling me there is no thread pool usage by that library either. This means there shouldn't be any thread pool usage anywhere in the application.

However there is a .NET SystemEvents thread that is in the background sometimes. I don't know if that affects anything however, and dotTrace indicates that it does a very minimal amount of work.

If the app is affinitized to fewer processors or if there are fewer processors, the background jitting could take up a larger portion time away from the app. If affinitized to one processor, it would take away time from the app for the whole duration. Is the process affinitized, and how many processors/cores does the system have?

No affinity changes done by me or the underyling OpenTK library. My processor has 6 cores/12 threads.

set COMPlus_TC_CallCountThreshold=1 in a terminal and start the process from the same terminal

Interesting! This resolves the stuttering issue when tiered compilation is on. There is a small little blip or two at the beginning (first 3-4 seconds of the app) but I am not sure if this is due to Nvidia's GeForce thing popping up and going away, or if it's something related to the C# VM. The tiny blip or two at the start is acceptable for me since a small little jitter at the beginning is okay and won't bother any of my users.

I toggled COMPlus_TC_CallCountThreshold on and off a few times, and the stutter will happen every time I run the program with it not being set, and then when I set COMPlus_TC_CallCountThreshold=1 the stutter goes away in every single run of the program I do. I can say with confidence that this env var effectively hides (or resolves?) whatever the issue is.

Let me know if there's anything else you want me to check for or to do.

[kouvel]

Thanks @WChrisK, that's good to know. The fix for that issue is planned for .NET Core 5 and probably would be too large to take for 3.0 or 3.1. For now your best option would be to turn off tiered compilation for the app through config.

[kouvel]

Closing in favor of https://github.com/dotnet/coreclr/issues/23596