WCF Performance Problem

[Lxiamail]

#https://github.com/dotnet/corefx/issues/32819
@cicciog90 commented a day ago
We experienced a serious performance problem using WCF .NET Core 2.1 client libs.
Connecting to a service, the first bunch of calls are very slow (about 2 seconds) but after several trials it becomes very fast (about 30 milliseconds).
The same code compiled with .NET Framework 4.7.1 is fast with no differences during all calls.
You can find a sample here: https://github.com/cicciog90/WCF_Performance_Test

I uploaded it on my drive.
This is the shared link: https://drive.google.com/file/d/1crWSI9HYRVHRwpRE5CUaNntEr_XIMgqO/view?usp=sharing

[KKhurin]

Sorry about the delay, @cicciog90.

Short answer: this has to do with the CLR thread pool starvation.
One workaround is to artificially increase the minimum thread count via environment variable or in the application config:
set ComPlus_ThreadPool_ForceMinWorkerThreads=100
"System.Threading.ThreadPool.MinThreads": 100
see https://github.com/dotnet/cli/blob/rel/1.0.0/Documentation/specs/runtime-configuration-file.md for more details.

Caution: this may create other performance side effects as now there will be too many threads chasing work (it got much better in 2.1 though).

Long answer: before we go to a "workaround mode" by forcing more threads in the runtime, the test can be improved to avoid blocking calls such as syncronously waiting for the Result of a task. Effectively this is blocking the thread and in many cases it would take a few seconds in create a few new threads to be able to ramp up the throughput. So consider rewriting the lambdas & the loop to await for the async calls.

With only 3 simple changes in your example I was able to get results that were comparable with the artificially increased ThreadPool limits:

    //make the parrallel.for lambda async:
    Parallel.For(0, iterationCount, async i => ...

    //await for client.GetPeopleAsync
    GetPeopleResponse response = await client.GetPeopleAsync(new GetPeopleRequest());

    //close asynchrounously
    await client.CloseAsync();

Another consideration is to enable server GC. Depending on the workload WCF can be relatively allocation heavy and the server GC will handle it much better especially on systems with many cores. The perf benefits of the server GC will come at a cost of more memory pre-allocated per CPU core so this is a tradeoff.

Finally, very very high workloads for Net.Tcp transport (500K requests/second) you would benefit from using multiple channel factories and "affinitizing" tasks/threads to use a dedicated channel per task. You probably won't see considerable difference for lower workloads though (<20K req/second).

Hi @KKhurin, thanks for the reply.
Will the bug be fixed in .NET Core 3.0?
Otherwise I would have to modify all the methods of our enterprise application to apply the async \ await pattern.

@KKhurin I tried to apply your workaround to the test code as you told me to do but the situation does not seem to be improved.

[KKhurin]

@cicciog90, sorry I missed one more critical element in overall startup picture.

Comparing to the full desktop framework CoreCLR does not have all assemblies natively compiled. Given a relatively large size of WCF libraries it takes roughly (depends on the CPU speed) 800ms to JIT the code while executing the first request(s). In theory one could tell CoreClr to natively compile WCF assemblies and this will eliminate the delay. However our quick try today didn't produce the desired result and we're still jitting even after successfully generating the native image. So we can not recommend this approach to you yet. I opened #3247 to track this.

To confirm that the remaining delay difference between CoreClr and Desktop is due to the jitting code you can modify the client to make an additional call to any service before you start the main measurement loop. This first call will take roughly 700-800ms but all the calls in the parallel.for loop will now be as fast as the desktop ones. Especially once you convert the app to the async await calls. It won't even matter which service you make a call to - this could be a different net.tcp endpoint / service - what matter is that the WCF code will be jitted by the time you start the perf measurements.

With regard to changing the code to async/await. In my opinion this would be one of the best improvements in any case (even without the starvation) especially for a high throughput high scalability app. Having a mixture of async/sync/async calls in a stack really hurts the performance and scalability. It defeats the purpose of an incredible number of small optimizations put in place by the CLR team to run the async code as scalable and as performant as possible. Not mixing async&sync code is among one of the best practices around. I am not aware of any plans in CoreCLR ThreadPool to improve the starvation detection logic. But I am sure it is possible to improve things. As for now there is at least a way to artificially tell the thread pool to increase the lower limit.

@KKhurin So, from what I have understood, I have no option to workaround the problem. Is it?

[KKhurin]

The mitigations for thread pool starvation are to change the code paths to async and if this is not enough (or not feasible) - artificially increase the thread pool limits.

The mitigation for the WCF code to not being JITted is to issue a preliminary request as early as possible in the app lifetime so you have a good chance to avoid the jitting delays when actual requests are made.

With those 2 workarounds I was able to match the desktop times in your sample app.

As we work on #3247 and find a way to make crossgened binary work the second part won't be necessary anymore.

[Lxiamail]

@cicciog90 hope @KKhurin's suggestion was helpful. Closing the issue due to being inactive for a while.