ASP.NET Core 2.2 kestrel server's performance issue

[vishwas-trivedi]

※Describe the bug

I'm facing problem with kestrel server's performance. I have following scenario :

TestClient(JMeter) -> DemoAPI-1(Kestrel) -> DemoAPI-2(IIS) 

I'm trying to create a sample application that could get the file content as and when requested. TestClient(100 Threads) requests to DemoAPI-1 which in turn request to DemoAPI-2. DemoAPI-2 reads a fixed XML file(1 MB max) and returns it's content as a response(In production DemoAPI-2 is not going to be exposed to outside world).

When I tested direct access from TestClient -> DemoAPI-2 I got expected result(good) which is following :

• Average : 368ms
• Minimum : 40ms
• Maximum : 1056ms
• Throughput : 40.1/sec

But when I tried to access it through DemoAPI-1 I got following result :

• Average : 48232ms
• Minimum : 21095ms
• Maximum : 49377ms
• Throughput : 2.0/sec

As you can see there is a huge difference.I'm not getting even the 10% throughput of DemoAPI-2. I was told kestrel is more efficient and fast compared to traditional IIS. Also because there is no problem in direct access, I think we can eliminate the possible of problem on DemoAPI-2.

※Code of DemoAPI-1 :

using (var httpClientHandler = new HttpClientHandler())
        {
            httpClientHandler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => { return true; };
            using (var client = new HttpClient(httpClientHandler))
            {
                var req = new HttpRequestMessage(HttpMethod.Get, url);
                req.Headers.Add(HeaderUserAgent, UserAgentValue);
                var response = await client.SendAsync(req).ConfigureAwait(false);
                string buffer = null;
                using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
                {
                    var read = new byte[DataChunkSize];
                    var bytes = stream.Read(read, 0, DataChunkSize);

                    while (bytes > 0)
                    {
                        buffer += System.Text.Encoding.UTF8.GetString(read, 0, bytes);
                        bytes = stream.Read(read, 0, DataChunkSize);
                    }
                }
                return buffer;
            }
        }

※Code of DemoAPI-2 :

[HttpGet("Demo2")]

public async Task<IActionResult> Demo2Async(int wait)
  {
      try
      {
          if (wait > 0)
          {
              await Task.Delay(wait);
          }
          var path = Path.Combine(Directory.GetCurrentDirectory(), "test.xml");
          var file = System.IO.File.ReadAllText(path);
          return Content(file);
      }
      catch (System.Exception ex)
      {
          return StatusCode(500, ex.Message);
      }
  }

Additional context

1. Both APIs are async.
2. Both APIs are hosted on different servers (Windows Server 2016).
3. DemoAPI-1(kestrel) is a self-contained API(without reverse proxy)
4. TestClient(jMeter) is set to 100 thread for this testing (want to test upto 1000 threads).
5. No other configuration is done for kestrel server as of now.
6. There are no action filter, middleware or logging that could effect the performance as of now.
7. Communication is done using SSL on 5001 port.
8. Wait parameter for DemoAPI2 is set to 0 as of now.
9. The CPU usage of DEMOAPI-1 is not over 40%.
10. I've performed similar test for 100KB and 500KB files and didn't have any problem.

https://stackoverflow.com/questions/56387959/asp-net-core-2-2-kestrel-servers-performance-issue

Screenshots

I've attached the performance monitor's screenshot.

If this is not the correct place for this question please let me know where should I ask about this.

[Tornhoof]

Looking at your Code you open/Close HttpClientHandler for each request, that could lead to the classic HttpClient port exhaustion problem.
See e.g. https://www.stevejgordon.co.uk/httpclient-creation-and-disposal-internals-should-i-dispose-of-httpclient on how to properly use these types.

Then you should rethink that buffer/String encoding logic as already noted on Stack Overflow, for the sync behaviour, the amount of String concatenation, Byte arrays etc.

[vishwas-trivedi]

@Tornhoof

Then you should rethink that buffer/String encoding logic as already noted on Stack Overflow, for the sync behaviour, the amount of String concatenation, Byte arrays etc.

Thanks for pointing it out.
I've updated read stream code to following but didn't see much difference:

string base64Encoded = null;
                var request = new HttpRequestMessage(HttpMethod.Get, url);
                var response = await this.httpClient.SendAsync(request, HttpCompletionOption.ResponseContentRead).ConfigureAwait(false);

                if (response.StatusCode.Equals(HttpStatusCode.OK))
                {
                    var content = await response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
                    base64Encoded = Convert.ToBase64String(content);
                }

if read was blocking then it should be should affect CPU usage in some way but as you can seen in the image CPU usage is around 40% only.

Looking at your Code you open/Close HttpClientHandler for each request, that could lead to the classic HttpClient port exhaustion problem.

So I should use HttpClientFactory instead of HttpClient to avoid overhead of re-initialization and disposal of HttpClient and related socket resources, right?

But...

1. Does it really have that big of a impact? I mean I'm getting performance below 10% of DemoAPI-2.
2. Can port exhaustion has that big of an impact?
3. Will similar problem occur on ASP.NET too? or is it just with ASP.NET Core?

Also, thanks for the link, it's an interesting article (didn't know about this issue).

Regards,
Vishwas

[Tornhoof]

Yes, you should use HttpClientFactory and yes, the Httpclient Problems exists on ASP.Net too.
As for impact, it's hard to tell, it's an obvious problem in your code and very easy to fix.

[vishwas-trivedi]

@Tornhoof

Thank you!
I'm working on a sample to deploy and re-check the same.
I'll share the results soon.

Regards,
Vishwas

[dcarr42]

Those bytes will be held in memory. I would suggest not buffering the response and read the raw network stream and hand it off to stream reader for processing. For larger requests it could be a consideration.

…

On Fri, 31 May 2019, 08:26 vishwas-trivedi, ***@***.***> wrote: @Tornhoof <https://github.com/Tornhoof> Thank you! I'm working on a sample to deploy and re-check the same. I'll share the results soon. Regards, Vishwas — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10681>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKSKTDSKYK465XG7ULMP4DPYDHIZANCNFSM4HRS7XAA> .

[vishwas-trivedi]

@Tornhoof

Thank you, I was able to confirm that the problem was indeed because of HttpClient.
I was able achieve comparatively better results which is as following with IHttpClientFactory:

• Average : 734ms
• Minimum : 63ms
• Maximum : 1841ms
• Throughput : 34.1/sec

It's not as fast as DemoAPI-2 but I think I can work with it.

Regards,
Vishwas

[vishwas-trivedi]

@DAllanCarr

Those bytes will be held in memory. I would suggest not buffering the response and read the raw network stream and hand it off to stream reader for processing. For larger requests it could be a consideration.
…
On Fri, 31 May 2019, 08:26 vishwas-trivedi, @.***> wrote: @Tornhoof https://github.com/Tornhoof Thank you! I'm working on a sample to deploy and re-check the same. I'll share the results soon. Regards, Vishwas — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10681?email_source=notifications&email_token=ADKSKTFLCHXFDC6V46MM2TDPYDHIZA5CNFSM4HRS7XAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWUNXJQ#issuecomment-497605542>, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKSKTDSKYK465XG7ULMP4DPYDHIZANCNFSM4HRS7XAA .

I would be grateful if you could provide a sample. I want to return response in base-64 format from DemoAPI-1 as you can see in my previous comment(#10681 (comment)).

Regards,
Vishwas

[dcarr42]

You can try adapting CryptoStream but it seems you have the perf hike from httpclient work. On Fri, 31 May 2019, 10:40 vishwas-trivedi, <notifications@github.com> wrote:

…

@DAllanCarr <https://github.com/DAllanCarr> Those bytes will be held in memory. I would suggest not buffering the response and read the raw network stream and hand it off to stream reader for processing. For larger requests it could be a consideration. … <#m_-8827034674454892256_> On Fri, 31 May 2019, 08:26 vishwas-trivedi, *@*.***> wrote: @Tornhoof <https://github.com/Tornhoof> https://github.com/Tornhoof Thank you! I'm working on a sample to deploy and re-check the same. I'll share the results soon. Regards, Vishwas — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10681 <#10681>, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKSKTDSKYK465XG7ULMP4DPYDHIZANCNFSM4HRS7XAA . I would be grateful if you could provide a sample. I want to return response in base-64 format from DemoAPI-1 as you can see in my previous comment(#10681 (comment) <#10681 (comment)> ). Regards, Vishwas — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10681>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKSKTHBA5JG5FHI4WASHBDPYDW77ANCNFSM4HRS7XAA> .

[vishwas-trivedi]

@Tornhoof

Just wanted to confirm whether there's a way to use this in .netstandard 2.0 library project as well?
All I could find from official document was how to use it with DI, so can it be used without DI as well?

Regards,
Vishwas

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!