Performance difference when using ResponseBuffering middleware

[chrisckc]

The default configuration of a new .Net Core project does not use Reponse Buffering, instead the response is sent in chunks so that the result payload can start to be sent while it is being serialized which is supposed to improve memory usage and performance. This is a new feature in .NET Core, in my initial testing when coming from the previous generation of .NET i was not aware of this new feature that was enabled by default.

I have done some basic testing to help decide whether it is worth sticking with the default or using the ResponseBuffering middleware which in theory would slow things down and consume more server RAM.

The tests where run from a .Net Core 2 Web API using an in memory data repository. The throughput tests where ran for 10 seconds several times over to obtain an average result. The payload size is the total size including the headers. The memory usage was monitored during a 60 second throughput test. (observing the 3 dotnet processes found).

Everything was performed locally on a Macbook Pro with 16GB RAM.

Throughput tests returning a 730 Byte JSON payload:
Default configuration (no response buffering):

• Average throughput: ~3500 requests/sec
• Peak memory usage ~900MB

Using the ResponseBuffering middleware:

• Average throughput: ~4000 requests/sec
• Peak memory usage ~900MB

The results were surprising, faster average throughput when using ResponseBuffering with no real difference in memory usage so i then tried the same with a 10KB payload...

Throughput tests returning a 10KB JSON payload:
Default configuration (no response buffering):

• Average throughput: ~2500 requests/sec
• Peak memory usage ~400MB

Using the ResponseBuffering middleware:

• Average throughput: ~2400 requests/sec
• Peak memory usage ~300MB

The results were very close, so i then tried the same with a 100KB payload...

Throughput tests returning a 100KB JSON payload:
Default configuration (no response buffering):

• Average throughput: ~350 requests/sec
• Peak memory usage ~700MB

Using the ResponseBuffering middleware:

• Average throughput: ~360 requests/sec
• Peak memory usage ~350MB

Again, the results very close, but notice the significantly less memory overhead when using the Response Buffering middleware, so i then tried the same with a 1MB payload...

Throughput tests returning a 1MB JSON payload:
Default configuration (no response buffering):

• Average throughput: ~61 requests/sec
• Peak memory usage ~400MB

Using the ResponseBuffering middleware:

• Average throughput: ~45 requests/sec
• Peak memory usage ~350MB

With a 1MB payload the default configuration has faster throughput, but the memory overhead is still higher. Obviously a 1MB payload is excessive and the API should implement paging to split into multiple smaller requests, but for the purposes of testing it was worth seeing what the result would be.

Considering the significant issues that the default configuration without Response Buffering causes when errors occur during serialization, my tests show that it is well worth adding Response Buffering as there seems to be no benefit to not using it, only downsides.

Refer to this thread for a description of the issues:
#2285

I did a quick test from a separate machine over the network as a sanity check and the results were not much different, but i plan to do a better test in the future with a more of a production setup.

Update:
I added a test for 10MB payload and updated the results for the 1MB payload after double checking the my figures:

Throughput tests returning a 10MB JSON payload:
Default configuration (no response buffering):

• Average throughput: ~6 requests/sec
• Peak memory usage ~450MB

Using the ResponseBuffering middleware:

• Average throughput: ~6 requests/sec
• Peak memory usage ~400MB

[Tratcher]

Any latency metrics?

@sebastienros

[sebastienros]

I can add a scenario for this piece of middleware and come back with more stats.

[chrisckc]

I couldn't remember there being much difference in the latency, but i went back to check it and here are the figures, i haven't included the latency for the 730 byte payload:

Latency tests returning a 10KB JSON payload:
Default configuration (no response buffering):

• Average latency: ~0.9 ms

Using the ResponseBuffering middleware:

• Average latency: ~0.8 ms

Latency tests returning a 100KB JSON payload:
Default configuration (no response buffering):

• Average latency: ~4.5 ms

Using the ResponseBuffering middleware:

• Average latency: ~4.6 ms

Latency tests returning a 1MB JSON payload:
Default configuration (no response buffering):

• Average latency: ~33 ms

Using the ResponseBuffering middleware:

• Average latency: ~44 ms

Latency tests returning a 10MB JSON payload:
Default configuration (no response buffering):

• Average latency: ~350 ms

Using the ResponseBuffering middleware:

• Average latency: ~340 ms

As you can see, any difference in latency for the most common use cases (up to 100KB payloads) is insignificant in comparison to the latency of even the fastest broadband connections.

Again, as in the throughput test results, its only on the 1MB payload where the results are notably better for the default config, i am not sure why that isn't the case for the 10MB payload as well.

I plan to test this over a network link from a linux server to a client, but a quick test from another machine to my Macbook showed very similar results in terms of the differences in throughput, but the throughput was less overall.

So far from this limited test, it seems that the only benefit of the default configuration without Response Buffering is for 1MB payloads, which i think is a rare and generally unwise use case. If further tests confirm this, it would make sense that Response Buffering should be enabled by default to suit the vast majority of use cases and provide much better error handling.

I can see the the response is delivered differently when not using Response Buffering as the time-to-first-byte for the 1MB example is 0ms and the complete request takes 33ms, showing that the chunked response transfer is working. When using Response Buffering the time-to-first-byte is 33ms and the request takes 33ms. This suggests that slower transfer speeds are required to see the benefits for the chunked response transfer.

It would be interesting to see how different network link speeds affect the throughput and latency for the 2 configurations.

[sebastienros]

I made a custom app there: https://github.com/aspnet/BasicMiddleware/tree/sebros/benchmarkapps/benchmarkapps/BufferingBenchmarks

Using our performance infrastructure, all combinations have better RPS and latency with non-buffered results. This is not a universal conclusion as this is only measuring a sample Github Json payload (more realistic). I also confirmed that MVC Views are already buffered so they should not be impacted and hence I didn't measure them.

Physical - Linux

Description	RPS	CPU (%)	Memory (MB)	Avg. Latency (ms)	Startup (ms)	First Request (ms)	Latency (ms)
Buffered	182,567	98	402	1.6	293	344.2	0.8
NonBuffered	193,437	98	398	1.5	289	337.6	0.9

Physical - Windows

Description	RPS	CPU (%)	Memory (MB)	Avg. Latency (ms)	Startup (ms)	First Request (ms)	Latency (ms)
Buffered	203,138	90	381	2.2	264	318.2	0.9
NonBuffered	210,276	88	383	2.1	260	264.8	0.9

Cloud - Linux

Description	RPS	CPU (%)	Memory (MB)	Avg. Latency (ms)	Startup (ms)	First Request (ms)	Latency (ms)
Buffered	52,965	99	309	5.2	457	519.3	1.4
NonBuffered	56,536	99	314	4.8	357	458.1	1.1

Cloud - Windows

Description	RPS	CPU (%)	Memory (MB)	Avg. Latency (ms)	Startup (ms)	First Request (ms)	Latency (ms)
Buffered	59,220	80	413	5.5	479	422.6	0.6
NonBuffered	61,171	86	413	5.1	497	406.5	1

[chrisckc]

I haven't ran your app yet, what was the size of the JSON response being measured?

Can you run this test for the response sizes that i have tested so see what you get?

Even for this example, for myself personally, there is not enough benefit to outweigh the significant disadvantages of NonBuffered. The differences are all less than about 6%

Ill try my next set of tests as soon as i get chance before i make a final decision.

I might even compare it to Node Express.... as like for like as possible of course.

[sebastienros]

I used this as an example: https://developer.github.com/v3/users/#get-the-authenticated-user

Feel free to add more scenarios to my app, even on your fork, I can run automated tests on anything that is in Github if it follows this app template. Also, I am testing using the dev branch of all aspnet, i.e.e 2.1, and not 2.0.

[Eilon]

Closing because no further action is planned for this issue.