Tune maximum connection pool size for Npgsql, enable multiplexing for non-platform benchmarks

[NinoFloris]

During offline conversations with @roji I noticed the TE Npgsql max pool size being at 256. As I was recently doing some tuning of my own https://twitter.com/NinoFloris/status/1381651573978316809 I experienced first hand how reducing max pool size can have a very positive effect on perf. @roji then tested a matrix of max pool sizes in the perf lab and landed on 18 connections for a 4% increase in RPS on platform fortunes. (from ~444,239 to ~461,284)

This PR reflects the changes to the connection strings in many of the places referring to this 256 max though I skipped orchard and mono. I'm not too familiar with the repo structure and all its yamls so I welcome a thorough review of the touched/missing files.

[NinoFloris]

OK results for non platform scenarios are in. They need a different number, in part (likely but untested) due to multiplexing being disabled in these cases.

[NinoFloris]

Confirmed, enabling multiplexing in the others brings similar improvements :)

Pushed commits with minimal files changed and modified the NoResetOnClose check to include Multiplexing as an alternative option (enabling multiplexing implies NoResetOnClose).

[roji]

Some detailed benchmark results:

Fortunes platform benchmark (raw)

Baseline:       444,239 (x5), current setup with MaxPoolSize=256

MaxPoolSize=1:  65,474  (x1)
MaxPoolSize=10: 373,926 (x1)
MaxPoolSize=12: 451,154 (x1)
MaxPoolSize=14: 446,962 (x3)
MaxPoolSize=15: 454,214 (x3)
MaxPoolSize=16: 459,294 (x3)
MaxPoolSize=17: 456,241 (x3)
MaxPoolSize=18: 461,284 (x5) 467,404 (x3)
MaxPoolSize=19: 457,492 (x3)
MaxPoolSize=20: 459,221 (x3)
MaxPoolSize=21: 457,131 (x5), 463,056 (x3)
MaxPoolSize=22: 458,505 (x5)
MaxPoolSize=25: 449,131 (x3)
MaxPoolSize=32: 439,758 (x1)
MaxPoolSize=64: 438,651 (x1)

With MaxPoolSize=18, we get an improvement of 461,284-444,239=17045, or 3.8% RPS.

Fortunes non-platform raw

Baseline (MaxPoolSize=256): 291,597 (x5)
With MaxPoolSize=18: 129,734 (x5)
With MaxPoolSize=18 and multiplexing (same connection string as platform): 304,009 (x3)

With multiplexing, we get an improvement of 304,009-291,597=12412, or 4.2% RPS.

Fortunes non-platform EF Core 5 (.NET 6.0, latest benchmark code)

Baseline 220,956 (x5)
With MaxPoolSize=18 and multiplexing: 232,890 (x5): 5%

With multiplexing, we get an improvement of 232,890-220,956=11934, or 5.4% RPS.

Fortunes non-platform EF Core 6 (.NET 6.0, latest benchmark code)

Before: 242,040 (x5)
After: 250,554 (x5): 3.5%

With multiplexing, we get an improvement of 250,554-242,040=8514, or 3.5% RPS.

---

Still to do: measure the non-Fortunes platform benchmarks, enable multiplexing and reduce MaxPoolSize accordingly.

Thanks again for this awesome work @NinoFloris - I'm just the benchmarking guy here :)

/cc @ajcvickers @davidfowl @benaadams

[benaadams]

By incorperating these changes; can Ben overtake Platform on Fortunes? 🤔😅 TechEmpower/FrameworkBenchmarks#6567

[sebastienros]

You are already running with Full PGO which is cheating ;)

[benaadams]

You are already running with Full PGO which is cheating ;)

Is also on .NET 6.0; not my fault the "offical" ones move so sloooowly 😉

[roji]

Here's what this does to all the platform benchmarks (5 iterations for each value):

Scenario	Before	After	Change	Notes
Fortunes (raw)	444,239	461,284	3.8%	Only changed MaxPoolSize (was already multiplexed)
Single query	386,557	396,706	2.6%	Only changed MaxPoolSize (was already multiplexed)
Multiple queries	22,723	23,861	5%	Turned on multiplexing and reduced MaxPoolSize
Caching	256,058	255,201	-0.3%	Turned on multiplexing and reduced MaxPoolSize
Updates	13,640	18,506	35%	Turned on multiplexing and reduced MaxPoolSize

Note the huge 35% jump for Updates! Caching regressed by 0.3%, though it's so small it might just be variance. Note that I didn't test out different MaxPoolSize values (except for Fortunes yesterday) - all the above were run with MaxPoolSize=18. At some point we can also do that - it's possible that different MaxPoolSize values fit different scenarios better.

So @NinoFloris I think it makes sense to just use the same connection string (i.e. the Fortunes one) globally for all the platform benchmarks (see the non-platform database YAML for an example), possibly overriding only for caching to turn this off.

[n-stefan]

Sorry to intrude but I remember seeing an improvement of around 40% in Fortunes non-platform (micro / raw), just by enabling Multiplexing (and keeping MaxPoolSize at 256). Even EF improved by about 20%, though I think wrk was reporting errors.

Could be wrong though, it was a while ago.

[roji]

@n-stefan no intrusion whatsoever! Always good to have comments/thoughts.

Here are some more tests for non-platform Fortunes raw (no dapper/ef, 3 iterations per scenario):

Scenario	RPS
No multiplexing + MaxPoolSize=256	289,834
Multiplexing + MaxPoolSize=256	297,662
Multiplexing + MaxPoolSize=18	303,632

So you seem to be right that multiplexing improves things even without changing Max Pool Size (though Max Pool Size does help a bit). I'm not seeing anything like 40% though - any chance you can look into this and see where it came from?

I do remember running a few quick benchmarks back when I was finishing the multiplexing feature, and seeing unsatisfactory results with the non-platform benchmarks. Something may have changed since then, e.g. the above is with latest .NET 6. It's also possible I was very focused on platform benchmark perf and neglected to properly test the non-platform benchmarks, I can't be sure any more...

[roji]

LGTM.

@sebastienros can you merge this?

[n-stefan]

@roji
I should have kept those results...
Was running the benchmarks locally, using the standard TechEmpower toolset. Npgsql version was 5 alpha or preview.
Could try to replicate the results, time permitting.

[n-stefan]

@roji
Using the latest TechEmpower sources the local results for aspcore-mw-ado-pg (Micro/Raw) Fortunes are:

Maximum pool size 256
Multiplexing off
23,591 rps
https://www.techempower.com/benchmarks/#section=test&shareid=b02ea711-f2f1-4df8-9083-505ff37f1e92

Maximum pool size 256
Multiplexing on
36,733 rps
https://www.techempower.com/benchmarks/#section=test&shareid=ce0140d6-4aa2-44a1-815e-a9a2c364a6d9

Maximum pool size 18
Multiplexing on
36,862 rps
https://www.techempower.com/benchmarks/#section=test&shareid=0d431229-3a7c-4a4c-8b19-5e9be773f04d

[roji]

@n-stefan what exactly are these runs, where are they coming from and where's their conf? The numbers are totally off compared to what I'm seeing for non-platform TechEmpower Fortunes on Microsoft's Citrine hardware (which is supposed to be identical to the TE machine) - see the numbers I posted above.

[n-stefan]

They are local results as I don't have access to Citrine, and this alone is an important difference.
Hardware is a Core i7 2600k quad core CPU with Hyperthreading (8 logical CPUs) at 4 GHz; Docker Desktop on Windows 10 set to 7 CPUs.
Otherwise I used the very latest source code, only changing Maximum Pool Size and Multiplexing in the connectionstring between tests.

[roji]

@n-stefan it's great to see the multiplexing improves perf on your machine (regardless of Max Pool Size). But it's expected that you'd see very different perf on your machine as compared to the TechEmpower hardware, which is considerably stronger (e.g. 12 cores).