Performance difference: fortunes, updates, fortunes, fortunes_dapper, multiple_queries ...

[pr-benchmarks]

No description provided.

[sebastienros]

/cc @roji

[sebastienros]

Was good for update everywhere
Was bad for windows and amd linux (more cores) everywhere else

[roji]

Wow... This is aspnet/Benchmarks#1667 right?

/cc @NinoFloris @ajcvickers

[roji]

Is everything the same or better on aspnet-citrine-lin as I saw in my manual tests?

[NinoFloris]

But it makes sense, max connection pool needs to be tuned to the hardware unfortunately. We could try to come up with a heuristic for figuring out the value during startup instead of static config.

[sebastienros]

Yes, it's better on all benchmarks for citrine-lin.

[roji]

Seems to also be (a lot) worse for updates on ARM-Linux.

Am talking with @NinoFloris, we'll investigate this ASAP and see what to do. Some sort of managing of the Max Pool Size based on number of cores/workload is probably needed... Maybe as a first step static Max Pool Size per machine profile (win vs lin, amd vs. intel) would work.

[roji]

Here are detailed runs with and without multiplexing, and with varying MaxPoolSize values, across all our machine profiles:

aspnet-citrine-lin

Hardware spec: Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz, 2195 Mhz, 14 Core(s), 28 Logical Processor(s)

Scenario	Multiplexing, MaxPoolSize=18	Multiplexing, MaxPoolSize=36	Multiplexing, MaxPoolSize=256	No multiplexing, Max PoolSize=256
Fortunes platform	456,793	448,634	435,797	305,571
Fortunes	302,252	296,910	300,862	287,126

aspnet-citrine-win

Hardware spec: Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz, 2195 Mhz, 14 Core(s), 28 Logical Processor(s)

Scenario	Multiplexing, MaxPoolSize=18	Multiplexing, MaxPoolSize=36	Multiplexing, MaxPoolSize=256	No multiplexing, Max PoolSize=256
Fortunes platform	318,573	339,404	346,469	337,615
Fortunes	235,880	237,455	236,041	283,866

aspnet-citrine-amd

Hardware spec: AMD EPYC 7402P 24-Core Processor (48 virtual), 2 sockets,

Scenario	Multiplexing, MaxPoolSize=18	Multiplexing, MaxPoolSize=36	Multiplexing, MaxPoolSize=256	No multiplexing, Max PoolSize=256
Fortunes platform	420,549	398,128	360,005	395,827
Fortunes	268,430	268,690	262,825	410,810

aspnet-citrine-arm

Hardware spec: ThinkSystem HR330A, 1x 32-Core/3.0GHz eMAG CPU

Scenario	Multiplexing, MaxPoolSize=18	Multiplexing, MaxPoolSize=36	Multiplexing, MaxPoolSize=256	No multiplexing, Max PoolSize=256
Fortunes platform	65,294	68,797	60,208	50,358
Fortunes	51,714	47,121	49,265	57,816

Thoughts:

• aspnet-citrine-lin is the profile tracked at TechEmpower, so it's probably important to keep that as optimized as possible. Based on Tune maximum connection pool size for Npgsql, enable multiplexing for non-platform benchmarks aspnet/Benchmarks#1667 and the identical results above, that means it should keep running with multiplexing and MaxPoolSize=18 for all scenarios.
• Other profiles aren't tracked at TechEmpower, so their main importance is probably more to track perf and identify regressions.
• Generally multiplexing does not work well for non-platform (except for aspnet-citrine-lin), and even when it does, the optimal MaxPoolSize varies in a way which we don't yet fully understand.
• So in the short term, I'd keep multiplexing enable for all aspnet-citrine-lin scenarios, and disabled for all the others; or we could just keep it enabled everywhere, with the regressions it brings to other platforms
• We could do an intricate config where each profile has its own settings, but I'm not sure there's much value in that. At some point we'd investigate multiplexing and MaxPoolSize in a more thorough way to understand exactly when it works well and why, and try to bring those changes into Npgsql itself, rather than as static configuration.
• (Note that this would regress some results in the graphs, since we used to have Fortunes and Single Query with multiplexing for all profiles, not just aspnet-perf-lin).

@sebastienros what do you think? We can define the environment variable at the machine profile level, right?

[roji]

(if we want, we can keep multiplexing on for the platform benchmarks, across all machine profiles - it does generally seem to perform better, except on Windows)

[roji]

BTW check out what multiplexing did to perf on the Azure cloud platform...!

[ajcvickers]

@roji At what point do we start switching multiplexing on by default in the provider?

[roji]

@ajcvickers I'd have loved to, but the table above isn't that positive :(

Seems like I'm mainly lacking an understanding of exactly when and how this feature improves things...