Improve EF Core (non-tracking) query performance on TechEmpower Fortunes

[ajcvickers]

We have been running the industry standard TechEmpower benchmarks on .NET against a PostgreSQL database for several years. In particular, the Fortunes benchmark is particularly relevant to EF scenarios. We have multiple variations of this benchmark, including:

• An implementation that uses ADO.NET directly. This is the fastest implementation of the three listed here.
• An implementation that uses Dapper. This is slower than using ADO.NET directly, but still fast.
• An implementation that uses EF Core. This is currently the slowest implementation of the three.

The goal for EF Core 6.0 is to get the EF Core performance to match that of Dapper on the TechEmpower Fortunes benchmark. This is a significant challenge which may not be fully achieved in EF Core 6.0. Nevertheless, we will get as close as we can.

[roji]

All benchmarks executed on aspnet-citrine-lin (TechEmpower hardware), 3 iterations.

Summary

Description	EF Core	Change	Dapper	Change	EF/Dapper Gap
Starting point (EF 5.0 + .NET 5.0)	147,852		230,394		55.8%
EF benchmark improvements (no product changes)	191,189	29.3%	230,394		20.5%
.NET 5 -> .NET 6	209,953	9.8%	263,314	14.3%	25.4%
EF 6.0 + .NET 6.0 (product optimizations)	251,992	31.8%	263,314		4.5%
Total change:		70.4%		14.3%

Detailed changes

Description	RPS	RPS Change	Change Type	Notes	Commits tested
Starting point (EF Core 5.0, .NET 5.0, no benchmark improvements)	147,852
Set EF DbContext pool size to 1024 (instead of default 128)	182,817	23%	Benchmark
Mark Fortunes message as required (elide null checks)	185,970	1.7%	Benchmark
Turn on Npgsql multiplexing and reduce Max PoolSize to 18	191,189	2.8%	Benchmark	Thanks @NinoFloris!!
Switch from .NET 5 to .NET 6	209,953	9.8%	.NET
Cache ResettableServices for DbContext pooling	209,339	-0.2%	EF		EF 3b0a18b, EFCore.PG 2e8bf9a4
Stop capturing in logging	212,273	1.4%	EF	10% reduction in number of instances allocated, 4% in bytes.	EF aecae63, EFCore.PG 2e8bf9a4
Make ConcurrencyDetector optional	226,511	6.7%	EF		EF 7dada2a, EFCore.PG 2e8bf9a4
Reduce allocation in query for execution strategy	226,963	0.1%	EF		EF ec6be85, EFCore.PG 2e8bf9a4
Skip accessing Transaction.Current	227,125	0%	EF		EF 083042b, EFCore.PG 6c6a3cad
Recycle relational and ADO.NET objects in query execution	231,960	2.1%	EF	29% reduction in number of instances allocated, 22% in bytes.	EF 3f5dc2c, EFCore.PG 6c6a3cad
Suppress logging checks in query when not required, caching for 1 second	248,186	7%	EF		EF 02bb112, EFCore.PG 70e000d2
Reduce accesses to virtual properties in query	251,992	1.5%	EF	Thanks @vonzshik!!	EF 6f7f730, EFCore.PG 70e000d2

Previous figures with aspnet-perf-lin (weaker machine)

Overview

Version	EF Core	Dapper	Gap
Initial (5.0)	69,653	93,341	34%
Latest (6.0)	97,816 (40% better)	104,629 (12% better)	6.9%

Detailed changes

Description	RPS	RPS Change	Change Type	Notes	Commits tested
Starting point (EF Core 5.0, .NET 5.0, no benchmark improvements)	147,852
Set EF DbContext pool size to 1024 (instead of default 128)	182,817	23%	Benchmark
Mark Fortunes message as required (elide null checks)	185,970	1.7%	Benchmark
Turn on Npgsql multiplexing and reduce Max PoolSize to 18	191,189	2.8%	Benchmark	Thanks @NinoFloris!!
Switch from net5.0 TFM to net6.0	209,953	9.8%	.NET
Cache ResettableServices for DbContext pooling	209,339	-0.2%	EF		EF 3b0a18b, EFCore.PG 2e8bf9a4
Stop capturing in logging	212,273	1.4%	EF	10% reduction in number of instances allocated, 4% in bytes.	EF aecae63, EFCore.PG 2e8bf9a4
Make ConcurrencyDetector optional	226,511	6.7%	EF		EF 7dada2a, EFCore.PG 2e8bf9a4
Reduce allocation in query for execution strategy	226,963	0.1%	EF		EF ec6be85, EFCore.PG 2e8bf9a4
Skip accessing Transaction.Current	227,125	0%	EF		EF 083042b, EFCore.PG 6c6a3cad
Recycle relational and ADO.NET objects in query execution	231,960	2.1%	EF	29% reduction in number of instances allocated, 22% in bytes.	EF 3f5dc2c, EFCore.PG 6c6a3cad
Suppress logging checks in query when not required, caching for 1 second	248,186	7%	EF		EF 02bb112, EFCore.PG 70e000d2
Reduce accesses to virtual properties in query	251,992	1.5%	EF	Thanks @vonzshik!!	EF 6f7f730, EFCore.PG 70e000d2

[roji]

Here's the memory reduction info. The following includes only improvements done between EF Core 5 and 6: no benchmark code changes or .NET-related changes, but including some minor Npgsql improvements.

Scenario	MB allocated	# objects allocated
EF Core 5	54.84	910,000
EF Core 6	31.14	379,700
Reduction	43.2%	58.3%

[roji]

Here's a comparison with and without EF's compiled query feature:

Scenario	RPS
With compiled	257,285
Without compiled	247,480

So disabling compiled queries reduces perf by 3.8%. Not a lot, but it's worth mentioning the that the Fortunes query itself is as minimal as can be (just enumerate the DbSet - no operators, no parameters); as the query becomes more complex, the gap will increase.

Crank cmdlines

With compiled:

dotnet run -- --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --scenario fortunes_ef --profile aspnet-citrine-lin --application.framework net6.0 --application.options.outputFiles Z:\projects\test\bin\Release\net5.0\publish\*.dll -i 5

Without compiled:

dotnet run -- --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --scenario fortunes_ef --profile aspnet-citrine-lin --application.framework net6.0 --application.options.outputFiles Z:\projects\test\bin\Release\net5.0\publish\*.dll --application.source.repository http://github.com/roji/AspNetBenchmarks --application.source.branchOrCommit NonCompiledQuery -i 5

/cc @ajcvickers

[roji]

Reran the numbers for EF5 vs. EF6, .NET 5 vs. .NET 6. All these use the latest version of the benchmarks - this includes the new connection string to use multiplexing and MaxPoolSize=18 (aspnet/Benchmarks#1667).

	.NET 5	.NET 6	Improvement
EF 5	194,607	205,257	5.47%
EF 6	240,795	249,957	3.80%
Improvement	23.73%	21.78%

Note I'm seeing less of an improvement than previously by going from .NET 5 to .NET 6 (5.47% only compared to 9.8% previously), possibly due to the multiplexing/MaxPoolSize change.

	.NET 5	.NET 6	Improvement
Dapper	228,501	265,513	16%

/cc @AndriySvyryd @ajcvickers

[AndriySvyryd]

@roji Whenever you work on this again I think it could be beneficial to determine where Dapper is benefitting from the move to .NET 6 in case a similar optimization can be applied to EF Core.

[roji]

I reran .NET 5 vs. .NET 6 for Dapper with the same benchmark version, to make sure we compare apples to apples (so with multiplexing and MaxPoolSize=18). I still see around the same improvement as before, even a bit more (16% instead of 14.3%).

Whenever you work on this again I think it could be beneficial to determine where Dapper is benefitting from the move to .NET 6 in case a similar optimization can be applied to EF Core.

Sure, though that's not a trivial question to answer... :)

[roji]

Am closing this for now, as we've achieved good results for 6.0 and don't have immediate plans for further work.