Compiled Regex 3x performance drop in 2.1

[ptupitsyn]

I have migrated my project to .NET Core 2.1 and noticed a huge performance drop in RegEx matching when using RegexOptions.Compiled in a loop.

Simple reproducer using BenchmarkDotNet:
https://github.com/ptupitsyn/netcore2.1-regex-perf/tree/master/src
(just run, change netcoreapp2.1 to netcoreapp2.0 in csproj file, run again), shows ~3x drop.

This uses a big number of Regex instances (~700), trying to match them in a loop. I could not reproduce the issue with a single Regex.

Removing RegexOptions.Compiled solves the issue. On .NET Core 2.0 there is no difference with or without Compiled.

Related SO question: https://stackoverflow.com/questions/50678580/net-core-2-1-regex-in-loop-200x-slower-than-2-0-3x-in-simple-benchmark

[stephentoub]

On .NET Core 2.0 there is no difference with or without Compiled.

That's because on .NET Core 2.0, Compiled was a nop, it didn't cost anything nor did it compile anything or make anything faster. Now in 2.1 it's actually implemented, so you pay for the compilation and get the resulting benefits.
dotnet/corefx#24158

[ptupitsyn]

@stephentoub is it some kind of on-demand compilation? Because I create Regex instances outside of the benchmarked method.

[stephentoub]

is it some kind of on-demand compilation? Because I create Regex instances outside of the benchmarked method.

At a minimum there will be significant JIT costs on first use, as the compiled IL is then JIT'd into assembly. And you'll have a lot more code that'll need to be executed, since every compiled regex will have its own assembly to be executed.

@ViktorHofer can speak better to the costs involved in Compiled, though.

[ViktorHofer]

is it some kind of on-demand compilation? Because I create Regex instances outside of the benchmarked method.

UPDATE: @stephentoub talked with me. My statement was wrong.
The startup costs in your example should be very high because of the IL generation of the many regexes. The runtime results should be significally (~30%) faster.

That said, I don't really understand the results and need more time investigating the issue. thanks!

[danmoseley]

I wonder whether increasing the regex cache size to 700+ would help.

[ptupitsyn]

I wonder whether increasing the regex cache size to 700+ would help.

It does not, and, according to the doc, it should not.

@ViktorHofer yes, BenchmarkDotNet does a proper warmup, so I would expect all startup costs to be outside of the benchmarked method.

[efaruk]

Thanks for the good catch @ptupitsyn 👍

[rangp]

Ist there any update on the issue? I recently migrated to dotnetcore 2.1 and suffered heavy performance losses. Took me some time to figure out it was because of Regex.Compiled. I can confirm that a compiled regex evaluation is significantly slower than a non-compiled one when evaluating the expressions in a loop (252 expressions in my case), even after the initial compilation overhead

[ViktorHofer]

I tried to reproduce the issue with a simple pattern but couldn't see slowdowns with compiled enabled: https://gist.github.com/ViktorHofer/0e1517bc639ce3bf4a4414927b60b22d. Maybe I'm testing something the wrong way...

[ptupitsyn]

@ViktorHofer please see my reproducer linked above.

Yes, with a single pattern it does not reproduce, you have to use a bunch of patterns.

[ViktorHofer]

This also happens on .NET Framework. We currently don't have plans to optimize this scenario. I think it makes sense to revisit this topic when the collectible assemblies feature is implemented (https://github.com/dotnet/coreclr/issues/552#issuecomment-410238253).

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.285 (1803/April2018Update/Redstone4)
Intel Core i7-6600U CPU 2.60GHz (Max: 2.61GHz) (Skylake), 1 CPU, 4 logical and 2 physical cores
Frequency=2742184 Hz, Resolution=364.6728 ns, Timer=TSC
.NET Core SDK=2.1.500-preview-009266
  [Host] : .NET Core 2.1.3-servicing-26724-03 (CoreCLR 4.6.26724.06, CoreFX 4.6.26724.03), 64bit RyuJIT
  Clr    : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3163.0
  Core   : .NET Core 2.1.3-servicing-26724-03 (CoreCLR 4.6.26724.06, CoreFX 4.6.26724.03), 64bit RyuJIT

Method	Job	Runtime	Mean	Error	StdDev	Allocated
Benchmark	Clr	Clr	107.2 ms	2.101 ms	2.248 ms	16 KB
Benchmark	Core	Core	112.0 ms	2.220 ms	2.643 ms	13.31 KB

[jkotas]

I think it makes sense to revisit this topic when the collectible assemblies feature is implemented

Why do you think that collectible assemblies will help you with this? RegExCompiler is using dynamic methods today that are collectible just like collectible assemblies: https://github.com/dotnet/corefx/blob/master/src/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexLWCGCompiler.cs#L50

[ptupitsyn]

@ViktorHofer any insight on the root cause of the issue?

[NinoFloris]

Any news on this? Seems like something that should be better tunable than it is today?

[danmoseley]

@NinoFloris what are you hoping to tune? Right now you can adjust the size of the cache, and choose whether to compile or not. For the original report in this issue, it should be possible to return to the original performance by removing the compilation flag.

[danmoseley]

I had a quick check, and compiled path is indeed slower because of JITting. Although creating the regex (which generates the IL) may be out of the benchmarked path, the JITing occurs when the regex is first used and that is the most costly part of compilation. As pointed out the cache doesn't help because each of these 798 is only used once. In short, this pattern of creating large numbers of regexes only used once is not a good fit for RegexOptions.Compiled. The cost is not amortized.

We are faster in 5.0 than 2.1 though:

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.19541
Intel Core i7-2760QM CPU 2.40GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-alpha.1.20061.5
  [Host]     : .NET Core 2.1.14 (CoreCLR 4.6.28207.04, CoreFX 4.6.28208.01), X64 RyuJIT
  DefaultJob : .NET Core 2.1.14 (CoreCLR 4.6.28207.04, CoreFX 4.6.28208.01), X64 RyuJIT


|               Method |      Mean |    Error |   StdDev |
|--------------------- |----------:|---------:|---------:|
| BenchmarkNotCompiled |  30.42 ms | 0.310 ms | 0.275 ms |
|    BenchmarkCompiled | 125.00 ms | 4.205 ms | 6.790 ms |

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.19541
Intel Core i7-2760QM CPU 2.40GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-alpha.1.20061.5
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.6007, CoreFX 5.0.20.6007), X64 RyuJIT
  DefaultJob : .NET Core 5.0.0 (CoreCLR 5.0.20.6007, CoreFX 5.0.20.6007), X64 RyuJIT


|               Method |     Mean |    Error |   StdDev |
|--------------------- |---------:|---------:|---------:|
| BenchmarkNotCompiled | 29.21 ms | 0.127 ms | 0.112 ms |
|    BenchmarkCompiled | 73.43 ms | 0.377 ms | 0.315 ms |

cc @stephentoub