Regression impacting ASP.NET MVC

[sebastienros]

Testing on a fixed aspnet version and varying dotnet runtime, we see a 6% regression on the MVC Plaintext scenario between these versions:
2.1.0-preview2-26324-02 https://github.com/dotnet/corefx/tree/23165cee77ae4dd7a7645b7927872948bb27a73e
and
2.1.0-preview2-26316-06 https://github.com/dotnet/corefx/tree/8130620b542457ca4e72eb2bc81c3eae50192a33

A diff on perfview traces for these two versions resulted in this:

The two full traces are available if necessary.

[sebastienros]

/cc @mkArtakMSFT

[davidfowl]

Hmm, nothing changed in pipelines namespace during those commits.

[davidfowl]

This commit was made to System.Memory in that timespan dotnet/corefx@6cc11f5.

[stephentoub]

Hmm, nothing changed in pipelines namespace during those commits.

I'm having trouble understanding GitHub's UI for commit diffs, but in the comparison between the commits:
dotnet/corefx@23165ce...8130620
it does include this one:
dotnet/corefx@573f9e5
which is a merge from master to release/2.1. Maybe that brought with it pipelines changes?

[pakrym]

Strange, dotnet/corefx@573f9e5 doesn't contain pipe changes it shows entire pipe source code as new.

[pakrym]

Also, perfview diff is slightly incorrect as it failed to recognize two casings of System.IO.PIpelines names and shows same functions multiple times.

[sebastienros]

You can get the two traces here: \scratch2\scratch\sebros\investigation-mvcplaintext-3.27

[joshfree]

Strange, 573f9e5 doesn't contain pipe changes it shows entire pipe source code as new.

I believe that's the initial Preview2 integration from corefx:master to corefx:/release/2.1; prior to that (preview1) pipelines was coming from https://github.com/dotnet/corefxlab/tree/release/2.1

[sebastienros]

Sharing the commands I used as I have been asked

PS benchmarks\src\BenchmarksDriver> dotnet run -c Debug --server http://asp-perf-win:5001 --client http://asp-perf-load:5002 `
>> -j ..\Benchmarks\benchmarks.plaintext.json -n MvcPlaintext --webHost KestrelSockets `
>> --aspnetCoreVersion 2.1.0-preview3-32057 --warmup 3 --duration 10 `
>> --runtimeVersion 2.1.0-preview2-26314-02 --collect-trace --trace-arguments Providers=.NETTasks:0:0

and

PS benchmarks\src\BenchmarksDriver> dotnet run -c Debug --server http://asp-perf-win:5001 --client http://asp-perf-load:5002 `
>> -j ..\Benchmarks\benchmarks.plaintext.json -n MvcPlaintext --webHost KestrelSockets `
>> --aspnetCoreVersion 2.1.0-preview3-32057 --warmup 3 --duration 10 `
>> --runtimeVersion 2.1.0-preview2-26316-06 --collect-trace --trace-arguments Providers=.NETTasks:0:0

[pakrym]

I did a bit more digging. Ran benchmarks with "Providers:.NETFramework:0:0,.NETTasks:0:0,*Microsoft-Diagnostics-DiagnosticSource:0:0;BufferSizeMB=1024;CircularMB=1024" to remove all diagnostics source noice.

It also normalized module names so the current diff looks like (\\scratch2\scratch\sebros\investigation-mvcplaintext-3.27\pakrym-no-ds):

Pipelines assembly that is used in both runs is the same 4.6.26313.01 because it's not in shared runtime and is referenced as a package by kestrel.

Module lists:

26314-02

ProcessName	ProcessID	Name	FileVersion	Commit	BuildTime	FilePath
dotnet	5080	mysqlconnector	0.38.0.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\mysqlconnector.dll
dotnet	5080	benchmarks	1.0.0.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\benchmarks.dll
dotnet	5080	cryptbase	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\cryptbase.dll
dotnet	5080	cryptsp	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\cryptsp.dll
dotnet	5080	imm32	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\imm32.dll
dotnet	5080	kernel.appcore	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\kernel.appcore.dll
dotnet	5080	msasn1	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\msasn1.dll
dotnet	5080	mswsock	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\mswsock.dll
dotnet	5080	napinsp	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\napinsp.dll
dotnet	5080	ncrypt	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\ncrypt.dll
dotnet	5080	nsi	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\nsi.dll
dotnet	5080	ntasn1	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\ntasn1.dll
dotnet	5080	profapi	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\profapi.dll
dotnet	5080	rsaenh	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\rsaenh.dll
dotnet	5080	shlwapi	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\shlwapi.dll
dotnet	5080	version	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\version.dll
dotnet	5080	winrnr	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\winrnr.dll
dotnet	5080	crypt32	10.0.14393.1198 (rs1_release_sec.170427-1353)	###########	c:\windows\system32\crypt32.dll
dotnet	5080	kernel32	10.0.14393.1198 (rs1_release_sec.170427-1353)	###########	c:\windows\system32\kernel32.dll
dotnet	5080	oleaut32	10.0.14393.1378 (rs1_release.170620-2008)	###########	c:\windows\system32\oleaut32.dll
dotnet	5080	sechost	10.0.14393.1613 (rs1_release_d.170807-1806)	###########	c:\windows\system32\sechost.dll
dotnet	5080	ntdll	10.0.14393.1715 (rs1_release_inmarket.170906-1810)	###########	c:\windows\system32\ntdll.dll
dotnet	5080	dnsapi	10.0.14393.1770 (rs1_release.170917-1700)	###########	c:\windows\system32\dnsapi.dll
dotnet	5080	gdi32full	10.0.14393.1770 (rs1_release.170917-1700)	###########	c:\windows\system32\gdi32full.dll
dotnet	5080	user32	10.0.14393.1770 (rs1_release.170917-1700)	###########	c:\windows\system32\user32.dll
dotnet	5080	advapi32	10.0.14393.2007 (rs1_release.171231-1800)	###########	c:\windows\system32\advapi32.dll
dotnet	5080	ncryptprov	10.0.14393.2007 (rs1_release.171231-1800)	###########	c:\windows\system32\ncryptprov.dll
dotnet	5080	nlaapi	10.0.14393.2007 (rs1_release.171231-1800)	###########	c:\windows\system32\nlaapi.dll
dotnet	5080	gdi32	10.0.14393.206 (rs1_release.160915-0644)	###########	c:\windows\system32\gdi32.dll
dotnet	5080	ws2_32	10.0.14393.206 (rs1_release.160915-0644)	###########	c:\windows\system32\ws2_32.dll
dotnet	5080	ole32	10.0.14393.2068 (rs1_release.180209-1727)	###########	c:\windows\system32\ole32.dll
dotnet	5080	bcryptprimitives	10.0.14393.2097 (rs1_release_1.180212-1105)	###########	c:\windows\system32\bcryptprimitives.dll
dotnet	5080	msvcp_win	10.0.14393.2097 (rs1_release_1.180212-1105)	###########	c:\windows\system32\msvcp_win.dll
dotnet	5080	ucrtbase	10.0.14393.2097 (rs1_release_1.180212-1105)	###########	c:\windows\system32\ucrtbase.dll
dotnet	5080	combase	10.0.14393.2125 (rs1_release.180301-2139)	###########	c:\windows\system32\combase.dll
dotnet	5080	kernelbase	10.0.14393.2125 (rs1_release.180301-2139)	###########	c:\windows\system32\kernelbase.dll
dotnet	5080	rpcrt4	10.0.14393.2125 (rs1_release.180301-2139)	###########	c:\windows\system32\rpcrt4.dll
dotnet	5080	win32u	10.0.14393.51 (rs1_release_inmarket.160801-1836)	###########	c:\windows\system32\win32u.dll
dotnet	5080	bcrypt	10.0.14393.576 (rs1_release_inmarket.161208-2252)	###########	c:\windows\system32\bcrypt.dll
dotnet	5080	iphlpapi	10.0.14393.953 (rs1_release_inmarket.170303-1614)	###########	c:\windows\system32\iphlpapi.dll
dotnet	5080	microsoft.extensions.dependencymodel	2.1.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.dependencymodel.dll
dotnet	5080	microsoft.aspnetcore.connections.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.connections.abstractions.dll
dotnet	5080	microsoft.aspnetcore.diagnostics	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.diagnostics.dll
dotnet	5080	microsoft.aspnetcore.hosting	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.hosting.dll
dotnet	5080	microsoft.aspnetcore.hosting.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.hosting.abstractions.dll
dotnet	5080	microsoft.aspnetcore.hosting.server.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.hosting.server.abstractions.dll
dotnet	5080	microsoft.aspnetcore.http	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.http.dll
dotnet	5080	microsoft.aspnetcore.http.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.http.abstractions.dll
dotnet	5080	microsoft.aspnetcore.http.features	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.http.features.dll
dotnet	5080	microsoft.aspnetcore.mvc.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.mvc.abstractions.dll
dotnet	5080	microsoft.aspnetcore.mvc.core	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.mvc.core.dll
dotnet	5080	microsoft.aspnetcore.mvc.formatters.json	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.mvc.formatters.json.dll
dotnet	5080	microsoft.aspnetcore.mvc.razor	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.mvc.razor.dll
dotnet	5080	microsoft.aspnetcore.mvc.viewfeatures	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.mvc.viewfeatures.dll
dotnet	5080	microsoft.aspnetcore.razor	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.razor.dll
dotnet	5080	microsoft.aspnetcore.responsecaching	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.responsecaching.dll
dotnet	5080	microsoft.aspnetcore.routing	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.routing.dll
dotnet	5080	microsoft.aspnetcore.routing.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.routing.abstractions.dll
dotnet	5080	microsoft.aspnetcore.server.httpsys	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.httpsys.dll
dotnet	5080	microsoft.aspnetcore.server.iisintegration	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.iisintegration.dll
dotnet	5080	microsoft.aspnetcore.server.kestrel	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.kestrel.dll
dotnet	5080	microsoft.aspnetcore.server.kestrel.core	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.kestrel.core.dll
dotnet	5080	microsoft.aspnetcore.server.kestrel.transport.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.kestrel.transport.abstractions.dll
dotnet	5080	microsoft.aspnetcore.server.kestrel.transport.libuv	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.kestrel.transport.libuv.dll
dotnet	5080	microsoft.aspnetcore.server.kestrel.transport.sockets	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.server.kestrel.transport.sockets.dll
dotnet	5080	microsoft.aspnetcore.staticfiles	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.staticfiles.dll
dotnet	5080	microsoft.aspnetcore.webutilities	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.aspnetcore.webutilities.dll
dotnet	5080	microsoft.entityframeworkcore	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.entityframeworkcore.dll
dotnet	5080	microsoft.entityframeworkcore.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.entityframeworkcore.abstractions.dll
dotnet	5080	microsoft.entityframeworkcore.relational	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.entityframeworkcore.relational.dll
dotnet	5080	microsoft.entityframeworkcore.sqlserver	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.entityframeworkcore.sqlserver.dll
dotnet	5080	microsoft.extensions.caching.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.caching.abstractions.dll
dotnet	5080	microsoft.extensions.caching.memory	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.caching.memory.dll
dotnet	5080	microsoft.extensions.configuration	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.dll
dotnet	5080	microsoft.extensions.configuration.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.abstractions.dll
dotnet	5080	microsoft.extensions.configuration.binder	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.binder.dll
dotnet	5080	microsoft.extensions.configuration.commandline	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.commandline.dll
dotnet	5080	microsoft.extensions.configuration.environmentvariables	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.environmentvariables.dll
dotnet	5080	microsoft.extensions.configuration.fileextensions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.fileextensions.dll
dotnet	5080	microsoft.extensions.configuration.json	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.configuration.json.dll
dotnet	5080	microsoft.extensions.dependencyinjection	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.dependencyinjection.dll
dotnet	5080	microsoft.extensions.dependencyinjection.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.dependencyinjection.abstractions.dll
dotnet	5080	microsoft.extensions.fileproviders.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.fileproviders.abstractions.dll
dotnet	5080	microsoft.extensions.fileproviders.physical	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.fileproviders.physical.dll
dotnet	5080	microsoft.extensions.filesystemglobbing	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.filesystemglobbing.dll
dotnet	5080	microsoft.extensions.hosting.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.hosting.abstractions.dll
dotnet	5080	microsoft.extensions.logging	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.logging.dll
dotnet	5080	microsoft.extensions.logging.abstractions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.logging.abstractions.dll
dotnet	5080	microsoft.extensions.logging.console	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.logging.console.dll
dotnet	5080	microsoft.extensions.objectpool	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.objectpool.dll
dotnet	5080	microsoft.extensions.options	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.options.dll
dotnet	5080	microsoft.extensions.options.configurationextensions	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.options.configurationextensions.dll
dotnet	5080	microsoft.extensions.primitives	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.primitives.dll
dotnet	5080	microsoft.extensions.webencoders	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.extensions.webencoders.dll
dotnet	5080	microsoft.net.http.headers	2.1.0.18086		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\microsoft.net.http.headers.dll
dotnet	5080	hostpolicy	2.1.26314.02 @BuiltBy: dlab-DDVSOWINAGE018 @Branch: HEAD @srccode: https://github.com/dotnet/core-setup/tree/8ac5f4fa222b9676ff74362878d02e02f07cd6ac. Commit Hash: 8ac5f4f	8ac5f4f	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\hostpolicy.dll
dotnet	5080	dotnet	2.1.26316.06 @BuiltBy: dlab-DDVSOWINAGE031 @Branch: HEAD @srccode: https://github.com/dotnet/core-setup/tree/e8d248c21937da58626ef828f6e55abfa22ca8a1. Commit Hash: e8d248c	e8d248c	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\dotnet.exe
dotnet	5080	hostfxr	2.1.26405.02 @BuiltBy: dlab-DDVSOWINAGE029 @Branch: master @srccode: https://github.com/dotnet/core-setup/tree/a876b38f2e6048ac6f4f968e6c39757986337a96. Commit Hash: a876b38	a876b38	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\host\fxr\2.1.0-preview3-26405-02\hostfxr.dll
dotnet	5080	remotion.linq	2.2.0.30000		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\remotion.linq.dll
dotnet	5080	mongodb.bson	2.5.0.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\mongodb.bson.dll
dotnet	5080	mongodb.driver	2.5.0.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\mongodb.driver.dll
dotnet	5080	mongodb.driver.core	2.5.0.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\mongodb.driver.core.dll
dotnet	5080	clbcatq	2001.12.10941.16384 (rs1_release.160715-1616)	###########	c:\windows\system32\clbcatq.dll
dotnet	5080	system.interactive.async	3.1.1.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\system.interactive.async.dll
dotnet	5080	npgsql	3.2.7.0		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\npgsql.dll
dotnet	5080	system.private.corelib	4.6.26310.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.private.corelib.dll
dotnet	5080	clrjit	4.6.26310.01 @BuiltBy: dlab-DDVSOWINAGE022 @Branch: HEAD @srccode: https://github.com/dotnet/coreclr/tree/c3aef92c137658cebb111da6c856ab653316fb0a	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\clrjit.dll
dotnet	5080	coreclr	4.6.26310.01 @BuiltBy: dlab-DDVSOWINAGE022 @Branch: HEAD @srccode: https://github.com/dotnet/coreclr/tree/c3aef92c137658cebb111da6c856ab653316fb0a	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\coreclr.dll
dotnet	5080	microsoft.win32.primitives	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\microsoft.win32.primitives.dll
dotnet	5080	system.appcontext	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.appcontext.dll
dotnet	5080	system.buffers	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.buffers.dll
dotnet	5080	system.collections	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.collections.dll
dotnet	5080	system.collections.concurrent	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.collections.concurrent.dll
dotnet	5080	system.collections.nongeneric	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.collections.nongeneric.dll
dotnet	5080	system.collections.specialized	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.collections.specialized.dll
dotnet	5080	system.componentmodel	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.componentmodel.dll
dotnet	5080	system.componentmodel.annotations	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.componentmodel.annotations.dll
dotnet	5080	system.componentmodel.primitives	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.componentmodel.primitives.dll
dotnet	5080	system.componentmodel.typeconverter	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.componentmodel.typeconverter.dll
dotnet	5080	system.console	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.console.dll
dotnet	5080	system.data.common	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.data.common.dll
dotnet	5080	system.data.sqlclient	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\runtimes\win\lib\netstandard2.0\system.data.sqlclient.dll
dotnet	5080	system.diagnostics.diagnosticsource	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.diagnostics.diagnosticsource.dll
dotnet	5080	system.diagnostics.tracing	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.diagnostics.tracing.dll
dotnet	5080	system.drawing.primitives	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.drawing.primitives.dll
dotnet	5080	system.io	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.io.dll
dotnet	5080	system.io.filesystem	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.io.filesystem.dll
dotnet	5080	system.io.filesystem.primitives	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.io.filesystem.primitives.dll
dotnet	5080	system.io.filesystem.watcher	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.io.filesystem.watcher.dll
dotnet	5080	system.io.pipelines	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\system.io.pipelines.dll
dotnet	5080	system.linq	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.linq.dll
dotnet	5080	system.linq.expressions	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.linq.expressions.dll
dotnet	5080	system.memory	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.memory.dll
dotnet	5080	system.net.nameresolution	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.net.nameresolution.dll
dotnet	5080	system.net.primitives	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.net.primitives.dll
dotnet	5080	system.net.sockets	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.net.sockets.dll
dotnet	5080	system.numerics.vectors	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.numerics.vectors.dll
dotnet	5080	system.objectmodel	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.objectmodel.dll
dotnet	5080	system.private.uri	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.private.uri.dll
dotnet	5080	system.reflection	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.reflection.dll
dotnet	5080	system.resources.resourcemanager	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.resources.resourcemanager.dll
dotnet	5080	system.runtime	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.runtime.dll
dotnet	5080	system.runtime.extensions	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.runtime.extensions.dll
dotnet	5080	system.runtime.interopservices	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.runtime.interopservices.dll
dotnet	5080	system.runtime.interopservices.runtimeinformation	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.runtime.interopservices.runtimeinformation.dll
dotnet	5080	system.runtime.numerics	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.runtime.numerics.dll
dotnet	5080	system.security.claims	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.claims.dll
dotnet	5080	system.security.cryptography.algorithms	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.cryptography.algorithms.dll
dotnet	5080	system.security.cryptography.cng	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.cryptography.cng.dll
dotnet	5080	system.security.cryptography.csp	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.cryptography.csp.dll
dotnet	5080	system.security.cryptography.encoding	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.cryptography.encoding.dll
dotnet	5080	system.security.cryptography.primitives	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.cryptography.primitives.dll
dotnet	5080	system.security.cryptography.x509certificates	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.cryptography.x509certificates.dll
dotnet	5080	system.security.principal	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.security.principal.dll
dotnet	5080	system.text.encoding.extensions	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.text.encoding.extensions.dll
dotnet	5080	system.text.encodings.web	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\kpxu2lqn.rbh\benchmarks\src\benchmarks\published\system.text.encodings.web.dll
dotnet	5080	system.threading	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.dll
dotnet	5080	system.threading.overlapped	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.overlapped.dll
dotnet	5080	system.threading.tasks	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.tasks.dll
dotnet	5080	system.threading.tasks.extensions	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.tasks.extensions.dll
dotnet	5080	system.threading.thread	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.thread.dll
dotnet	5080	system.threading.threadpool	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.threadpool.dll
dotnet	5080	system.threading.timer	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.threading.timer.dll
dotnet	5080	system.transactions.local	4.6.26313.01		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\system.transactions.local.dll
dotnet	5080	netstandard	4.6.26313.1		###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26314-02\netstandard.dll
dotnet	5080	msvcrt	7.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\msvcrt.dll

26316-04

dotnet	2084	advapi32	10.0.14393.2007 (rs1_release.171231-1800)	###########	c:\windows\system32\advapi32.dll
dotnet	2084	bcrypt	10.0.14393.576 (rs1_release_inmarket.161208-2252)	###########	c:\windows\system32\bcrypt.dll
dotnet	2084	bcryptprimitives	10.0.14393.2097 (rs1_release_1.180212-1105)	###########	c:\windows\system32\bcryptprimitives.dll
dotnet	2084	benchmarks	1.0.0.0		###########
dotnet	2084	clbcatq	2001.12.10941.16384 (rs1_release.160715-1616)	###########	c:\windows\system32\clbcatq.dll
dotnet	2084	clrjit	4.6.26316.03 @BuiltBy: dlab-DDVSOWINAGE001 @Branch: HEAD @srccode: https://github.com/dotnet/coreclr/tree/9d0e7065b788333c712a31d11fc46fe561887b66	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26316-06\clrjit.dll
dotnet	2084	combase	10.0.14393.2125 (rs1_release.180301-2139)	###########	c:\windows\system32\combase.dll
dotnet	2084	coreclr	4.6.26316.03 @BuiltBy: dlab-DDVSOWINAGE001 @Branch: HEAD @srccode: https://github.com/dotnet/coreclr/tree/9d0e7065b788333c712a31d11fc46fe561887b66	###########	c:\users\administrator\appdata\local\temp\tmp1f01.tmp\tezgmjim.fot\shared\microsoft.netcore.app\2.1.0-preview2-26316-06\coreclr.dll
dotnet	2084	crypt32	10.0.14393.1198 (rs1_release_sec.170427-1353)	###########	c:\windows\system32\crypt32.dll
dotnet	2084	cryptbase	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\cryptbase.dll
dotnet	2084	cryptsp	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\cryptsp.dll
dotnet	2084	dnsapi	10.0.14393.1770 (rs1_release.170917-1700)	###########	c:\windows\system32\dnsapi.dll
dotnet	2084	dotnet	2.1.26316.06 @BuiltBy: dlab-DDVSOWINAGE031 @Branch: HEAD @srccode: https://github.com/dotnet/core-setup/tree/e8d248c21937da58626ef828f6e55abfa22ca8a1. Commit Hash: e8d248c	e8d248c	###########
dotnet	2084	gdi32	10.0.14393.206 (rs1_release.160915-0644)	###########	c:\windows\system32\gdi32.dll
dotnet	2084	gdi32full	10.0.14393.1770 (rs1_release.170917-1700)	###########	c:\windows\system32\gdi32full.dll
dotnet	2084	hostfxr	2.1.26405.02 @BuiltBy: dlab-DDVSOWINAGE029 @Branch: master @srccode: https://github.com/dotnet/core-setup/tree/a876b38f2e6048ac6f4f968e6c39757986337a96. Commit Hash: a876b38	a876b38	###########
dotnet	2084	hostpolicy	2.1.26316.06 @BuiltBy: dlab-DDVSOWINAGE031 @Branch: HEAD @srccode: https://github.com/dotnet/core-setup/tree/e8d248c21937da58626ef828f6e55abfa22ca8a1. Commit Hash: e8d248c	e8d248c	###########
dotnet	2084	imm32	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\imm32.dll
dotnet	2084	iphlpapi	10.0.14393.953 (rs1_release_inmarket.170303-1614)	###########	c:\windows\system32\iphlpapi.dll
dotnet	2084	kernel.appcore	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\kernel.appcore.dll
dotnet	2084	kernel32	10.0.14393.1198 (rs1_release_sec.170427-1353)	###########	c:\windows\system32\kernel32.dll
dotnet	2084	kernelbase	10.0.14393.2125 (rs1_release.180301-2139)	###########	c:\windows\system32\kernelbase.dll
dotnet	2084	microsoft.aspnetcore.connections.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.diagnostics	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.hosting	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.hosting.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.hosting.server.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.http	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.http.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.http.features	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.mvc.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.mvc.core	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.mvc.formatters.json	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.mvc.razor	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.mvc.viewfeatures	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.razor	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.responsecaching	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.routing	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.routing.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.httpsys	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.iisintegration	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.kestrel	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.kestrel.core	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.kestrel.transport.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.kestrel.transport.libuv	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.server.kestrel.transport.sockets	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.staticfiles	2.1.0.18086		###########
dotnet	2084	microsoft.aspnetcore.webutilities	2.1.0.18086		###########
dotnet	2084	microsoft.entityframeworkcore	2.1.0.18086		###########
dotnet	2084	microsoft.entityframeworkcore.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.entityframeworkcore.relational	2.1.0.18086		###########
dotnet	2084	microsoft.entityframeworkcore.sqlserver	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.caching.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.caching.memory	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration.binder	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration.commandline	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration.environmentvariables	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration.fileextensions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.configuration.json	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.dependencyinjection	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.dependencyinjection.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.dependencymodel	2.1.0		###########
dotnet	2084	microsoft.extensions.fileproviders.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.fileproviders.physical	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.filesystemglobbing	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.hosting.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.logging	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.logging.abstractions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.logging.console	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.objectpool	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.options	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.options.configurationextensions	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.primitives	2.1.0.18086		###########
dotnet	2084	microsoft.extensions.webencoders	2.1.0.18086		###########
dotnet	2084	microsoft.net.http.headers	2.1.0.18086		###########
dotnet	2084	microsoft.win32.primitives	4.6.26316.03		###########
dotnet	2084	mongodb.bson	2.5.0.0		###########
dotnet	2084	mongodb.driver	2.5.0.0		###########
dotnet	2084	mongodb.driver.core	2.5.0.0		###########
dotnet	2084	msasn1	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\msasn1.dll
dotnet	2084	msvcp_win	10.0.14393.2097 (rs1_release_1.180212-1105)	###########	c:\windows\system32\msvcp_win.dll
dotnet	2084	msvcrt	7.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\msvcrt.dll
dotnet	2084	mswsock	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\mswsock.dll
dotnet	2084	mysqlconnector	0.38.0.0		###########
dotnet	2084	napinsp	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\napinsp.dll
dotnet	2084	ncrypt	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\ncrypt.dll
dotnet	2084	ncryptprov	10.0.14393.2007 (rs1_release.171231-1800)	###########	c:\windows\system32\ncryptprov.dll
dotnet	2084	netstandard	4.6.26316.3		###########
dotnet	2084	nlaapi	10.0.14393.2007 (rs1_release.171231-1800)	###########	c:\windows\system32\nlaapi.dll
dotnet	2084	npgsql	3.2.7.0		###########
dotnet	2084	nsi	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\nsi.dll
dotnet	2084	ntasn1	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\ntasn1.dll
dotnet	2084	ntdll	10.0.14393.1715 (rs1_release_inmarket.170906-1810)	###########	c:\windows\system32\ntdll.dll
dotnet	2084	ole32	10.0.14393.2068 (rs1_release.180209-1727)	###########	c:\windows\system32\ole32.dll
dotnet	2084	oleaut32	10.0.14393.1378 (rs1_release.170620-2008)	###########	c:\windows\system32\oleaut32.dll
dotnet	2084	profapi	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\profapi.dll
dotnet	2084	remotion.linq	2.2.0.30000		###########
dotnet	2084	rpcrt4	10.0.14393.2125 (rs1_release.180301-2139)	###########	c:\windows\system32\rpcrt4.dll
dotnet	2084	rsaenh	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\rsaenh.dll
dotnet	2084	sechost	10.0.14393.1613 (rs1_release_d.170807-1806)	###########	c:\windows\system32\sechost.dll
dotnet	2084	shlwapi	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\shlwapi.dll
dotnet	2084	system.appcontext	4.6.26316.03		###########
dotnet	2084	system.buffers	4.6.26316.03		###########
dotnet	2084	system.collections	4.6.26316.03		###########
dotnet	2084	system.collections.concurrent	4.6.26316.03		###########
dotnet	2084	system.collections.nongeneric	4.6.26316.03		###########
dotnet	2084	system.collections.specialized	4.6.26316.03		###########
dotnet	2084	system.componentmodel	4.6.26316.03		###########
dotnet	2084	system.componentmodel.annotations	4.6.26316.03		###########
dotnet	2084	system.componentmodel.primitives	4.6.26316.03		###########
dotnet	2084	system.componentmodel.typeconverter	4.6.26316.03		###########
dotnet	2084	system.console	4.6.26316.03		###########
dotnet	2084	system.data.common	4.6.26316.03		###########
dotnet	2084	system.data.sqlclient	4.6.26313.01		###########
dotnet	2084	system.diagnostics.diagnosticsource	4.6.26316.03		###########
dotnet	2084	system.diagnostics.tracing	4.6.26316.03		###########
dotnet	2084	system.drawing.primitives	4.6.26316.03		###########
dotnet	2084	system.interactive.async	3.1.1.0		###########
dotnet	2084	system.io	4.6.26316.03		###########
dotnet	2084	system.io.filesystem	4.6.26316.03		###########
dotnet	2084	system.io.filesystem.primitives	4.6.26316.03		###########
dotnet	2084	system.io.filesystem.watcher	4.6.26316.03		###########
dotnet	2084	system.io.pipelines	4.6.26313.01		###########
dotnet	2084	system.linq	4.6.26316.03		###########
dotnet	2084	system.linq.expressions	4.6.26316.03		###########
dotnet	2084	system.memory	4.6.26316.03		###########
dotnet	2084	system.net.nameresolution	4.6.26316.03		###########
dotnet	2084	system.net.primitives	4.6.26316.03		###########
dotnet	2084	system.net.sockets	4.6.26316.03		###########
dotnet	2084	system.numerics.vectors	4.6.26316.03		###########
dotnet	2084	system.objectmodel	4.6.26316.03		###########
dotnet	2084	system.private.corelib	4.6.26316.03		###########
dotnet	2084	system.private.uri	4.6.26316.03		###########
dotnet	2084	system.reflection	4.6.26316.03		###########
dotnet	2084	system.resources.resourcemanager	4.6.26316.03		###########
dotnet	2084	system.runtime	4.6.26316.03		###########
dotnet	2084	system.runtime.extensions	4.6.26316.03		###########
dotnet	2084	system.runtime.interopservices	4.6.26316.03		###########
dotnet	2084	system.runtime.interopservices.runtimeinformation	4.6.26316.03		###########
dotnet	2084	system.runtime.numerics	4.6.26316.03		###########
dotnet	2084	system.security.claims	4.6.26316.03		###########
dotnet	2084	system.security.cryptography.algorithms	4.6.26316.03		###########
dotnet	2084	system.security.cryptography.cng	4.6.26316.03		###########
dotnet	2084	system.security.cryptography.csp	4.6.26316.03		###########
dotnet	2084	system.security.cryptography.encoding	4.6.26316.03		###########
dotnet	2084	system.security.cryptography.primitives	4.6.26316.03		###########
dotnet	2084	system.security.cryptography.x509certificates	4.6.26316.03		###########
dotnet	2084	system.security.principal	4.6.26316.03		###########
dotnet	2084	system.text.encoding.extensions	4.6.26316.03		###########
dotnet	2084	system.text.encodings.web	4.6.26313.01		###########
dotnet	2084	system.threading	4.6.26316.03		###########
dotnet	2084	system.threading.overlapped	4.6.26316.03		###########
dotnet	2084	system.threading.tasks	4.6.26316.03		###########
dotnet	2084	system.threading.tasks.extensions	4.6.26316.03		###########
dotnet	2084	system.threading.thread	4.6.26316.03		###########
dotnet	2084	system.threading.threadpool	4.6.26316.03		###########
dotnet	2084	system.threading.timer	4.6.26316.03		###########
dotnet	2084	system.transactions.local	4.6.26316.03		###########
dotnet	2084	ucrtbase	10.0.14393.2097 (rs1_release_1.180212-1105)	###########	c:\windows\system32\ucrtbase.dll
dotnet	2084	user32	10.0.14393.1770 (rs1_release.170917-1700)	###########	c:\windows\system32\user32.dll
dotnet	2084	version	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\version.dll
dotnet	2084	win32u	10.0.14393.51 (rs1_release_inmarket.160801-1836)	###########	c:\windows\system32\win32u.dll
dotnet	2084	winrnr	10.0.14393.0 (rs1_release.160715-1616)	###########	c:\windows\system32\winrnr.dll
dotnet	2084	ws2_32	10.0.14393.206 (rs1_release.160915-0644)	###########	c:\windows\system32\ws2_32.dll

[joshfree]

@brianrob @valenis could you help nudge this 2.1 performance investigation forward?

[sebastienros]

I isolated the regression in System.Private.Corelib.dll, should we move the issue to coreclr?

[tarekgh]

CC @jkotas

[brianrob]

FYI, I have the traces from @sebastienros and am investigating.

[brianrob]

It looks like there is a 2.5 second CPU regression here that shows up in System.Private.CoreLib:

It looks like it stems from a dictionary lookup in ASP.NET routing. There is more CPU time spent exclusively in these methods (the number on the far right is the exclusive CPU time in milliseconds):

Interestingly, when I look at these two commits / versions that you called out as the regression / baseline versions, they both consume the same version of CoreCLR:

2.1.0-preview2-26324-02 https://github.com/dotnet/corefx/tree/23165cee77ae4dd7a7645b7927872948bb27a73e
and
2.1.0-preview2-26316-06 https://github.com/dotnet/corefx/tree/8130620b542457ca4e72eb2bc81c3eae50192a33

Thus, I am not quite sure how this can happen. @sebastienros, can you confirm that these regression / baseline commits are valid?

[sebastienros]

Yes, I also found out about the Dictionary regression, and found some commits around that time. It makes totally sense as the Router is only used in MVC and a big client for dictionary lookups.

[stephentoub]

Was this regression only on Windows or also on Linux?

[sebastienros]

To be honest I don't know anyone that can correlate the commit numbers (that I got from the assemblies themselves) and what code is in it. It's due to how merges are done in corefx, no rebase/squash, and code copies. The git history is a maze ... it makes it very hard to get meaningful code diffs.

A simple way would be to decompile the two dlls I am testing, which I will do asap.

[stephentoub]

(I'm wondering if it may have been somehow caused by my changes in dotnet/coreclr#17391. Having the decompiled diff would be helpful.)

[sebastienros]

It's only visible on Windows

[stephentoub]

Ah, thanks, definitely not dotnet/coreclr#17391 then, as that wasn't merged until after the drop. Maybe dotnet/coreclr#16945; just speculating based on what Brian shared.

[stephentoub]

I shared Brian's question, though. Both of the linked versions are depending on the exact same corelib package:
https://github.com/dotnet/corefx/blob/23165cee77ae4dd7a7645b7927872948bb27a73e/dependencies.props#L35-L36
https://github.com/dotnet/corefx/blob/8130620b542457ca4e72eb2bc81c3eae50192a33/dependencies.props#L35-L36

[stephentoub]

I think it's very likely it's due to dotnet/coreclr#16945. Trying to repro now...

[brianrob]

Dug into the trace a bit more and also spoke with @stephentoub. Here are some more stacks that help point at the difference.

Regressed Case:

Baseline Case:

A good chunk of the regression comes from System.Globalization.CompareInfo.GetIgnoreCaseHash, which was introduced in dotnet/coreclr#16945.

[tarekgh]

CC @ViktorHofer @danmosemsft

[stephentoub]

This was due to the change to OrdinalIgnoreCase to add in necessary functionality. While that change did decrease perf build-over-build, even with the impact of that one change, .NET Core 2.1’s OrdinalIgnoreCase is still 4-5x faster than it was in .NET Core 2.0 (other changes after this also helped). Closing as by design.