Switch to BenchmarkDotNet and new csproj format

[lahma]

I don't know how active this comparison work is, but I'm creating this PR to show the changes. Might benefit or not. Now running each engine as benchmark that records times with memory allocations. BenchmarkDotNet gives all the goodies like warmup etc for more reliable results. Only downside is that now each engine has separate result file, one need to copy&paste from HTML or CSV to build same kind of comparison you have in your blog.

I'm doing some performance work in Jint so this helps by allowing running the Jint suite with different versions. Easiest way to test the new functionality is to run dotnet run -c Release in project folder and optionally adding all parameter to run all instead of prompt.

[rushfrisby]

Sounds good to me. I'll take a look at what's in the PR this weekend.

[lahma]

Cool, forgot to show example output so here it is, taken from JintBenchmark GitHub flavored output file.

BenchmarkDotNet=v0.10.11, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)
Processor=Intel Core i7-6820HQ CPU 2.70GHz (Skylake), ProcessorCount=8
Frequency=2648435 Hz, Resolution=377.5815 ns, Timer=TSC
  [Host]     : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2600.0
  Job-ZRUPTL : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2600.0

InvocationCount=4  LaunchCount=1  TargetCount=3  
UnrollFactor=4  WarmupCount=3  

Method	FileName	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
Run	d3.min.js	76.847 ms	6.475 ms	0.3658 ms	4250.0000	1250.0000	250.0000	23.99 MB
Run	dromaeo-3d-cube.js	113.220 ms	2.128 ms	0.1202 ms	7750.0000	250.0000	-	31.79 MB
Run	dromaeo-core-eval.js	27.411 ms	1.509 ms	0.0852 ms	1500.0000	-	-	6.92 MB
Run	dromaeo-object-array.js	299.209 ms	3.654 ms	0.2064 ms	44750.0000	1750.0000	500.0000	186.88 MB
Run	dromaeo-object-regexp.js	1,111.224 ms	187.994 ms	10.6220 ms	65333.3333	26333.3333	15250.0000	481.33 MB
Run	dromaeo-object-string.js	1,292.309 ms	14.275 ms	0.8066 ms	157666.6667	91333.3333	86583.3333	1461.85 MB
Run	dromaeo-string-base64.js	282.852 ms	13.666 ms	0.7721 ms	16500.0000	250.0000	-	67.53 MB
Run	handlebars-v4.0.5.js	22.403 ms	1.488 ms	0.0841 ms	1250.0000	250.0000	-	8.41 MB
Run	knockout-3.4.0.js	15.576 ms	2.729 ms	0.1542 ms	1000.0000	250.0000	-	5.83 MB
Run	lodash.min.js	30.666 ms	24.788 ms	1.4006 ms	1750.0000	500.0000	-	10.36 MB
Run	qunit-2.0.1.js	8.585 ms	1.087 ms	0.0614 ms	750.0000	250.0000	-	3.94 MB
Run	stopwatch.js	2,009.370 ms	139.832 ms	7.9008 ms	123000.0000	-	-	492.86 MB
Run	sunspider-3d-morph.js	92.695 ms	1.599 ms	0.0903 ms	4500.0000	250.0000	-	19.58 MB
Run	sunspider-3d-raytrace.js	179.195 ms	3.548 ms	0.2005 ms	13500.0000	250.0000	-	56.88 MB
Run	sunspider-access-binary-trees.js	110.672 ms	2.903 ms	0.1640 ms	15000.0000	250.0000	-	60.85 MB
Run	sunspider-access-fannkuch.js	293.107 ms	30.272 ms	1.7105 ms	17750.0000	-	-	71.08 MB
Run	sunspider-access-nbody.js	162.629 ms	2.730 ms	0.1542 ms	11500.0000	250.0000	-	46.35 MB
Run	sunspider-access-nsieve.js	454.588 ms	8.138 ms	0.4598 ms	26500.0000	4750.0000	1250.0000	109.34 MB
Run	sunspider-bitops-3bit-bits-in-byte.js	287.820 ms	2.662 ms	0.1504 ms	27000.0000	-	-	108.99 MB
Run	sunspider-bitops-bits-in-byte.js	282.639 ms	4.260 ms	0.2407 ms	22750.0000	-	-	91.87 MB
Run	sunspider-bitops-bitwise-and.js	59.649 ms	9.249 ms	0.5226 ms	4250.0000	-	-	17.88 MB
Run	sunspider-bitops-nsieve-bits.js	256.511 ms	4.166 ms	0.2354 ms	15000.0000	-	-	60.19 MB
Run	sunspider-controlflow-recursive.js	380.519 ms	7.343 ms	0.4149 ms	58500.0000	-	-	234.18 MB
Run	sunspider-crypto-aes.js	718.301 ms	103.669 ms	5.8575 ms	44500.0000	250.0000	-	180.05 MB
Run	sunspider-crypto-md5.js	113.627 ms	29.484 ms	1.6659 ms	13000.0000	250.0000	-	53.13 MB
Run	sunspider-crypto-sha1.js	115.935 ms	40.692 ms	2.2992 ms	12500.0000	250.0000	-	50.2 MB
Run	sunspider-date-format-tofte.js	49.055 ms	19.937 ms	1.1265 ms	3750.0000	250.0000	-	15.3 MB
Run	sunspider-date-format-xparb.js	41.580 ms	48.973 ms	2.7671 ms	2000.0000	250.0000	-	8.19 MB
Run	sunspider-math-cordic.js	127.661 ms	32.116 ms	1.8146 ms	9750.0000	-	-	39.9 MB
Run	sunspider-math-partial-sums.js	31.510 ms	2.654 ms	0.1500 ms	1500.0000	-	-	6.94 MB
Run	sunspider-math-spectral-norm.js	132.727 ms	46.389 ms	2.6211 ms	11750.0000	-	-	47.41 MB
Run	sunspider-regexp-dna.js	318.276 ms	46.025 ms	2.6005 ms	1000.0000	500.0000	500.0000	11.42 MB
Run	sunspider-string-fasta.js	179.489 ms	27.271 ms	1.5408 ms	15250.0000	-	-	61.27 MB
Run	sunspider-string-tagcloud.js	77.260 ms	3.256 ms	0.1840 ms	7750.0000	750.0000	-	35.15 MB
Run	sunspider-string-unpack-code.js	111.175 ms	76.693 ms	4.3333 ms	14500.0000	3500.0000	750.0000	62.14 MB
Run	sunspider-string-validate-input.js	113.846 ms	3.175 ms	0.1794 ms	8250.0000	750.0000	-	34.96 MB
Run	underscore-min.js	5.164 ms	4.194 ms	0.2370 ms	250.0000	-	-	2.26 MB
Run	v8-crypto.js	6,041.613 ms	198.248 ms	11.2014 ms	395000.0000	750.0000	-	1582.95 MB
Run	v8-deltablue.js	177.181 ms	9.592 ms	0.5420 ms	23000.0000	250.0000	-	95.65 MB
Run	v8-earley-boyer.js	3,666.232 ms	5,538.255 ms	312.9217 ms	272333.3333	12666.6667	2750.0000	1353.47 MB
Run	v8-raytrace.js	836.401 ms	216.073 ms	12.2085 ms	89750.0000	250.0000	-	363.2 MB
Run	v8-richards.js	113.451 ms	3.380 ms	0.1910 ms	11750.0000	250.0000	-	47.04 MB

[rushfrisby]

Getting some compiler errors on JintBenchmark. Missing a reference for Esprima.

I'm not familiar with the Esprima project. Did @sebastienros abandon Jint and replace it with a .NET port of Esprima to support a newer ECMA spec? Are both still being worked on? Should there be tests for both here?

[sebastienros]

Ah, the package version is on myget only I assume, will push to nuget since we now it's stable.

@rushfrisby Esprima is just the parser part. Jint is the interpreter. I did the split since people might just care about reading/analyzing JS.

[rushfrisby]

Cool - I'm looking forward to being able to run this!

[lahma]

The using directive might be redundant. I tested 3.x series too which has esprima. 2.x should not need it.

[lahma]

@rushfrisby I have added a commit that upgrades to fresh Jint 3.0.0-beta-1101 that was released to public NuGet just hours ago. I also updated the table above to reflect results from the latest release.

/cc @sebastienros

[lahma]

@rushfrisby I've force-pushed an update for dependencies, should have the latest and greatest. Some nice improvements on Jint side too. I hope you have some in the future to check it out, maybe even give us an update in the form of a blog post. Cheers!

[rushfrisby]

dang, thought I had merged this! I'll do it now. I'll work on an updated blog post - I get a few requests/yr to update.