Create initial benchmark.

[nojaf]

Following up on #867, a first attempt to create a benchmark for the largest file in Fantomas.

So this works on my machine ™, how do we want to run this on CI?
It took around 30 min to run so I'm not sure we want to have this for each PR build.

[jindraivanek]

works on my machine ™

Interesting, I got
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\dev\github\fantomas\src\Fantomas.Benchmarks\bin\Release\netcoreapp3.1\c1cfcd54-1773-4652-b944-12f39eec3430\bin\Release\netcoreapp3.1\paket-files\fsprojects\fantomas\src\Fantomas\CodePrinter.fs'.

let path = __SOURCE_DIRECTORY__ + "/../../paket-files/fsprojects/fantomas/src/Fantomas/CodePrinter.fs" fixes it.

[nojaf]

works on my machine ™

@jindraivanek, hmm yeah that might have been a lie there 🙈

I'm thinking about pushing the results of the benchmark to Azure, in Application Insights.
That might be an easy way to dump the result and query the progress.

[baronfel]

I found a github action that does easy performance comparisons over time for perf tests: https://github.com/rhysd/github-action-benchmark

It doesn't currently have support for benchmark.net, but it looks very easy to add support. We'd just have to PR a benchmark.net-specific parsing function for the generated files right here. I mostly am pointing this out because I see that your current CI setup is github-actions-centric.

[baronfel]

Regarding the overall speed of running the tests, I'd say the most impactful things to do would be to lower the amount of warming up/pre-work measurement that's done. Benchmark.net actually does a ton of additional work before running your tests to determine what sorts of jitter/constant factors to account for in your tests. Then, once the warmup is done it does your tests a large number of times.

Therefore you can speed up the iterations by

• reducing the warmup count
• reducing the number of times your tests are run to get averages.

[7sharp9]

I think you can tweak the attributes so that not as many tests are done so the CI would run faster. the memory one does do a lot of iterations.

[nojaf]

CI now runs between nine and elven minutes. Results are stored in Azure Table Storage.
I'm merging to master to see if the data gets pushed (Github actions doesn't allow usage of secrets on PR).