Test benchmark theory

[hubertp]

There is a minor but consistent uptick in our benchmarks. Testing Pavel's theory that it is due to logging.

The slowdown of VectorBenchmarks occurred after #8620:

[Akirathan]

I have removed the "Ready to merge" label. Please, let's first run benchmarks on this PR manually and compare it with the results on develop before merging. The only case when merging this change to develop eagerly makes sense is if we wanted to observe the VectorBenchmarks for a longer time. But even then, it is possible to manually schedule benchmarks on this PR for multiple times and compare all the results to the results from develop.

TLDR; Let's not be hasty with the merge. Let's first make sure that we understand what exactly causes the slowdown.

[Akirathan]

Engine benchmarks manually scheduled in https://github.com/enso-org/enso/actions/runs/7400803887

GitHub

Benchmark Engine · enso-org/enso@8b2a8d7

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

[JaroslavTulach]

The PR description says:

Testing Pavel's theory that it is due to logging.

How is the logging related to this change?

[Akirathan]

Because the parameter v is unused, and if there is no logger, i.e., NopLoggingProvider, no warning is produced, but if there is org.enso.logger.TestLogProvider, a warning is produced to the stderr. That is just a theory. Generally, I think that the size of the data for this particular benchmark is too small, the average time of a single iteration is 0.0017 ms.

[Akirathan]

Looking into other engine benchmarks, I noticed there are several other slowdowns after #8620:

• VectorBenchmarks.
• org_enso_interpreter_bench_benchmarks_semantic_StringBenchmarks_lengthOfStrings
• TypePatternBenchmarks.

All these benchmarks are engine benchmarks, they are small - they have a very short duration of an iteration, under 1 ms. These benchmarks are also pretty stable, and after #8620, we can see at least +10% difference that is stable as well.

I have an alternative theory. Apart from changing the log providers, I have also introduced module patching (options like --patch-module, --add-exports, etc.) so that the runtime.jar fat jar does not have to be reassembled. I might have been too hasty there. Maybe there is some observable overhead when you are too creative with module patching and JVM does not like it. I will try to revert these changes, and run the engine benchmarks with built runtime.jar fat jar, to mimic the runtime properties of the engine distribution.

[Akirathan]

Engine benchmarks manually scheduled in enso-org/enso/actions/runs/7400803887

GitHub**Benchmark Engine · enso-org/enso@8b2a8d7**Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

Getting rid of the warning in org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverVector (8b2a8d7) did not provide any differences in the benchmarks

[Akirathan]

Manually scheduling engine benchmarks after 72cbb39 in https://github.com/enso-org/enso/actions/runs/7410754751

GitHub

Benchmark Engine · enso-org/enso@72cbb39

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

[Akirathan]

Comparison of engine benchmark from https://github.com/enso-org/enso/actions/runs/7410754751 to the benchmarks on develop is at generated_site.zip. 72cbb39 did improve the performance to its previous values. So the slowdown was most likely caused by the module patching.

Let's merge this PR.

GitHub

Benchmark Engine · enso-org/enso@72cbb39

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.