Semi-automation of engine benchmarks processing

[Akirathan]

Recent issues with Engine performance made me think about semi-automate processing of benchmark reports once again. This time, I think I have a solution that is easy to implement and very useful.

Current state

Currently, there are Engine benchmark actions defined as post-merge jobs, that usually wait about 15 hours to be scheduled, and run for roughly 3 hours. These jobs upload a single bench-report.xml artifact that contains benchmark names with their associated score (iteration duration in ms), nothing else. There is no automation whatsoever, for benchmark comparison between two different PRs, one needs to manually find the appropriate benchmark jobs, download the bench-report.xml artifact, and manually compare these two XML files. I doubt that there is a single person from the Engine team doing that regularly.

TL;DR; All the benchmark post-merge jobs nowadays are essentially waste of at least 3 hours of CPU time. Pretty intensive CPU time, I assume, since the Graal compilation is distributed among many threads on the background.

Proposition

Let me summarize the proposition into a simple list of steps. More detailed description can be found in #5714.

• Improve benchmark reports from the Engine - add fields like benchmark version, date, etc.
  • Trivial to do on the engine side, since we use JMH for benchmarks and
• Provide a MongoDB endpoint, where all the benchmark results will be uploaded.
  • This requires setting up a MongoDB on some of our nodes, and providing some tool (Python/Rust script?, should be straightforward) for uploading JSON files to that database.

We can then create an Enso project that will connect to the MongoDB, and process, say, last n entries and create a table from that that will be manually inspected by someone from the engine team on a weekly basis. Related issue for that is #5165.

cc @wdanilo @xvcgreg @JaroslavTulach

[wdanilo]

We can also split benchmarks into multiple machines - what do you think about it?

[Akirathan]

The problem is not that we wait too long for benchmark jobs. That is expected. They are post-merge by design, and will stay as post-merge. The problem is that the results from these jobs are not collected to a central place for further processing - for every separate benchmark jobs, there is a separate benchmark-report.xml artifact, that you need to download manually and compare with other XML.

[wdanilo]

Sure, but I was referring to tests running long - I believe this is another issue, right?

[Akirathan]

Sure, but I was referring to tests running long - I believe this is another issue, right?

Exactly. That is a completely different story.

[jdunkerley]

We are intending adding XML reading into Enso at some point soon. There is a ticket for us to build a workflow to analyze these benchmarks in future.

[Akirathan]

XML reading/MongoDB connection into Enso is a second part. The first part that we need to do ASAP is the benchmark data collection. It is now scattered across different benchmark jobs which makes it difficult for any kind of analysis, without some GitHub API magic.

[jdunkerley]

Well that is clearly the other part to deal with :)
We should be able to get the GH API into Enso and work with it I'd hope.

[Akirathan]

My point is, we should do that rather soon. Because now, the benchmark jobs are essentially wasting tons of CPU time. Collecting it into a giant JSON / MongoDB (instead of waiting until we implement/provide a facade for GH API in Enso) would be pretty quick and would make this whole process a lot easier, I believe.

[JaroslavTulach]

get the GH API into Enso and work with it

+1 - yes, that's what I'd like to see. Issue an HTTP request, download JSON, parse it, download all bench-report.xml from develop branch produced in the last week.

Having that would be an amazing start.

[Akirathan]

What about using gh CLI tool and skip GH API for now? gh CLI tool has all of the GH REST API, and it handles authentication for you. The only thing required for it is the ability to start subprocesses, which we already have.

[hubertp]

I think it would be even enough if we just drop benchmark results in AWS bucket. Adding MongoDB to the equation means that someone has to maintain it.