Running on CI machines give inherently unstable results

[eregon]

The benchmarks seem to be run in GitHub Actions on GitHub hosted runners.

Those runners are hosted in the cloud, they are probably shared machines running multiple workloads at the same time, etc.
So the results will likely be very noisy.

Is there a plan to address that?

Until then I think it would be useful to run each measurement a few times, or reuse the previous runs to compute the standard deviation or some other estimator of the variance, as this a big caveat.

[hanabi1224]

I think it would be useful to run each measurement a few times

Currently, it runs each benchmark task multiple times and then calculates the average, we can increase the number of runs and further eliminate biased data (with standard deviation or variance you suggested) in the future.

My take is that GitHub or MS has largely redundant CI machine resources and that race condition does not likely to happen (by now).

If you look at the history of build time (of some slow ones, say, rust), which is rather CPU consuming and takes several minutes to run, it's rather stable across different runs, a benchmark task that finished in milliseconds or seconds would much less likely be more unstable (but still good to have some more estimator to eliminate bias)

So the results will likely be very noisy.

That was my worry too, and that's why I used multiple runs at the very beginning. But so far I found the results pretty stable with the tens of runs I've scheduled. (Which you can fork and test yourself). The last resort can always be self-hosting CI machines tho.

BTW, if you meant any specific result, can you please point it out so that I can investigate.

or reuse the previous runs to compute the standard deviation or some other estimator of the variance

Hmm, I'm not sure if it's a good idea to use historical data to build estimators here because hardware spec on the cloud is unknown and may vary at some point. That's why there's no table or chart for historical data

[eregon]

Right, comparing on the same run and therefore same machine is already something.

I think the main thing missing then is stddev or so on the website.
I don't see any estimator of variance on e,g, https://programming-language-benchmarks.vercel.app/ruby currently.

[hanabi1224]

@eregon Just updated site with std dev column, please check again, each task runs 3 times currently.

[eregon]

Thank you, that is useful.

One issue though is that it seems different languages are run in different CI jobs, looking at
https://github.com/hanabi1224/Programming-Language-Benchmarks/runs/2343185468?check_suite_focus=true

That means comparing between languages might be very misleading.
For instance, we found in ruby/setup-ruby that some GH hosted Ubuntu runners actually have different CPU models.

A way to address that would be to run all benchmarks in the same job, but I'm not sure if that's practical.
If not, I think a big warning about this would be warranted.

[hanabi1224]

we found in ruby/setup-ruby that some GH hosted Ubuntu runners actually have different CPU models.

Does that happen on the same or different ubuntu version? I have not noticed differences when I'm using the same ubuntu version (2004) here, but I believe ruby/setup-ruby run on different versions tho.

To mitigate this, I plan to gather and publish hardware information as well. (I guess only cpu / mem are needed)

[eregon]

Also on the same ubuntu version: ruby/setup-ruby#141 (comment)

[hanabi1224]

All benchmark programs run in the same CI VM now