Added multi-os runners for benchmark & implemented luau analyze #542
Conversation
TODO: Test this upstream
Removed duplicate storage of benchmark output files
Using the one from the README now as opposed to the one from the build.yml
AllanJeremy
force-pushed
the
feat/benchmarks-multi-os
branch
from
d3aec20
to
ee128af
Compare
This was a previous attempt at re-usability that didn't exactly pan out well
|
Any sense as to what the variance is on Windows/macOS? The Linux data so far makes me think we might have to refocus on Cachegrind exclusively (which would be Linux only obviously). |
@zeux The variance on windows and MacOS seems erratic to me. On another note though, I would presume the performance on Windows builds and MacOS builds would vary because of the variance in how the different OSes handle resources. On the other hand, it may make sense to refocus to Cachegrind (if we can use the performance of the build on a single OS as a baseline metric for measuring general performance). However, does that mean that we would get rid of the non-cachegrind runs on linux or does it simply mean that we would get rid of the Windows and MacOS jobs? |
|
The performance is definitely going to be different based on OS, it's just that any build that has a high variance in the measurement is likely not going to be a useful data source, as our optimizations tend to be rather small individually. That said, I think for now the best path is to merge this and then assess the variance across more than just a couple builds, and then we can make the judgment as to which data sources to keep. |
Summary
Added benchmark runners for the following operating systems:
Implemented luau analyze static analysis on LuauPolyfill's Map implementation:
Test runs
The only thing that has changed since those runs is enabling the pushing of results as well as updating the workflow trigger to only be on push to master.
Resolves #527
Resolves #529
Resolves #539
Resolves #540
Resolves #541