Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalised major GCs display the wrong baseline counts #123

Closed
kayceesrk opened this issue Apr 24, 2023 · 1 comment · Fixed by #124
Closed

Normalised major GCs display the wrong baseline counts #123

kayceesrk opened this issue Apr 24, 2023 · 1 comment · Fixed by #124
Assignees

Comments

@kayceesrk
Copy link
Contributor

kayceesrk commented Apr 24, 2023

Consider the normalised major collections result here

We see that the baseline for this graph is turing_5.2.0+trunk+sequential_20230423_063894d and the plotted result is for the variant is turing_5.2.0+trunk+fix-idle-gc+sequential_20230423_7440f24.

Problem

Observe that the right-most benchmark is labelled LU_decomposition.1024 (577.00)

The intention is that "577" is the number associated for the baseline i.e, trunk. That is trunk does 577 major GCs and relative that fix-dle-gc variant does 100x more major GCs.

However, it turns out that the labels are incorrect, if you look at the data, then trunk does only 6 major collections where as fix-idle-gc does 577 collections.

Intended behaviour

The label should be LU_decomposition.1024 (6). In the course of it, can we also not display this label as a floating point number as we are counting the number of GCs and this can never not be a natural number?

Please can we also verify that other metrics don't also have this issue?

@punchagan punchagan self-assigned this Apr 24, 2023
@punchagan punchagan transferred this issue from ocaml-bench/sandmark Apr 24, 2023
punchagan added a commit to punchagan/sandmark-nightly that referenced this issue Apr 24, 2023
1468d7c incorrectly fixed an issue with
`add_display_name` where the baseline and variant having different benchmarks.

Closes ocaml-bench#123
@punchagan
Copy link
Contributor

Thanks for the clear and detailed bug report.

Other metrics also had this issues, since a utility function that generated the name with the baseline value was buggy. Apologies for the inconvenience.

This should be fixed soon with #124.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants