Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent composite score best score computation #8621

Open
franz1981 opened this issue Dec 5, 2023 · 2 comments
Open

Inconsistent composite score best score computation #8621

franz1981 opened this issue Dec 5, 2023 · 2 comments

Comments

@franz1981
Copy link
Contributor

franz1981 commented Dec 5, 2023

Looking at plaintext, for officefloor: https://www.techempower.com/benchmarks/#hw=ph&test=plaintext&section=data-r22&c=e&o=e&l=xan3i7-cn3

it scores its best at 33,107, but while moving to the Composite Score tab (at https://www.techempower.com/benchmarks/#hw=ph&test=composite&section=data-r22&c=e&o=e&l=xan3i7-cn3), it get 5,648,781 resulting far higher in the overall score than it should be.

it looks like a bug...

@fakeshadow
Copy link
Contributor

fakeshadow commented Dec 5, 2023

This is not a bug from what I see. “display_name” and "framework" are separate fields in benchmark_config.json. The composite score just add every best score from every category under the same framework regardless the display names of multiple tests.

That said I don't feel it's the correct behavior. IMO the composite score should either take the display names into account and give multiple scores for one framework(if it's has many display names). Or choose one display name with the best overall score for one framework. This also means one test must implement all bench categories in order to be shown in composite score list.

@franz1981
Copy link
Contributor Author

franz1981 commented Dec 5, 2023

Ah, it's surprising, because I've forcibly added a filter (which is the same for both composite/plaintext), which means, as a user, I expect it to be applied regardless which tab I click ie the best plaintext result of a framework "flavour" which doesn't belong to the filter, should not appear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants