Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip scenarios in submission checker #1692

Merged
merged 2 commits into from
Jun 25, 2024
Merged

Skip scenarios in submission checker #1692

merged 2 commits into from
Jun 25, 2024

Conversation

yeandy
Copy link
Contributor

@yeandy yeandy commented May 2, 2024

Addresses the feature request here #1691

@yeandy yeandy requested a review from a team as a code owner May 2, 2024 21:09
@github-actions GitHub Actions
Copy link

github-actions bot commented May 2, 2024 •
edited
Loading

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@yeandy
Copy link
Contributor Author

yeandy commented May 2, 2024

What manual tests need to be done to confirm the correctness of this?

@arjunsuresh
Copy link
Contributor

@yeandy this PR should add the required test but it is not yet merged.

Thank you for adding this option to the submission checker. The same logic can also apply to benchmarks, right? Like restricting the submission checker to run for certain benchmarks.

@yeandy
Copy link
Contributor Author

yeandy commented May 3, 2024

This logic has not been tested to skip entire benchmarks. I only tested to see if it skipped Server and Offline scenarios.

mrmhodak
mrmhodak previously approved these changes May 21, 2024
View reviewed changes
@pgmpablo157321
Copy link
Contributor

@yeandy Can you resolve the conflicts in order to merge this PR?

@yeandy yeandy force-pushed the yeandy_skip_scenarios branch 2 times, most recently from f117cee to f0c9ab9 Compare May 21, 2024 23:09
@arjunsuresh
Copy link
Contributor

Hi @yeandy please ignore the test failure. It is due to a bug in the code in the inference repo which was missed by the github action. This PR should fix the test and I'll give another PR for the submission checker fix.

#1705

@yeandy yeandy requested a review from mrmhodak May 22, 2024 14:30
@mrmhodak mrmhodak merged commit 799b30a into mlcommons:master Jun 25, 2024
6 of 9 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jun 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mrmhodak mrmhodak mrmhodak approved these changes

@mrasquinha-g mrasquinha-g mrasquinha-g approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yeandy @arjunsuresh @pgmpablo157321 @mrmhodak @mrasquinha-g