Remove "NotImplemented" flags if the test is limited by hardware capacity #781
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a test is flagged as "NotImplemented", there are actually two cases:
eval()
tests, or the pyhpc models don't havetrain()
tests.V100
orA100
.This PR is to eliminate the second case, so that the test can still run through
run.py
orrun_sweep.py
interfaces. Instead, we flag the test to benot_implemented
in themetadata.yaml
, and the CI scripttest.py
ortest_bench.py
will read from the metadata and determine they are not suitable to run on the CI machine.This fixes #688, #626, and #598