Submission checker version 4.0 #1560
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| @@ -1309,6 +1509,7 @@ def __init__( | |||
| self.seeds = self.base["seeds"] | |||
| self.test05_seeds = self.base["test05_seeds"] | |||
| self.accuracy_target = self.base["accuracy-target"] | |||
| self.accuracy_upper_limit = self.base["accuracy-upper-limit"] | |||
There was a problem hiding this comment.
I think we should use self.base.get("accuracy-upper-limit") as it is not there for all the benchmarks.
There was a problem hiding this comment.
This is done here: https://github.com/pgmpablo157321/inference/blob/f7c755a0e156f51c1684f984889b309b8afaffc2/tools/submission/submission_checker.py#L1618
The script passes the dictionary to the config and calls get for each model
|
@pgmpablo157321 can you also change the loadgen version similar to: |
| "3d-unet-99.9", | ||
| "gptj-99", | ||
| "gptj-99.9", | ||
| "llama-v2-70b-99", |
There was a problem hiding this comment.
I believe the model name should be llama2-70b according to the official website: https://huggingface.co/docs/transformers/main/model_doc/llama2
There was a problem hiding this comment.
Same for all occurrences
There was a problem hiding this comment.
I copied it from here: https://github.com/pgmpablo157321/inference/blob/f7c755a0e156f51c1684f984889b309b8afaffc2/language/llama2-70b/mlperf.conf#L59C1-L59C1
Should we change all the references in that folder as well?
There was a problem hiding this comment.
I think we should change all to llama2. The llama-v2 name will cause confusion.
pgmpablo157321
force-pushed
the
submission_checker_4.0
branch
2 times, most recently
from
f7c755a
to
14efc69
Compare
pgmpablo157321
force-pushed
the
submission_checker_4.0
branch
from
03a7a0f
to
9d3d964
Compare
|
Thank you Pablo! Can we merge this ASAP so logs can be collected 🙏 |
This PR contains the necessary changes for a v4.0 submissions with the exception of some pending changes for Llama2