From 41291591f89767a79839f43d3cf5c36493f713d4 Mon Sep 17 00:00:00 2001 From: johntran-nv Date: Wed, 1 Mar 2023 11:38:32 -0700 Subject: [PATCH 1/3] Preview Invalidation proposal Proposing formal text for how to invalidate preview submissions. This has been discussed in the Inference and Training WG's at some length already, and folks recommended putting something in writing so we have something to debate concretely. --- submission_rules.adoc | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/submission_rules.adoc b/submission_rules.adoc index 89376db..5ca8588 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -749,6 +749,13 @@ Examples and counterexamples: * At this point in time a hardware component that is not an ML accelerator, CPU, or NIC, is presumed to not meet the "substantial contributor to the determination of ML performance" criteria. Other possible cases must be brought to the relevant working group for consideration. +##### Preview Invalidation + +If a submission is unable to meet the _Available_ requirements in the following round where it is required to be _Available_, then the submission will be _Invalidated_. This invalidation will involve: + +1. The results table of the Preview submission will strike through the results in red. If a row in the results table has some results that are valid and some that will be invalid, we will split that row into two rows, and only strike through the invalidated results, appending an "i" to the new Submission ID for the invalid row. Additionally, the System Name will be prefixed with "INVALID" with a footnote that will say "Submission was marked invalid in because it failed to meet the Preview to Available requirements. See https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#preview-systems." + +2. In the press statement for the following round where it is required to be _Available_, there will be a statement that says "Some Preview submissions in were invalidated for failing to meet the Preview to Available requirements. Please refer to that round's results table for more details." #### Research, Development, or Internal Systems From 291387ac0a2cdc12b5caa56f83f56de3a540b315 Mon Sep 17 00:00:00 2001 From: johntran-nv Date: Wed, 5 Apr 2023 20:34:50 -0600 Subject: [PATCH 2/3] Update submission_rules.adoc Co-authored-by: Dilip Sequeira --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 5ca8588..769565e 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -755,7 +755,7 @@ If a submission is unable to meet the _Available_ requirements in the following 1. The results table of the Preview submission will strike through the results in red. If a row in the results table has some results that are valid and some that will be invalid, we will split that row into two rows, and only strike through the invalidated results, appending an "i" to the new Submission ID for the invalid row. Additionally, the System Name will be prefixed with "INVALID" with a footnote that will say "Submission was marked invalid in because it failed to meet the Preview to Available requirements. See https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#preview-systems." -2. In the press statement for the following round where it is required to be _Available_, there will be a statement that says "Some Preview submissions in were invalidated for failing to meet the Preview to Available requirements. Please refer to that round's results table for more details." +2. In the press statement for the round where it is required to be _Available_, there will be a statement that says "Some Preview submissions in were invalidated for failing to meet the Preview to Available requirements. Please refer to that round's results table for more details." #### Research, Development, or Internal Systems From ed6e4bae587ba0360ba27a69ad3dd553d26adac3 Mon Sep 17 00:00:00 2001 From: johntran-nv Date: Wed, 5 Apr 2023 20:35:20 -0600 Subject: [PATCH 3/3] Update submission_rules.adoc Co-authored-by: Dilip Sequeira --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 769565e..fc053cd 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -753,7 +753,7 @@ Examples and counterexamples: If a submission is unable to meet the _Available_ requirements in the following round where it is required to be _Available_, then the submission will be _Invalidated_. This invalidation will involve: -1. The results table of the Preview submission will strike through the results in red. If a row in the results table has some results that are valid and some that will be invalid, we will split that row into two rows, and only strike through the invalidated results, appending an "i" to the new Submission ID for the invalid row. Additionally, the System Name will be prefixed with "INVALID" with a footnote that will say "Submission was marked invalid in because it failed to meet the Preview to Available requirements. See https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#preview-systems." +1. The invalid row will be in red and struck through with an "i" appended to the Submission ID and "INVALID" prepended to the System Name, with a footnote saying "Submission was marked invalid in because it failed to meet the Preview to Available requirements. See https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#preview-systems." If a row in the results table has some results that are valid and some that are invalid, it will be split into two rows. 2. In the press statement for the round where it is required to be _Available_, there will be a statement that says "Some Preview submissions in were invalidated for failing to meet the Preview to Available requirements. Please refer to that round's results table for more details."