From 298c2e3595cf41ce9fa02f810bf94f20ce44bf51 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Tue, 16 Aug 2022 16:29:29 +0100 Subject: [PATCH 01/10] Update submission_rules.adoc Post-submission extra grace period to avoid late submissions due to submission packaging related activities --- submission_rules.adoc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index a5ab3cc..1514be8 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -267,7 +267,10 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta ### Late Submissions #### Post-submission grace period (submission deadline + 60 minutes): -MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. This grace period will be advertised as little as possible. The 60 minute limit will be strictly enforced. +MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The 60 minute limit will be strictly enforced. + +#### Post-submission extra grace period (submission deadline + 120 minutes): +MLPerf will allow submissions for up to 120 minutes from the published deadline or 60 minutes after the normal grace period provided all the run logs are taken within the normal grace period. This extra grace period is only to ensure that no submitter asks for a time extension due to submission packaging related activities and ideally, a submitter should not use this grace period. Even if all the results are taken before the deadline, unless a valid submission tarball is submitted before this extra grace period, no submission will be considered except for extraordinary circumstances detailed in the next section. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): If a submitter notifies the submission chair that their submission will be delayed due to force-majeure-type circumstances (e.g. blizzards, hurricanes, terrorism, etc.), the submission chair will delay sharing results for up to 72 hours to allow that submitter more time to make their submission. The extraordinary nature of the circumstances must be approved by the review committee at the first committee meeting or the submission will be disregarded. From 07ad17a7911edcf8f6b1766be795bd6d17ab2890 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Wed, 17 Aug 2022 23:59:18 +0100 Subject: [PATCH 02/10] Update submission_rules.adoc Adding suggestions from Nvidia for late submission rules --- submission_rules.adoc | 33 +++------------------------------ 1 file changed, 3 insertions(+), 30 deletions(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 1514be8..00bc453 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -267,10 +267,9 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta ### Late Submissions #### Post-submission grace period (submission deadline + 60 minutes): -MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The 60 minute limit will be strictly enforced. +MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. In case the submitter cannot upload the valid submission tarball by the hard deadline due to a defect in the submission checker, a hash of the valid submission tarball must be sent to the chairs before the deadline. The burden is on the submitter to demonstrate that the tarball in question could not be uploaded via the web UI because the submission checker was broken, and the tarball must have the complete results including all required compliance tests and must the tarball must pass the submission checker once the required fix for submission checker is done. The 60 minute limit will be strictly enforced. + -#### Post-submission extra grace period (submission deadline + 120 minutes): -MLPerf will allow submissions for up to 120 minutes from the published deadline or 60 minutes after the normal grace period provided all the run logs are taken within the normal grace period. This extra grace period is only to ensure that no submitter asks for a time extension due to submission packaging related activities and ideally, a submitter should not use this grace period. Even if all the results are taken before the deadline, unless a valid submission tarball is submitted before this extra grace period, no submission will be considered except for extraordinary circumstances detailed in the next section. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): If a submitter notifies the submission chair that their submission will be delayed due to force-majeure-type circumstances (e.g. blizzards, hurricanes, terrorism, etc.), the submission chair will delay sharing results for up to 72 hours to allow that submitter more time to make their submission. The extraordinary nature of the circumstances must be approved by the review committee at the first committee meeting or the submission will be disregarded. @@ -593,36 +592,10 @@ This section in progress [TODO]. #### Inference -Submitters must run the compliance tests for their closed and network divisions submissions to verify that their submission achieves a basic level of compliance with a subset of the MLPerf rules. If compliance testing identifies a potential issue with the submission, the onus is on the submitter to provide an adequate explanation to the results review committee. +Submitters must run the compliance tests for their closed and network divisions submissions to verify that their submission achieves a basic level of compliance with a subset of the MLPerf rules. If compliance testing identifies a potential issue, submission checker will not allow a submission to closed division and the submission must go to the open division. Such submissions can be moved to closed division provided the submitter is able to provide an adequate explanation to the results review committee. Refer to the documentation found under https://github.com/mlperf/inference/tree/master/compliance/nvidia -### Submission Privacy - -Submitters may adopt the following procedure to ensure that their submission is shared only with others who have made valid submissions: - -By the submission deadline, the submitter will email the following items to submissions@mlcommons.org to show Proof of Work: - -- A URL to the location in public blob storage of an encrypted tarball containing the directory structure and files described in the <> section above. Any storage mechanism that supports curl may be used. -- A password to the encrypted tarball -- A sha1sum of the tarball -- The last two lines of submission_checker_log.txt as cursory evidence of a valid submission - -A simple script has been provided to facilitate this process: https://github.com/mlcommons/logging/blob/master/pack_submission.sh - -When the deadline is reached, repository access will be removed for everyone EXCEPT the following parties: - -- Chairs -- Submitters who have submitted valid submissions - -For each submitted tarball, the results chair will then: - -- download the encrypted tarball -- verify it matches the hash -- decrypt the tarball -- verify that it is a valid submission -- add the submission to the repository -- enable repository access for the submitter ## Review From eca94d15e086d2b60ae4d092bda34e0c026100c6 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Thu, 18 Aug 2022 18:39:48 +0100 Subject: [PATCH 03/10] Update submission_rules.adoc Co-authored-by: Dilip Sequeira --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 00bc453..53f52ea 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -592,7 +592,7 @@ This section in progress [TODO]. #### Inference -Submitters must run the compliance tests for their closed and network divisions submissions to verify that their submission achieves a basic level of compliance with a subset of the MLPerf rules. If compliance testing identifies a potential issue, submission checker will not allow a submission to closed division and the submission must go to the open division. Such submissions can be moved to closed division provided the submitter is able to provide an adequate explanation to the results review committee. +Submitters must run the compliance tests for their closed and network divisions submissions to verify that their submission achieves a basic level of compliance with a subset of the MLPerf rules. If a compliance test fails, the submission checker will not allow a submission to closed division. If the submission is instead made to open division, it may be moved to closed division upon an adequate explanation of the compliance test failure to the review committee. Refer to the documentation found under https://github.com/mlperf/inference/tree/master/compliance/nvidia From 271d3c8d8eabfeb17604e361ba11b6695db02341 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Thu, 18 Aug 2022 23:48:15 +0100 Subject: [PATCH 04/10] Update submission_rules.adoc Co-authored-by: Dilip Sequeira --- submission_rules.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 53f52ea..e9e47ae 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -267,7 +267,9 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta ### Late Submissions #### Post-submission grace period (submission deadline + 60 minutes): -MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. In case the submitter cannot upload the valid submission tarball by the hard deadline due to a defect in the submission checker, a hash of the valid submission tarball must be sent to the chairs before the deadline. The burden is on the submitter to demonstrate that the tarball in question could not be uploaded via the web UI because the submission checker was broken, and the tarball must have the complete results including all required compliance tests and must the tarball must pass the submission checker once the required fix for submission checker is done. The 60 minute limit will be strictly enforced. +MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. + +If a submitter cannot upload a valid submission tarball by the hard deadline due to a defect in the submission checker, a link to a the tarball and a hash of the tarball must be sent to the chairs before the deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. From 9341e97b2d42d101c5656abd5d6febbb6ce1e44d Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Thu, 18 Aug 2022 23:50:40 +0100 Subject: [PATCH 05/10] Update submission_rules.adoc Typo correction --- submission_rules.adoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index e9e47ae..5ee35e6 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -269,8 +269,7 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta #### Post-submission grace period (submission deadline + 60 minutes): MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. -If a submitter cannot upload a valid submission tarball by the hard deadline due to a defect in the submission checker, a link to a the tarball and a hash of the tarball must be sent to the chairs before the deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. - +If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a link to the tarball and a hash of the tarball must be sent to the chairs before the deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): From 9e51ccb6d54768ac00cfb591eff73b0d3f3c33a1 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Fri, 26 Aug 2022 09:25:08 +0100 Subject: [PATCH 06/10] Update submission_rules.adoc --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 5ee35e6..c2a7f89 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -269,7 +269,7 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta #### Post-submission grace period (submission deadline + 60 minutes): MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. -If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a link to the tarball and a hash of the tarball must be sent to the chairs before the deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. +If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a link to the tarball and a hash of the tarball must be sent to the submitters mailing list before the deadline. An issue must be raised on the relevant GitHub repository detailing the failure reason of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix must be allowed. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): From eb7018b2f2bef9f599bc3a93b7050af95d0be45d Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Fri, 26 Aug 2022 11:27:02 +0100 Subject: [PATCH 07/10] Update submission_rules.adoc Co-authored-by: Dilip Sequeira --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index c2a7f89..df06d91 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -269,7 +269,7 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta #### Post-submission grace period (submission deadline + 60 minutes): MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. -If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a link to the tarball and a hash of the tarball must be sent to the submitters mailing list before the deadline. An issue must be raised on the relevant GitHub repository detailing the failure reason of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix must be allowed. +If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a link to the tarball and a hash of the tarball must be sent to the submitters mailing list before the deadline. An issue must be raised on the relevant GitHub repository detailing the failure reason of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): From 7ade442f7c1384669a4c38da69dfbe81c7644cba Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Wed, 31 Aug 2022 09:58:06 +0100 Subject: [PATCH 08/10] Late submission due to submission checker issue Adding the requirements for late submissions due to any issue with submission checker --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index df06d91..46eb52a 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -269,7 +269,7 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta #### Post-submission grace period (submission deadline + 60 minutes): MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. -If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a link to the tarball and a hash of the tarball must be sent to the submitters mailing list before the deadline. An issue must be raised on the relevant GitHub repository detailing the failure reason of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. +If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a hash of the tarball must be sent to the submitters mailing list before the submission deadline and the submission tarball must be mailed to the submitters mailing list not later than 30 minutes from the submission deadline. An issue must also be raised on the relevant GitHub repository detailing the failure of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): From d2795299b835aca534b64075fe2895e778644aba Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Wed, 14 Sep 2022 14:48:52 +0100 Subject: [PATCH 09/10] Update submission_rules.adoc Added examples for what's allowed for late submission due to a bug in submission checker --- submission_rules.adoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 46eb52a..e98865c 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -269,8 +269,7 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta #### Post-submission grace period (submission deadline + 60 minutes): MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. -If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a hash of the tarball must be sent to the submitters mailing list before the submission deadline and the submission tarball must be mailed to the submitters mailing list not later than 30 minutes from the submission deadline. An issue must also be raised on the relevant GitHub repository detailing the failure of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. - +If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a hash of the tarball must be sent to the submitters mailing list before the submission deadline and the submission tarball must be mailed to the submitters mailing list not later than 30 minutes from the submission deadline. An issue must also be raised on the relevant GitHub repository detailing the failure of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. For example, if the submission checker did not mandate one or more of the required compliance tests or any other mandatory file due to the bug, they can be later added to the submission tarball. But if the submission checker has already thrown out an error for one or missing compliance tests or any other mandatory file even before the bug is fixed, then they cannot be after wards added to the submission tarball. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): If a submitter notifies the submission chair that their submission will be delayed due to force-majeure-type circumstances (e.g. blizzards, hurricanes, terrorism, etc.), the submission chair will delay sharing results for up to 72 hours to allow that submitter more time to make their submission. The extraordinary nature of the circumstances must be approved by the review committee at the first committee meeting or the submission will be disregarded. From 48bf07b382831f13143e3ff4ea36473629e6d7a0 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Wed, 14 Sep 2022 16:54:11 +0100 Subject: [PATCH 10/10] Update submission_rules.adoc Co-authored-by: Dilip Sequeira --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index e98865c..0fe0b12 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -269,7 +269,7 @@ MLCommons provides a web UI [TODO: URL] for verifying scores contained in the ta #### Post-submission grace period (submission deadline + 60 minutes): MLPerf will allow submissions for up to 60 minutes after the published deadline without explanation or penalty. The end of this grace period constitutes a hard deadline which will be strictly enforced. -If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a hash of the tarball must be sent to the submitters mailing list before the submission deadline and the submission tarball must be mailed to the submitters mailing list not later than 30 minutes from the submission deadline. An issue must also be raised on the relevant GitHub repository detailing the failure of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. For example, if the submission checker did not mandate one or more of the required compliance tests or any other mandatory file due to the bug, they can be later added to the submission tarball. But if the submission checker has already thrown out an error for one or missing compliance tests or any other mandatory file even before the bug is fixed, then they cannot be after wards added to the submission tarball. +If a submitter cannot upload a valid submission tarball by the hard deadline due to some defect in the submission checker, a hash of the tarball must be sent to the submitters mailing list before the submission deadline and the submission tarball must be mailed to the submitters mailing list not later than 30 minutes from the submission deadline. An issue must also be raised on the relevant GitHub repository detailing the failure of the submission checker before the submission deadline. The burden is on the submitter to demonstrate that the submission could not be uploaded to via the web GUI because the checker was defective, and that the tarball passes the submission checker once the defect has been addressed. No other changes to the tarball other than those directly related to the submission checker fix will be allowed. For example, if the submission checker did not mandate one or more of the required compliance tests or any other mandatory file due to the bug, they can be later added to the submission tarball. But if the submission checker has already reported an error for one or missing compliance tests or any other mandatory file even before the bug is fixed, then they cannot be afterwards added to the submission tarball. #### Post-submission extension for extraordinary circumstances (submission deadline + 72 hours): If a submitter notifies the submission chair that their submission will be delayed due to force-majeure-type circumstances (e.g. blizzards, hurricanes, terrorism, etc.), the submission chair will delay sharing results for up to 72 hours to allow that submitter more time to make their submission. The extraordinary nature of the circumstances must be approved by the review committee at the first committee meeting or the submission will be disregarded.