From da411b56fbe8b21b92df0c4e2418994252a4eb49 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 8 Nov 2022 11:42:32 -0800 Subject: [PATCH 01/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index f36f1cb..2858aa5 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -12,6 +12,8 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. +Any unverified or Non-MLCommons Reviewed results derived for competitor product or service for use in public claims should use the competitor's submission code repository if one exists. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available. + == Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results If you used an MLPerf benchmark to obtain a result for your product or service, you submitted your result for MLCommons review, and your result was verified through such review, you may indicate your results are verified when publishing or otherwise discussing your results, by indicating your results are “verified” or “official” or by otherwise following the examples below for verified results. You may also choose to use this language: “_Result verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. From 43ef60d4e7adaa3bbb7b9e424d6081c3b49775fc Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 8 Nov 2022 11:47:21 -0800 Subject: [PATCH 02/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 2858aa5..848d306 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -12,7 +12,7 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. -Any unverified or Non-MLCommons Reviewed results derived for competitor product or service for use in public claims should use the competitor's submission code repository if one exists. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available. +Any unverified or Non-MLCommons Reviewed results derived for a competitor product or service for use in public claims should use the competitor's submission code repository if one exists instead of reference code repository. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available. == Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results From d709b077e82a8268bd98bd64c0890fe506c6ead8 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 8 Nov 2022 12:53:06 -0800 Subject: [PATCH 03/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 848d306..f967fb3 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -12,7 +12,7 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. -Any unverified or Non-MLCommons Reviewed results derived for a competitor product or service for use in public claims should use the competitor's submission code repository if one exists instead of reference code repository. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available. +Any unverified or Non-MLCommons Reviewed results derived for a competitor product or service for use in public claims must use the competitor's submission code repository if one exists instead of reference code repository. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available. == Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results From 9dac7332b93a4ba70c380e55263322b837f49043 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 15 Nov 2022 17:42:13 -0800 Subject: [PATCH 04/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index f967fb3..e26f26e 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -204,9 +204,9 @@ The MLPerf mark is owned by MLCommons, and only MLCommons and its authorized lic Any MLCommons Member, Test Partner, or third party may report a violation of these Guidelines via email to the MLCommons Executive Director (“ED”) & Working Group (“WG”) chairs of the appropriate benchmark. Upon confirming the violation in their discretion, ED & WG chairs would inform the potential violator and request remedial action. If the ED, WG chairs, and potential violator are unable to reach a mutually satisfactory conclusion, the issue can be raised in WG to seek resolution via WG vote. -A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. +Violating content must be taken down within first 24 hours of violation being reported. A non-exhaustive list of other possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. -1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. +1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. 2. If the violation was at a public event such as a conference, the WG may direct the violator to issue a public statement to correct claims in ways that conform to these Guidelines. 3. The WG may issue a public statement citing the violation. 4. The WG may prohibit the violator from submitting MLPerf benchmark results for MLCommons review in the future. From 3ba70cbfece6a19faec171e08202eb350f3ad632 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 15 Nov 2022 17:43:15 -0800 Subject: [PATCH 05/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index e26f26e..6b52d5b 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -206,7 +206,7 @@ Any MLCommons Member, Test Partner, or third party may report a violation of the Violating content must be taken down within first 24 hours of violation being reported. A non-exhaustive list of other possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. -1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. +1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. 2. If the violation was at a public event such as a conference, the WG may direct the violator to issue a public statement to correct claims in ways that conform to these Guidelines. 3. The WG may issue a public statement citing the violation. 4. The WG may prohibit the violator from submitting MLPerf benchmark results for MLCommons review in the future. From 793199b649eceb4a2bde53b70620132afdf15841 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 15 Nov 2022 17:45:06 -0800 Subject: [PATCH 06/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 6b52d5b..60425c9 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -204,7 +204,7 @@ The MLPerf mark is owned by MLCommons, and only MLCommons and its authorized lic Any MLCommons Member, Test Partner, or third party may report a violation of these Guidelines via email to the MLCommons Executive Director (“ED”) & Working Group (“WG”) chairs of the appropriate benchmark. Upon confirming the violation in their discretion, ED & WG chairs would inform the potential violator and request remedial action. If the ED, WG chairs, and potential violator are unable to reach a mutually satisfactory conclusion, the issue can be raised in WG to seek resolution via WG vote. -Violating content must be taken down within first 24 hours of violation being reported. A non-exhaustive list of other possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. +Violating content must be taken down within first 24 hours of violation being reported. A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. 1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. 2. If the violation was at a public event such as a conference, the WG may direct the violator to issue a public statement to correct claims in ways that conform to these Guidelines. From 86b733fdb34aa22b8a446390f763ff9c897b676b Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Thu, 9 Feb 2023 12:19:54 -0800 Subject: [PATCH 07/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 60425c9..5f11c5e 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -12,7 +12,7 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. -Any unverified or Non-MLCommons Reviewed results derived for a competitor product or service for use in public claims must use the competitor's submission code repository if one exists instead of reference code repository. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available. +If the components (eg: HW) that substantially determine ML performance of an "unverified" score also have a verified official score (eg: same HW with a different submission SW stack), it is required to reference the official submission scores in any public comparisons. == Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results @@ -80,7 +80,9 @@ MLPerf results may not be compared against non-MLPerf results. For example, an M == When comparing MLPerf results, you must identify any submission differences When comparing results the main text, table, or figure must clearly identify any difference in version, division, category, verified or unverified status, scenario or chip count (count of the compute devices executing the largest number of ops, which could be processors or accelerators). When comparing Open and Closed division results, any ways in which the Open result would not qualify as a Closed result must be identified. - + +It is prohibited to compare to an "unverified" or unofficial score derived from reference benchmark code implementation for a submitter who already has an official submission score (references often prioritize readability over performance). + **Example for Non-MLCommons Reviewed Result**: ____ From 729d5c3f99b68f3612ad3e8ad580da7c6791c961 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Wed, 15 Feb 2023 11:38:45 -0800 Subject: [PATCH 08/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 5f11c5e..4710f05 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -12,7 +12,7 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. -If the components (eg: HW) that substantially determine ML performance of an "unverified" score also have a verified official score (eg: same HW with a different submission SW stack), it is required to reference the official submission scores in any public comparisons. +If the components (e.g. HW) that substantially determine ML performance of an "unverified" score also have a verified official score (e.g. same HW with a different submission SW stack), it is required to state the official submission scores in any public comparisons. == Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results From 6822be6d6661cc4be239937a8ca97494af9d309a Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Wed, 15 Feb 2023 12:04:29 -0800 Subject: [PATCH 09/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 4710f05..a996f96 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -14,6 +14,8 @@ Since your results have not gone through MLCommons review and, therefore, have n If the components (e.g. HW) that substantially determine ML performance of an "unverified" score also have a verified official score (e.g. same HW with a different submission SW stack), it is required to state the official submission scores in any public comparisons. +Note that benchmark reference code implementations often prioritize readability and should not be mistaken for performance optimized implementations (e.g. reference code performance should not be equated with "out-of-box" performance). + == Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results If you used an MLPerf benchmark to obtain a result for your product or service, you submitted your result for MLCommons review, and your result was verified through such review, you may indicate your results are verified when publishing or otherwise discussing your results, by indicating your results are “verified” or “official” or by otherwise following the examples below for verified results. You may also choose to use this language: “_Result verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. @@ -81,8 +83,6 @@ MLPerf results may not be compared against non-MLPerf results. For example, an M When comparing results the main text, table, or figure must clearly identify any difference in version, division, category, verified or unverified status, scenario or chip count (count of the compute devices executing the largest number of ops, which could be processors or accelerators). When comparing Open and Closed division results, any ways in which the Open result would not qualify as a Closed result must be identified. -It is prohibited to compare to an "unverified" or unofficial score derived from reference benchmark code implementation for a submitter who already has an official submission score (references often prioritize readability over performance). - **Example for Non-MLCommons Reviewed Result**: ____ @@ -206,7 +206,7 @@ The MLPerf mark is owned by MLCommons, and only MLCommons and its authorized lic Any MLCommons Member, Test Partner, or third party may report a violation of these Guidelines via email to the MLCommons Executive Director (“ED”) & Working Group (“WG”) chairs of the appropriate benchmark. Upon confirming the violation in their discretion, ED & WG chairs would inform the potential violator and request remedial action. If the ED, WG chairs, and potential violator are unable to reach a mutually satisfactory conclusion, the issue can be raised in WG to seek resolution via WG vote. -Violating content must be taken down within first 24 hours of violation being reported. A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. +Violating content must be taken down within first 48 hours of violation being reported. A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. 1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. 2. If the violation was at a public event such as a conference, the WG may direct the violator to issue a public statement to correct claims in ways that conform to these Guidelines. From 3ad0d10dfb11e87d9eab54c57f91e15651cdbd78 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Thu, 9 Mar 2023 09:21:52 -0800 Subject: [PATCH 10/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index a996f96..5434479 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -12,7 +12,7 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below. -If the components (e.g. HW) that substantially determine ML performance of an "unverified" score also have a verified official score (e.g. same HW with a different submission SW stack), it is required to state the official submission scores in any public comparisons. +If the components (e.g. HW) that substantially determine ML performance of an "unverified" score also have a verified official score (e.g. same HW with a different submission SW stack), it is required to state the official submission score of the closest available system in any public comparisons. Note that benchmark reference code implementations often prioritize readability and should not be mistaken for performance optimized implementations (e.g. reference code performance should not be equated with "out-of-box" performance). From 7484070c6825aed4e7598c73b5686d2520d2f5c4 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Wed, 19 Apr 2023 09:55:50 -0700 Subject: [PATCH 11/12] Update MLPerf_Results_Messaging_Guidelines.adoc Removed violation addressing deadline from this PR as it would be more productive to discuss it in a separate PR. Added language to disallow misrepresentation of performance from 3rd party submitters. --- MLPerf_Results_Messaging_Guidelines.adoc | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 5434479..7d381a8 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -81,7 +81,7 @@ MLPerf results may not be compared against non-MLPerf results. For example, an M == When comparing MLPerf results, you must identify any submission differences -When comparing results the main text, table, or figure must clearly identify any difference in version, division, category, verified or unverified status, scenario or chip count (count of the compute devices executing the largest number of ops, which could be processors or accelerators). When comparing Open and Closed division results, any ways in which the Open result would not qualify as a Closed result must be identified. +When comparing results the main text, table, or figure must clearly identify any difference in version, division, category, verified or unverified status, scenario or chip count (count of the compute devices executing the largest number of ops, which could be processors or accelerators) or submitter name. When comparing Open and Closed division results, any ways in which the Open result would not qualify as a Closed result must be identified. When making comparisons, submissions must not be portrayed as representing the performance available from an Independent Hardware Vendor (IHV) unless the submission was by that IHV. **Example for Non-MLCommons Reviewed Result**: @@ -99,6 +99,14 @@ SmartAI Corp achieved a score of 0.6 on the MLPerf™ Image Classification bench **Required Footnote**: “[1]Result verified by MLCommons Association. MLPerf™ name and logo are trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.” ____ +or + +____ +SmartAI Corp achieved a score of 1.2 on the MLPerf™ NLP benchmark using a SmartCluster with 8 chips in the Available category of Closed Division which is faster than the result of 7.2 achieved by LessSmartAI Corp with 16 chips from HardwareVendorX in the Available on-premise category of Closed Division.[1] + +**Required Footnote**: “[1]Result verified by MLCommons Association. MLPerf™ name and logo are trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.” +____ + Furthermore, a comparison of an unverified result with a verified result must include the following statement in a footnote: “_Unverified results have not been through an MLPerf™ review and may use measurement methodologies and/or workload implementations that are inconsistent with the MLPerf™ specification for verified results._” **Example (applicable to Non-MLCommons Reviewed Result)**: @@ -206,7 +214,7 @@ The MLPerf mark is owned by MLCommons, and only MLCommons and its authorized lic Any MLCommons Member, Test Partner, or third party may report a violation of these Guidelines via email to the MLCommons Executive Director (“ED”) & Working Group (“WG”) chairs of the appropriate benchmark. Upon confirming the violation in their discretion, ED & WG chairs would inform the potential violator and request remedial action. If the ED, WG chairs, and potential violator are unable to reach a mutually satisfactory conclusion, the issue can be raised in WG to seek resolution via WG vote. -Violating content must be taken down within first 48 hours of violation being reported. A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. +A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark. 1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media. 2. If the violation was at a public event such as a conference, the WG may direct the violator to issue a public statement to correct claims in ways that conform to these Guidelines. From 75439b988d062fbfd3ea38073b57ddcc8381375a Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Wed, 19 Apr 2023 17:26:30 -0700 Subject: [PATCH 12/12] Update MLPerf_Results_Messaging_Guidelines.adoc --- MLPerf_Results_Messaging_Guidelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MLPerf_Results_Messaging_Guidelines.adoc b/MLPerf_Results_Messaging_Guidelines.adoc index 7d381a8..ccc22df 100644 --- a/MLPerf_Results_Messaging_Guidelines.adoc +++ b/MLPerf_Results_Messaging_Guidelines.adoc @@ -81,7 +81,7 @@ MLPerf results may not be compared against non-MLPerf results. For example, an M == When comparing MLPerf results, you must identify any submission differences -When comparing results the main text, table, or figure must clearly identify any difference in version, division, category, verified or unverified status, scenario or chip count (count of the compute devices executing the largest number of ops, which could be processors or accelerators) or submitter name. When comparing Open and Closed division results, any ways in which the Open result would not qualify as a Closed result must be identified. When making comparisons, submissions must not be portrayed as representing the performance available from an Independent Hardware Vendor (IHV) unless the submission was by that IHV. +When comparing results the main text, table, or figure must clearly identify any difference in version, division, category, verified or unverified status, scenario or chip count (count of the compute devices executing the largest number of ops, which could be processors or accelerators) or submitter name. When comparing Open and Closed division results, any ways in which the Open result would not qualify as a Closed result must be identified. When making comparisons, submissions must not be portrayed as representing the performance available from a submitter unless the submission was by that submitter - e.g. a submission by SuperServers Inc that happened to use an accelerator from AccelCorp Ltd must not be portrayed as representing AccelCorp's performance. **Example for Non-MLCommons Reviewed Result**: