2 false errors from submission-checker.py

[xgu3]

Just tried latest repo checker, and there’re 2 potential false errors.

1.ERROR:main:XXX/mobilenet/Offline/accuracy has no accuracy.txt. Generate it with accuracy-imagenet.py or accuracy-coco.py or process_accuracy.py
The rule spec doesn’t requires this accuracy.txt (only a json file is listed). The spec is here https://github.com/mlperf/policies/blob/master/submission_rules.adoc#inference

It seems the scrip expect we redirect the output of accuracy-imagenet.py to a txt file, guessing from here https://github.com/mlperf/inference/blob/master/SubmissionExample.ipynb

# accuracy run
 ./run_local.sh $@ --scenario $scenario --accuracy --output $output_dir/$scenario/accuracy
 python $cmd --mlperf-accuracy-file $output_dir/$scenario/accuracy/mlperf_log_accuracy.json \
         >  $output_dir/$scenario/accuracy/accuracy.txt
 cat $output_dir/$scenario/accuracy/accuracy.txt

So which one is correct, the rules or the script?

2.ERROR:main:XXX/resnet/Server/accuracy has file list mismatch ([u'mlperf_log_detail.txt', u'mlperf_log_summary.txt', u'accuracy.txt'])
It seems the script is looking for mlperf_log_detail.txt and mlperf_log_summary.txt, but per the rules, those 2 files shall be under /performance, like below,

performance/
•	run_x/ # 1 run for single stream and offline, 5 otherwise
o	mlperf_log_summary.txt
o	mlperf_log_detail.txt
o	mlperf_log_trace.json

I think the script shall be modified to follow rule spec.

And one more question, the rule requires compliance_checker_log.txt, is this just redirecting the output of submission-checker.py to a txt file?

@christ1ne @schmuell @guschmue

[aaronzhongii]

I raised the same question before as in #473 (comment)

But I think these are minor file adjustment issues. And Vijay has asked Peter to change the rule accordingly in last meeting.

[schmuell]

Yes, I know. Bunch of files that we'd want are not in the doc. I field a issue for it here: mlcommons/policies#6 There is no other record of the accuracy so it would be helpful for the reviewers to have accuracy.txt, Its not the end of the world if it is not in the submission ... I could bulk process this after the submission is closed.

…

On Thu, Oct 10, 2019 at 6:37 AM AARON ZHONG ***@***.***> wrote: I raised the same question before as in #473 (comment) <#473 (comment)> But I think these are minor file adjustment issues. And Vijay has asked Peter to change the rule accordingly in last meeting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#504>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATDG7U4CKPEIRAFBE7EX2LQN4VZRANCNFSM4I7KKCWA> .