New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmarks: Unify metric names of benchmarks #252
Conversation
Codecov Report
@@ Coverage Diff @@
## main #252 +/- ##
==========================================
+ Coverage 87.61% 87.67% +0.06%
==========================================
Files 70 70
Lines 3947 3967 +20
==========================================
+ Hits 3458 3478 +20
Misses 489 489
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
two general questions:
- do we need to have a unified metrics format, e.g.,
benchmark/description_type_statistic
? type can be time/bw/flops/count, etc., statistic can be min/max/avg, etc. - is it necessary to include the unit in the metrics? like
time_ms
, etc.
superbench/benchmarks/micro_benchmarks/ib_loopback_performance.py
Outdated
Show resolved
Hide resolved
superbench/benchmarks/micro_benchmarks/ib_validation_performance.py
Outdated
Show resolved
Hide resolved
docs/design-docs/benchmarks.md
Outdated
'throughput-train-fp32': [[step1_time, ..., stepK_time], ..., […]], | ||
'throughput-train-fp16': [[step1_time, ..., stepK_time], ..., […]], | ||
'throughput-inference-fp32': [[step1_time, ..., stepK_time], ..., […]], | ||
'throughput-inference-fp16': [[step1_time, ..., stepK_time], ..., […]], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change to avg_throughput
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
Description
Revise metric names of benchmarks.