Skip to content

Commit

Permalink
Benchmarks: Unify metric names of benchmarks (#252)
Browse files Browse the repository at this point in the history 
**Description**
Unify metric names of benchmarks.
  • Loading branch information
Yuting Jiang committed Dec 9, 2021
1 parent c13ed2a commit 9f56b21
Show file tree
Hide file tree
Showing 44 changed files with 435 additions and 413 deletions.
24 changes: 12 additions & 12 deletions docs/design-docs/benchmarks.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -238,22 +238,22 @@ result = {
'run_count': N,
'return_code': 0,
'raw_data': {
'throughput-train-float32': [[step1_time, ..., stepK_time], ..., […]],
'throughput-train-float16': [[step1_time, ..., stepK_time], ..., […]],
'throughput-inference-float32': [[step1_time, ..., stepK_time], ..., […]],
'throughput-inference-float16': [[step1_time, ..., stepK_time], ..., […]],
'fp32_train_throughput': [[step1_throughput, ..., stepK_throughput], ..., […]],
'fp16_train_throughput': [[step1_throughput, ..., stepK_throughput], ..., […]],
'fp32_inference_throughput': [[step1_throughput, ..., stepK_throughput], ..., […]],
'fp16_inference_throughput': [[step1_throughput, ..., stepK_throughput], ..., […]],
},
'result': {
'throughput-train-float32': [avg_throughput1, ..., avg_throughputN],
'throughput-train-float16': [avg_throughput1, ..., avg_throughputN],
'throughput-inference-float32': [avg_throughput1, ..., avg_throughputN],
'throughput-inference-float16': [avg_throughput1, ..., avg_throughputN],
'fp32_train_throughput': [avg_throughput1, ..., avg_throughputN],
'fp16_train_throughput': [avg_throughput1, ..., avg_throughputN],
'fp32_inference_throughput': [avg_throughput1, ..., avg_throughputN],
'fp16_inference_throughput': [avg_throughput1, ..., avg_throughputN],
},
'reduce_op': {
'throughput-train-float32': 'min',
'throughput-train-float16': 'min',
'throughput-inference-float32': None,
'throughput-inference-float16': None,
'fp32_train_throughput': 'min',
'fp16_train_throughput': 'min',
'fp32_inference_throughput': None,
'fp16_inference_throughput': None,
},
}
```
Expand Down
24 changes: 12 additions & 12 deletions docs/user-tutorial/benchmarks/docker-benchmarks.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,15 @@ Run the rocm onnxruntime model training benchmarks packaged in docker `superbenc

#### Metrics

| Name | Unit | Description |
|-------------------------------------------------------|------------------------|-----------------------------------------------------------|
| onnxruntime-ort-models/bert_large_uncased_ngpu_1 | throughput (samples/s) | The throughput of bert large uncased model on 1 GPU. |
| onnxruntime-ort-models/bert_large_uncased_ngpu_8 | throughput (samples/s) | The throughput of bert large uncased model on 8 GPU. |
| onnxruntime-ort-models/distilbert_base_uncased_ngpu_1 | throughput (samples/s) | The throughput of distilbert base uncased model on 1 GPU. |
| onnxruntime-ort-models/distilbert_base_uncased_ngpu_8 | throughput (samples/s) | The throughput of distilbert base uncased model on 8 GPU. |
| onnxruntime-ort-models/gpt2_ngpu_1 | throughput (samples/s) | The throughput of gpt2 model on 1 GPU. |
| onnxruntime-ort-models/gpt2_ngpu_8 | throughput (samples/s) | The throughput of gpt2 model on 8 GPU. |
| onnxruntime-ort-models/facebook_bart_large_ngpu_1 | throughput (samples/s) | The throughput of facebook bart large model on 1 GPU. |
| onnxruntime-ort-models/facebook_bart_large_ngpu_8 | throughput (samples/s) | The throughput of facebook bart large model on 8 GPU. |
| onnxruntime-ort-models/roberta_large_ngpu_1 | throughput (samples/s) | The throughput of roberta large model on 1 GPU. |
| onnxruntime-ort-models/roberta_large_ngpu_8 | throughput (samples/s) | The throughput of roberta large model on 8 GPU. |
| Name | Unit | Description |
|------------------------------------------------------------------------|------------------------|-----------------------------------------------------------|
| onnxruntime-ort-models/bert_large_uncased_ngpu_1_train_throughput | throughput (samples/s) | The throughput of bert large uncased model on 1 GPU. |
| onnxruntime-ort-models/bert_large_uncased_ngpu_8_train_throughput | throughput (samples/s) | The throughput of bert large uncased model on 8 GPU. |
| onnxruntime-ort-models/distilbert_base_uncased_ngpu_1_train_throughput | throughput (samples/s) | The throughput of distilbert base uncased model on 1 GPU. |
| onnxruntime-ort-models/distilbert_base_uncased_ngpu_8_train_throughput | throughput (samples/s) | The throughput of distilbert base uncased model on 8 GPU. |
| onnxruntime-ort-models/gpt2_ngpu_1_train_throughput | throughput (samples/s) | The throughput of gpt2 model on 1 GPU. |
| onnxruntime-ort-models/gpt2_ngpu_8_train_throughput | throughput (samples/s) | The throughput of gpt2 model on 8 GPU. |
| onnxruntime-ort-models/facebook_bart_large_ngpu_1_train_throughput | throughput (samples/s) | The throughput of facebook bart large model on 1 GPU. |
| onnxruntime-ort-models/facebook_bart_large_ngpu_8_train_throughput | throughput (samples/s) | The throughput of facebook bart large model on 8 GPU. |
| onnxruntime-ort-models/roberta_large_ngpu_1_train_throughput | throughput (samples/s) | The throughput of roberta large model on 1 GPU. |
| onnxruntime-ort-models/roberta_large_ngpu_8_train_throughput | throughput (samples/s) | The throughput of roberta large model on 8 GPU. |

0 comments on commit 9f56b21

Please sign in to comment.