Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add speed test in ci #5496

Merged
merged 23 commits into from
Jul 21, 2021
Merged

add speed test in ci #5496

merged 23 commits into from
Jul 21, 2021

Conversation

daquexian
Copy link
Contributor

@daquexian daquexian commented Jul 14, 2021 •
edited
Loading

在 ci 里对比和 pytorch 的相对速度，慢于一个阈值时报错

@daquexian
add speed test in ci
e7f62c7 
Signed-off-by: daquexian <daquexian566@gmail.com>
@oneflow-ci-bot oneflow-ci-bot removed their request for review July 14, 2021 21:46
@daquexian
use actions/checkout@v2
d2c0a75 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian
add more input size
3e2d678 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 15, 2021 02:02
@daquexian daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 15, 2021 07:11
@daquexian
Merge remote-tracking branch 'origin/master' into ci_speed_test
af2e43e 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian
update test.yml to use oneflow-inc/models master
b65ffd1 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian
only clone models repo in cuda_new_interface
36dd89f 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 16, 2021 08:57
@daquexian
Merge remote-tracking branch 'origin/master' into ci_speed_test
2e845d0 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian
update test_speed.sh to check rel speed
3e5fde0 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian changed the title [WIP] add speed test in ci add speed test in ci Jul 18, 2021
jackalcooper
jackalcooper reviewed Jul 18, 2021
View reviewed changes
ci/test_multi_client/test_speed.sh

cd $ONEFLOW_MODEL_DIR

function check_relative_speed {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个脚本考虑用python写？这样可以方便之后把这个数据上传到云端汇总，也方便脚本的维护。

@github-actions
Copy link
Contributor 

<details>
<summary>Speed stats:</summary>

```
PyTorch resnet50 time: 0.14173929691314696 (= 2.8347859382629395 / 20, input_shape=[16, 3, 224, 224], backward is enabled)

OneFlow resnet50 time: 0.12600611448287963 (= 2.5201222896575928 / 20, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.1248604680402632 (= 0.14173929691314696 / 0.12600611448287963

PyTorch resnet50 time: 0.08470697402954101 (= 1.6941394805908203 / 20, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.07367438077926636 (= 1.4734876155853271 / 20, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.1497480281962476 (= 0.08470697402954101 / 0.07367438077926636

PyTorch resnet50 time: 0.058075356483459475 (= 1.1615071296691895 / 20, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.05648525953292847 (= 1.1297051906585693 / 20, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.028150653173578 (= 0.058075356483459475 / 0.05648525953292847

PyTorch resnet50 time: 0.04933222532272339 (= 0.9866445064544678 / 20, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.051469063758850096 (= 1.029381275177002 / 20, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 0.9584830521468486 (= 0.04933222532272339 / 0.051469063758850096

PyTorch resnet50 time: 0.04698525667190552 (= 0.9397051334381104 / 20, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.05037866830825806 (= 1.0075733661651611 / 20, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 0.9326418948673105 (= 0.04698525667190552 / 0.05037866830825806
```

</details>

@daquexian
collapse speed stats
0b169f8 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 21, 2021 04:11
@github-actions
Copy link
Contributor

Speed stats: 
PyTorch resnet50 time: 0.13908222913742066 (= 2.781644582748413 / 20, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.12625898122787477 (= 2.525179624557495 / 20, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.1015630554344664 (= 0.13908222913742066 / 0.12625898122787477

PyTorch resnet50 time: 0.08417870998382568 (= 1.6835741996765137 / 20, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.07348083257675171 (= 1.4696166515350342 / 20, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.14558731892293 (= 0.08417870998382568 / 0.07348083257675171

PyTorch resnet50 time: 0.05918576717376709 (= 1.1837153434753418 / 20, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.048660409450531 (= 0.9732081890106201 / 20, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.216302284384523 (= 0.05918576717376709 / 0.048660409450531

PyTorch resnet50 time: 0.04971511363983154 (= 0.9943022727966309 / 20, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.04637584686279297 (= 0.9275169372558594 / 20, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.0720044377177216 (= 0.04971511363983154 / 0.04637584686279297

PyTorch resnet50 time: 0.045691609382629395 (= 0.9138321876525879 / 20, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.04797089099884033 (= 0.9594178199768066 / 20, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 0.9524861521486846 (= 0.045691609382629395 / 0.04797089099884033

@oneflow-ci-bot oneflow-ci-bot removed their request for review July 21, 2021 05:12
@daquexian
fix comments
dd432a9 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian
set fixed commit and refine msg
a1f447c 
Signed-off-by: daquexian <daquexian566@gmail.com>
@github-actions
Copy link
Contributor

Speed stats: 
PyTorch resnet50 time: 0.13912445306777954 (= 2.782489061355591 / 20, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.12573709487915039 (= 2.514741897583008 / 20, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.1064710314923065 (= 0.13912445306777954 / 0.12573709487915039)

PyTorch resnet50 time: 0.08320977687835693 (= 1.6641955375671387 / 20, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.07331347465515137 (= 1.4662694931030273 / 20, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.1349861300361952 (= 0.08320977687835693 / 0.07331347465515137)

PyTorch resnet50 time: 0.06398413181304932 (= 1.2796826362609863 / 20, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.048484301567077635 (= 0.9696860313415527 / 20, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.3196876049565816 (= 0.06398413181304932 / 0.048484301567077635)

PyTorch resnet50 time: 0.04668329954147339 (= 0.9336659908294678 / 20, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.045202243328094485 (= 0.9040448665618896 / 20, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.032765104214604 (= 0.04668329954147339 / 0.045202243328094485)

PyTorch resnet50 time: 0.046851587295532224 (= 0.9370317459106445 / 20, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 0.04562036991119385 (= 0.912407398223877 / 20, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.0269883253190428 (= 0.046851587295532224 / 0.04562036991119385)

@daquexian daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 21, 2021 07:09
@daquexian
test 30 times instead of 20, relax the threshold
f9bcc62 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 21, 2021 08:06
@github-actions
Copy link
Contributor

Speed stats: 
GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 141.3ms (= 2825.9ms / 20, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 126.3ms (= 2525.6ms / 20, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.12 (= 141.3ms / 126.3ms)

PyTorch resnet50 time: 83.8ms (= 1676.8ms / 20, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 73.6ms (= 1472.3ms / 20, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.14 (= 83.8ms / 73.6ms)

PyTorch resnet50 time: 58.8ms (= 1176.2ms / 20, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 50.2ms (= 1003.6ms / 20, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.17 (= 58.8ms / 50.2ms)

PyTorch resnet50 time: 46.0ms (= 919.4ms / 20, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 53.4ms (= 1067.3ms / 20, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 0.86 (= 46.0ms / 53.4ms)

PyTorch resnet50 time: 45.9ms (= 918.7ms / 20, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 46.5ms (= 930.7ms / 20, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 0.99 (= 45.9ms / 46.5ms)

@oneflow-ci-bot oneflow-ci-bot removed their request for review July 21, 2021 09:22
@daquexian
update oneflow-models
17f5d8f 
Signed-off-by: daquexian <daquexian566@gmail.com>
@github-actions
Copy link
Contributor

Speed stats: 
GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 140.8ms (= 4225.1ms / 30, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 126.4ms (= 3792.7ms / 30, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 140.8ms / 126.4ms)

PyTorch resnet50 time: 85.2ms (= 2555.5ms / 30, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 73.5ms (= 2205.8ms / 30, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.16 (= 85.2ms / 73.5ms)

PyTorch resnet50 time: 64.8ms (= 1945.2ms / 30, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 50.3ms (= 1507.9ms / 30, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.29 (= 64.8ms / 50.3ms)

PyTorch resnet50 time: 50.3ms (= 1508.2ms / 30, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 59.1ms (= 1772.1ms / 30, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 0.85 (= 50.3ms / 59.1ms)

PyTorch resnet50 time: 49.5ms (= 1484.6ms / 30, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 55.3ms (= 1658.4ms / 30, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 0.90 (= 49.5ms / 55.3ms)

@oneflow-ci-bot oneflow-ci-bot removed their request for review July 21, 2021 10:57
@oneflow-ci-bot oneflow-ci-bot self-requested a review July 21, 2021 11:38
@github-actions
Copy link
Contributor

Speed stats: 
GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 139.7ms (= 4190.5ms / 30, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 125.9ms (= 3778.4ms / 30, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 139.7ms / 125.9ms)

PyTorch resnet50 time: 84.4ms (= 2531.9ms / 30, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 73.3ms (= 2199.2ms / 30, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.15 (= 84.4ms / 73.3ms)

PyTorch resnet50 time: 55.2ms (= 1654.7ms / 30, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 48.6ms (= 1458.3ms / 30, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.13 (= 55.2ms / 48.6ms)

PyTorch resnet50 time: 44.4ms (= 1333.1ms / 30, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 45.2ms (= 1355.4ms / 30, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 0.98 (= 44.4ms / 45.2ms)

PyTorch resnet50 time: 45.2ms (= 1355.2ms / 30, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.6ms (= 1429.5ms / 30, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 0.95 (= 45.2ms / 47.6ms)

@oneflow-ci-bot oneflow-ci-bot merged commit 7775595 into master Jul 21, 2021
@oneflow-ci-bot oneflow-ci-bot deleted the ci_speed_test branch July 21, 2021 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jackalcooper jackalcooper jackalcooper approved these changes

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
automerge ci feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@daquexian @jackalcooper @oneflow-ci-bot