impl of reflection_pad1d and replication_pad1d #8724

Flowingsun007 · 2022-07-21T15:00:28Z

重构pad2d_kernels.cpp，独立出reflection_pad_kernels、replication_pad_kernels
新增relfection_pad1d相关module、op/kernel、test case
新增replication_pad1d相关module、op/kernel、test case；by:@zhongshsh

github-actions · 2022-07-22T04:29:30Z

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

…/oneflow into dev_support_reflection_pad1d

github-actions · 2022-07-22T04:33:44Z

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

github-actions · 2022-07-25T10:46:51Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.4ms (= 12944.1ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 144.8ms (= 14478.7ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.12 (= 144.8ms / 129.4ms)

OneFlow resnet50 time: 76.1ms (= 7607.6ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.6ms (= 8561.9ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.13 (= 85.6ms / 76.1ms)

OneFlow resnet50 time: 48.2ms (= 9649.6ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.9ms (= 11585.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.20 (= 57.9ms / 48.2ms)

OneFlow resnet50 time: 36.9ms (= 7377.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 43.6ms (= 8722.0ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.18 (= 43.6ms / 36.9ms)

OneFlow resnet50 time: 32.3ms (= 6451.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 37.1ms (= 7417.7ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.15 (= 37.1ms / 32.3ms)

OneFlow swin dataloader time: 0.282s (= 56.309s / 200, num_workers=1)
PyTorch swin dataloader time: 0.151s (= 30.241s / 200, num_workers=1)
Relative speed: 0.537 (= 0.151s / 0.282s)

OneFlow swin dataloader time: 0.080s (= 16.069s / 200, num_workers=4)
PyTorch swin dataloader time: 0.042s (= 8.473s / 200, num_workers=4)
Relative speed: 0.527 (= 0.042s / 0.080s)

OneFlow swin dataloader time: 0.045s (= 9.067s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.474s / 200, num_workers=8)
Relative speed: 0.493 (= 0.022s / 0.045s)

❌ OneFlow resnet50 time: 145.1ms (= 14507.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 169.8ms (= 16977.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 169.8ms / 145.1ms)

OneFlow resnet50 time: 94.5ms (= 9449.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 113.1ms (= 11310.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 113.1ms / 94.5ms)

OneFlow resnet50 time: 67.2ms (= 13443.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 97.0ms (= 19405.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 97.0ms / 67.2ms)

OneFlow resnet50 time: 56.2ms (= 11245.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 83.3ms (= 16665.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.48 (= 83.3ms / 56.2ms)

OneFlow resnet50 time: 49.4ms (= 9884.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.6ms (= 15512.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.57 (= 77.6ms / 49.4ms)

github-actions · 2022-07-25T13:51:48Z

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

github-actions · 2022-07-25T14:25:44Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8724/

github-actions · 2022-07-25T14:26:35Z

Speed stats:

github-actions · 2022-07-25T15:51:01Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8724/

github-actions · 2022-07-25T15:55:27Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.6ms (= 12955.6ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 143.4ms (= 14342.0ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.11 (= 143.4ms / 129.6ms)

OneFlow resnet50 time: 75.8ms (= 7578.5ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.3ms (= 8534.0ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.13 (= 85.3ms / 75.8ms)

OneFlow resnet50 time: 48.4ms (= 9680.1ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 59.3ms (= 11866.2ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.23 (= 59.3ms / 48.4ms)

OneFlow resnet50 time: 37.0ms (= 7401.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 41.8ms (= 8362.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.13 (= 41.8ms / 37.0ms)

OneFlow resnet50 time: 31.5ms (= 6302.5ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 35.1ms (= 7025.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.11 (= 35.1ms / 31.5ms)

OneFlow swin dataloader time: 0.420s (= 84.062s / 200, num_workers=1)
PyTorch swin dataloader time: 0.149s (= 29.835s / 200, num_workers=1)
Relative speed: 0.355 (= 0.149s / 0.420s)

OneFlow swin dataloader time: 0.076s (= 15.274s / 200, num_workers=4)
PyTorch swin dataloader time: 0.042s (= 8.357s / 200, num_workers=4)
Relative speed: 0.547 (= 0.042s / 0.076s)

OneFlow swin dataloader time: 0.043s (= 8.585s / 200, num_workers=8)
PyTorch swin dataloader time: 0.024s (= 4.722s / 200, num_workers=8)
Relative speed: 0.550 (= 0.024s / 0.043s)

❌ OneFlow resnet50 time: 146.2ms (= 14615.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 171.3ms (= 17126.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 171.3ms / 146.2ms)

OneFlow resnet50 time: 94.1ms (= 9414.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.5ms (= 11247.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 112.5ms / 94.1ms)

OneFlow resnet50 time: 67.4ms (= 13480.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 88.2ms (= 17648.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 88.2ms / 67.4ms)

OneFlow resnet50 time: 56.3ms (= 11250.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 80.6ms (= 16111.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.43 (= 80.6ms / 56.3ms)

OneFlow resnet50 time: 50.3ms (= 10058.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 73.4ms (= 14675.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.46 (= 73.4ms / 50.3ms)

github-actions · 2022-07-26T02:27:24Z

CI failed when running job: cpu-module. PR label automerge has been removed

github-actions · 2022-07-26T02:28:31Z

Speed stats:

github-actions · 2022-07-26T02:57:14Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8724/

github-actions · 2022-07-26T02:59:40Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.7ms (= 12967.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 143.9ms (= 14393.0ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.11 (= 143.9ms / 129.7ms)

OneFlow resnet50 time: 75.9ms (= 7592.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.9ms (= 8391.4ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.11 (= 83.9ms / 75.9ms)

OneFlow resnet50 time: 48.4ms (= 9686.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.3ms (= 11453.1ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.18 (= 57.3ms / 48.4ms)

OneFlow resnet50 time: 36.8ms (= 7363.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 42.4ms (= 8486.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.15 (= 42.4ms / 36.8ms)

OneFlow resnet50 time: 31.2ms (= 6234.9ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.7ms (= 7733.3ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.24 (= 38.7ms / 31.2ms)

OneFlow swin dataloader time: 0.281s (= 56.265s / 200, num_workers=1)
PyTorch swin dataloader time: 0.158s (= 31.630s / 200, num_workers=1)
Relative speed: 0.562 (= 0.158s / 0.281s)

OneFlow swin dataloader time: 0.079s (= 15.799s / 200, num_workers=4)
PyTorch swin dataloader time: 0.045s (= 9.093s / 200, num_workers=4)
Relative speed: 0.576 (= 0.045s / 0.079s)

OneFlow swin dataloader time: 0.043s (= 8.655s / 200, num_workers=8)
PyTorch swin dataloader time: 0.023s (= 4.516s / 200, num_workers=8)
Relative speed: 0.522 (= 0.023s / 0.043s)

❌ OneFlow resnet50 time: 145.0ms (= 14503.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 169.8ms (= 16976.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 169.8ms / 145.0ms)

OneFlow resnet50 time: 94.0ms (= 9404.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 116.7ms (= 11672.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.24 (= 116.7ms / 94.0ms)

OneFlow resnet50 time: 66.8ms (= 13357.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 90.2ms (= 18032.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 90.2ms / 66.8ms)

OneFlow resnet50 time: 55.8ms (= 11157.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.2ms (= 15247.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 76.2ms / 55.8ms)

OneFlow resnet50 time: 50.7ms (= 10133.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.9ms (= 13987.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.38 (= 69.9ms / 50.7ms)

lixinqi · 2022-07-26T05:00:12Z

oneflow/core/functional/impl/nn_functor.cpp

@@ -2147,25 +2147,31 @@ class PadFunctor {
 public:
  PadFunctor() {


这个函数实现得太糟糕了。一个函数那么长，嵌套了那么多的if。这是奔向屎山的节奏。

这个函数实现得太糟糕了。一个函数那么长，嵌套了那么多的if。这是奔向屎山的节奏。

这个PadFunctor确实需要一些if判断，因为需要处理constant/reflection/replication pad的逻辑，以及对齐torch的相关check

必须在意代码的篇幅，拆成小函数之后才好看清代码的逻辑。

这个函数实现得太糟糕了。一个函数那么长，嵌套了那么多的if。这是奔向屎山的节奏。

这个PadFunctor确实需要一些if判断，因为需要处理constant/reflection/replication pad的逻辑，以及对齐torch的相关check

拆成小函数也不耽误这些逻辑的处理，也不会耽误对齐pytorch

好的，我后面拆分一下

Flowingsun007 added 8 commits July 21, 2022 14:57

impl of reflection_pad1d

f897ca9

Merge branch 'master' into dev_support_reflection_pad1d

4086978

refine

46c75a8

Merge branch 'master' into dev_support_reflection_pad1d

c544512

add check msg

a5fa320

Merge branch 'master' into dev_support_reflection_pad1d

3dd8b33

use MutOutputShape

8ebd542

align check to torch

164e2c2

Flowingsun007 marked this pull request as ready for review July 22, 2022 04:24

Flowingsun007 requested review from doombeaker, jackalcooper, hjchen2, BBuf, daquexian, liujuncheng, guo-ran and MARD1NO as code owners July 22, 2022 04:24

Flowingsun007 added enhancement eager op labels Jul 22, 2022

Flowingsun007 requested a review from oneflow-ci-bot July 22, 2022 04:28

auto format by CI

213e074

Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 22, 2022 04:30

Flowingsun007 and others added 4 commits July 22, 2022 12:31

Merge branch 'master' into dev_support_reflection_pad1d

c4532e6

refine

18e7918

Merge branch 'dev_support_reflection_pad1d' of github.com:Oneflow-Inc…

5c710cf

…/oneflow into dev_support_reflection_pad1d

auto format by CI

b5b301c

Flowingsun007 and others added 3 commits July 25, 2022 13:49

fix exception test

d488cfd

Merge branch 'master' into dev_support_reflection_pad1d

0ac82af

auto format by CI

5983149

Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 25, 2022 13:56

Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 25, 2022 14:31

Merge branch 'master' into dev_support_reflection_pad1d

0e1b908

Flowingsun007 added the automerge label Jul 26, 2022

Merge branch 'master' into dev_support_reflection_pad1d

a390f8b

Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 26, 2022 02:18

github-actions bot removed the automerge label Jul 26, 2022

Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 26, 2022 02:35

Flowingsun007 merged commit 7007647 into master Jul 26, 2022

Flowingsun007 deleted the dev_support_reflection_pad1d branch July 26, 2022 03:46

lixinqi reviewed Jul 26, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

impl of reflection_pad1d and replication_pad1d #8724

impl of reflection_pad1d and replication_pad1d #8724

Flowingsun007 commented Jul 21, 2022 •

edited

Loading

github-actions bot commented Jul 22, 2022

github-actions bot commented Jul 22, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 26, 2022

github-actions bot commented Jul 26, 2022

github-actions bot commented Jul 26, 2022

github-actions bot commented Jul 26, 2022

lixinqi Jul 26, 2022

Flowingsun007 Jul 26, 2022

lixinqi Jul 26, 2022

lixinqi Jul 26, 2022

Flowingsun007 Jul 26, 2022

		@@ -2147,25 +2147,31 @@ class PadFunctor {
		public:
		PadFunctor() {

impl of reflection_pad1d and replication_pad1d #8724

impl of reflection_pad1d and replication_pad1d #8724

Conversation

Flowingsun007 commented Jul 21, 2022 • edited Loading

github-actions bot commented Jul 22, 2022

github-actions bot commented Jul 22, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 26, 2022

github-actions bot commented Jul 26, 2022

github-actions bot commented Jul 26, 2022

github-actions bot commented Jul 26, 2022

lixinqi Jul 26, 2022

Choose a reason for hiding this comment

Flowingsun007 Jul 26, 2022

Choose a reason for hiding this comment

lixinqi Jul 26, 2022

Choose a reason for hiding this comment

lixinqi Jul 26, 2022

Choose a reason for hiding this comment

Flowingsun007 Jul 26, 2022

Choose a reason for hiding this comment

Flowingsun007 commented Jul 21, 2022 •

edited

Loading