Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feat.] nn.Graph support grad acc with input/output tensor #6155

Merged
merged 20 commits into from
Sep 8, 2021
Merged

[Feat.] nn.Graph support grad acc with input/output tensor #6155

merged 20 commits into from
Sep 8, 2021

Conversation

chengtbf
Copy link
Contributor

@chengtbf chengtbf commented Sep 3, 2021 •
edited
Loading

  • nn.Graph 支持 gradient accumulation
  • GradientAccumulationRewritePass 支持处理 Multi-Client 下的 Input/Output，约定给 Input 插入 Unpack，给 Output 插入 Pack；同时处理 Output 的 Scalar Pack 问题
  • Pack Op 支持处理 Scalar ，将其 stack 成为一个 1-D Tensor
  • nn.Graph Build OutputTensor 支持上述情况下的 Scalar Pack 操作
  • 讨论设计方案，支持 nn.Graph 延迟创建 output tensor，创建依据 经过 JobBuildAndInferCtx Complete 之后的真正的 shape
  • 添加单测跑通

@chengtbf chengtbf added the WIP work in progress label Sep 3, 2021
@chengtbf chengtbf added automerge bottleneck blocking another feature/PR feature interface system and removed WIP work in progress labels Sep 7, 2021
@chengtbf chengtbf marked this pull request as ready for review September 7, 2021 05:38
@chengtbf chengtbf requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 8, 2021 04:50
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 8, 2021 04:52
@github-actions github-actions bot removed the automerge label Sep 8, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Sep 8, 2021

CI failed, removing label automerge

@oneflow-ci-bot oneflow-ci-bot removed their request for review September 8, 2021 05:49
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 8, 2021 07:41
@github-actions
Copy link
Contributor

github-actions bot commented Sep 8, 2021

Speed stats: 
GPU Name: GeForce GTX 1080 

OneFlow resnet50 time: 128.4ms (= 6418.2ms / 50, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 139.9ms (= 6996.7ms / 50, input_shape=[16, 3, 224, 224])
Relative speed: 1.09 (= 139.9ms / 128.4ms)

OneFlow resnet50 time: 74.4ms (= 3722.0ms / 50, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.4ms (= 4268.5ms / 50, input_shape=[8, 3, 224, 224])
Relative speed: 1.15 (= 85.4ms / 74.4ms)

OneFlow resnet50 time: 50.9ms (= 2547.3ms / 50, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.9ms (= 2945.2ms / 50, input_shape=[4, 3, 224, 224])
Relative speed: 1.16 (= 58.9ms / 50.9ms)

OneFlow resnet50 time: 49.7ms (= 2483.1ms / 50, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 49.9ms (= 2497.1ms / 50, input_shape=[2, 3, 224, 224])
Relative speed: 1.01 (= 49.9ms / 49.7ms)

OneFlow resnet50 time: 35.4ms (= 1770.6ms / 50, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 43.3ms (= 2163.3ms / 50, input_shape=[1, 3, 224, 224])
Relative speed: 1.22 (= 43.3ms / 35.4ms)

OneFlow resnet50 time: 154.3ms (= 7713.1ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 157.8ms (= 7887.7ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
Relative speed: 1.02 (= 157.8ms / 154.3ms)

OneFlow resnet50 time: 103.0ms (= 5151.0ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 99.9ms (= 4993.5ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
Relative speed: 0.97 (= 99.9ms / 103.0ms)

OneFlow resnet50 time: 80.5ms (= 4022.6ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 73.2ms (= 3660.6ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
Relative speed: 0.91 (= 73.2ms / 80.5ms)

OneFlow resnet50 time: 69.3ms (= 3466.2ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.9ms (= 3342.7ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
Relative speed: 0.96 (= 66.9ms / 69.3ms)

OneFlow resnet50 time: 66.8ms (= 3338.4ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.0ms (= 3250.1ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
Relative speed: 0.97 (= 65.0ms / 66.8ms)

@oneflow-ci-bot oneflow-ci-bot merged commit ef98f72 into master Sep 8, 2021
@oneflow-ci-bot oneflow-ci-bot deleted the dev_cc_nngraph_grad_acc branch September 8, 2021 09:51
strint added a commit that referenced this pull request Sep 9, 2021
@strint @liujuncheng
@oneflow-ci-bot @clackhan @chengtbf @wyg1997 @Ikkyu321 @hengzi @PragmaTwice @daquexian @MARD1NO @Flowingsun007 @BBuf @hjchen2
Fea/checkpoint stop identity (#6216)
716cc69 
* Primitive (#6183)

* Add Primitive

* #ifdef WITH_CUDA

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Disable implicit boxing when parallel num eq one (#6188)

* mv_boxing_folder_to_core

* minor fix

* disable_implicit_boxing_when_parallel_num_eq_one

* Update eager_consistent_op_interpreter.cpp

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Lazy support Scalar (#6181)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix LayerNorm check bug (#6196)

* fix(Layernorm): fix check bug

* fix judge whether cpu or not

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add glu op (#6065)

* add glu op

* del glu_op export，align with torch

* mod glu_op

* mov op logic to C++

* Solve problems

* solve conflict

* delete gradient functor

* add ndim check

* add GLU test

* delete blank line

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>

* Primitive based copy task node (#6195)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* KernelState (#6198)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* container_util: fix VectorAt, remove useless MutMapAt (#6172)

* fcontainer_util: fix VectorAt, remove useless MutMapAt

* fcontainer_util: format

* MapAt: add default value version

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refine StreamContext (#6191)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cpu symetric s to s (#6153)

* mv_boxing_folder_to_core

* minor fix

* cpu_symetric_s_to_s

* add test case

* auto format by CI

* minor fix

* refine

* Update eager_nccl_kernels.cpp

* minor fix

* fix bug

* minor fix

* Update oneflow/user/kernels/eager_nccl_kernels.cpp

Co-authored-by: daquexian <daquexian566@gmail.com>

* Update eager_nccl_kernels.cpp

* Update eager_nccl_kernels.cpp

* minor fix

* Update eager_nccl_kernels.cpp

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>

* fix bug (#6197)

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix consistent tensor zeros (#6202)

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* [Feat.] nn.Graph support grad acc with input/output tensor (#6155)

* nn.Graph support grad acc with input/output tensor

* dirty pass grad acc

* revert tensor.backward hack

* fix indent

* default S0 -> B

* Pack op/kernel support scalar input

* nn.Graph output pack support loss scalar

* add test script

* pass test

* Lazy build output eager tensors after job complete

* non scalar output test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Dev eliminate gcc warnings (#6199)

* fix gcc warning

* refine

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* StreamContextAdapter (#6205)

* StreamContextAdapter

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Autotest generate input tensor (#6206)

* Add tensor yaml, support export tensor functional api.

* refine

* Remove packed functor signature

* remove unused file

* Refine

* refine

* add activation op import

* reinit oneflow init.py

* add oneflow abs and exp

* add oneflow abs and exp

* add acos

* add arccosh

* add more op

* add more ops

* add more op

* add more ops

* add log1p

* add more smaples

* add more ops

* add more ops

* add more ops

* add more ops

* Complete tensor functional apis.

* Fix pybind call

* add more ops

* add ops done

* Add target of_functional_tensor_obj

* Disable throw visibility warnings

* fix target link

* fix

* fix incorrect use of flow.Tensor.

* Fix error merge

* fix

* fix add unittest

* refine

* refine

* fix

* fix

* add tensor doc

* auto format by CI

* refine

* Fix

* Add doc for python function

* refine

* add tensor method docstring

* fix some bug

* fix docs bug

* Fix

* auto format by CI

* Tensor->tensor

* Tensor->tensor

* refine Tensor->tensor

* fix

* fix

* fix

* fix conflict

* fix bug

* fix ci bug

* fix

* delete diag op

* fix conflict

* Fix segment

* fix

* merge

* autotest framework generate input tensor

* autotest framework generate input tensor

* fix bug

* fix impl bug

* refine

* refine

* refine

* fix

* fix

* fix comments

* delete useless

* fix ci error

* fix ci error

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cleanup KernelUtil (#6212)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename flow to oneflow in user hint (#6190)

* style(*): rename flow to oneflow in user hint

* fix(*): fix doctest

* auto format by CI

* remove ddp speed test

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>

* merg and refactor

* refact code

* add io identity for activation checkpointing

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
oneflow-ci-bot added a commit that referenced this pull request Sep 10, 2021
@tingkuanpei @strint
@liujuncheng @oneflow-ci-bot @clackhan @chengtbf @wyg1997 @Ikkyu321 @hengzi @PragmaTwice @daquexian @MARD1NO @Flowingsun007 @BBuf @hjchen2
graph activation checkpointing (#6192)
1f354f1 
* Add test_graph_activation_checkpoint.py

* Fea/checkpoint stop identity (#6216)

* Primitive (#6183)

* Add Primitive

* #ifdef WITH_CUDA

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Disable implicit boxing when parallel num eq one (#6188)

* mv_boxing_folder_to_core

* minor fix

* disable_implicit_boxing_when_parallel_num_eq_one

* Update eager_consistent_op_interpreter.cpp

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Lazy support Scalar (#6181)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix LayerNorm check bug (#6196)

* fix(Layernorm): fix check bug

* fix judge whether cpu or not

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add glu op (#6065)

* add glu op

* del glu_op export，align with torch

* mod glu_op

* mov op logic to C++

* Solve problems

* solve conflict

* delete gradient functor

* add ndim check

* add GLU test

* delete blank line

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>

* Primitive based copy task node (#6195)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* KernelState (#6198)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* container_util: fix VectorAt, remove useless MutMapAt (#6172)

* fcontainer_util: fix VectorAt, remove useless MutMapAt

* fcontainer_util: format

* MapAt: add default value version

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refine StreamContext (#6191)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cpu symetric s to s (#6153)

* mv_boxing_folder_to_core

* minor fix

* cpu_symetric_s_to_s

* add test case

* auto format by CI

* minor fix

* refine

* Update eager_nccl_kernels.cpp

* minor fix

* fix bug

* minor fix

* Update oneflow/user/kernels/eager_nccl_kernels.cpp

Co-authored-by: daquexian <daquexian566@gmail.com>

* Update eager_nccl_kernels.cpp

* Update eager_nccl_kernels.cpp

* minor fix

* Update eager_nccl_kernels.cpp

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>

* fix bug (#6197)

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix consistent tensor zeros (#6202)

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* [Feat.] nn.Graph support grad acc with input/output tensor (#6155)

* nn.Graph support grad acc with input/output tensor

* dirty pass grad acc

* revert tensor.backward hack

* fix indent

* default S0 -> B

* Pack op/kernel support scalar input

* nn.Graph output pack support loss scalar

* add test script

* pass test

* Lazy build output eager tensors after job complete

* non scalar output test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Dev eliminate gcc warnings (#6199)

* fix gcc warning

* refine

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* StreamContextAdapter (#6205)

* StreamContextAdapter

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Autotest generate input tensor (#6206)

* Add tensor yaml, support export tensor functional api.

* refine

* Remove packed functor signature

* remove unused file

* Refine

* refine

* add activation op import

* reinit oneflow init.py

* add oneflow abs and exp

* add oneflow abs and exp

* add acos

* add arccosh

* add more op

* add more ops

* add more op

* add more ops

* add log1p

* add more smaples

* add more ops

* add more ops

* add more ops

* add more ops

* Complete tensor functional apis.

* Fix pybind call

* add more ops

* add ops done

* Add target of_functional_tensor_obj

* Disable throw visibility warnings

* fix target link

* fix

* fix incorrect use of flow.Tensor.

* Fix error merge

* fix

* fix add unittest

* refine

* refine

* fix

* fix

* add tensor doc

* auto format by CI

* refine

* Fix

* Add doc for python function

* refine

* add tensor method docstring

* fix some bug

* fix docs bug

* Fix

* auto format by CI

* Tensor->tensor

* Tensor->tensor

* refine Tensor->tensor

* fix

* fix

* fix

* fix conflict

* fix bug

* fix ci bug

* fix

* delete diag op

* fix conflict

* Fix segment

* fix

* merge

* autotest framework generate input tensor

* autotest framework generate input tensor

* fix bug

* fix impl bug

* refine

* refine

* refine

* fix

* fix

* fix comments

* delete useless

* fix ci error

* fix ci error

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cleanup KernelUtil (#6212)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename flow to oneflow in user hint (#6190)

* style(*): rename flow to oneflow in user hint

* fix(*): fix doctest

* auto format by CI

* remove ddp speed test

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>

* merg and refactor

* refact code

* add io identity for activation checkpointing

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>

* rm useless code

* refien test

* add skip

* import os

* fix test

* auto format by CI

Co-authored-by: strint <xiaoyulink@gmail.com>
Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
oneflow-ci-bot added a commit that referenced this pull request Sep 12, 2021
@strint @chengtbf
@tingkuanpei @oneflow-ci-bot @liujuncheng @clackhan @wyg1997 @Ikkyu321 @hengzi @PragmaTwice @daquexian @MARD1NO @Flowingsun007 @BBuf @hjchen2
Fea/pipeline in graph (#6105)
1b901ee 
* add demo

* add test draft

* debug pipeline graph

* add pp

* add

* support 0-d tensor spilt

* add repr of conf for debug

* add acc grad

* add tag

* hack stage id of to_consistent

* Pipeline buffer pass stage id check log

* FixPipelineStageIdPass interface

* nn.Graph support grad acc with input/output tensor

* dirty pass grad acc

* Lazy support Scalar

* add log

* revert tensor.backward hack

* fix indent

* default S0 -> B

* merge

* Pack op/kernel support scalar input

* nn.Graph output pack support loss scalar

* add test script

* add pack

* pass test

* Add test_graph_activation_checkpoint.py

* Lazy build output eager tensors after job complete

* non scalar output test

* stale input

* add debug

* add

* add test

* add test

* deal with empty kernel of_record tensor buffer

* add test

* refine test

* New scope by stage id

* rm debug log

* finish test

* half impl of stage id infer

* Fix pipeline stage id by close max group

* remove note

* remove debug code

* Fea/checkpoint stop identity (#6216)

* Primitive (#6183)

* Add Primitive

* #ifdef WITH_CUDA

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Disable implicit boxing when parallel num eq one (#6188)

* mv_boxing_folder_to_core

* minor fix

* disable_implicit_boxing_when_parallel_num_eq_one

* Update eager_consistent_op_interpreter.cpp

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Lazy support Scalar (#6181)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix LayerNorm check bug (#6196)

* fix(Layernorm): fix check bug

* fix judge whether cpu or not

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add glu op (#6065)

* add glu op

* del glu_op export，align with torch

* mod glu_op

* mov op logic to C++

* Solve problems

* solve conflict

* delete gradient functor

* add ndim check

* add GLU test

* delete blank line

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>

* Primitive based copy task node (#6195)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* KernelState (#6198)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* container_util: fix VectorAt, remove useless MutMapAt (#6172)

* fcontainer_util: fix VectorAt, remove useless MutMapAt

* fcontainer_util: format

* MapAt: add default value version

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refine StreamContext (#6191)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cpu symetric s to s (#6153)

* mv_boxing_folder_to_core

* minor fix

* cpu_symetric_s_to_s

* add test case

* auto format by CI

* minor fix

* refine

* Update eager_nccl_kernels.cpp

* minor fix

* fix bug

* minor fix

* Update oneflow/user/kernels/eager_nccl_kernels.cpp

Co-authored-by: daquexian <daquexian566@gmail.com>

* Update eager_nccl_kernels.cpp

* Update eager_nccl_kernels.cpp

* minor fix

* Update eager_nccl_kernels.cpp

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>

* fix bug (#6197)

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix consistent tensor zeros (#6202)

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* [Feat.] nn.Graph support grad acc with input/output tensor (#6155)

* nn.Graph support grad acc with input/output tensor

* dirty pass grad acc

* revert tensor.backward hack

* fix indent

* default S0 -> B

* Pack op/kernel support scalar input

* nn.Graph output pack support loss scalar

* add test script

* pass test

* Lazy build output eager tensors after job complete

* non scalar output test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Dev eliminate gcc warnings (#6199)

* fix gcc warning

* refine

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* StreamContextAdapter (#6205)

* StreamContextAdapter

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Autotest generate input tensor (#6206)

* Add tensor yaml, support export tensor functional api.

* refine

* Remove packed functor signature

* remove unused file

* Refine

* refine

* add activation op import

* reinit oneflow init.py

* add oneflow abs and exp

* add oneflow abs and exp

* add acos

* add arccosh

* add more op

* add more ops

* add more op

* add more ops

* add log1p

* add more smaples

* add more ops

* add more ops

* add more ops

* add more ops

* Complete tensor functional apis.

* Fix pybind call

* add more ops

* add ops done

* Add target of_functional_tensor_obj

* Disable throw visibility warnings

* fix target link

* fix

* fix incorrect use of flow.Tensor.

* Fix error merge

* fix

* fix add unittest

* refine

* refine

* fix

* fix

* add tensor doc

* auto format by CI

* refine

* Fix

* Add doc for python function

* refine

* add tensor method docstring

* fix some bug

* fix docs bug

* Fix

* auto format by CI

* Tensor->tensor

* Tensor->tensor

* refine Tensor->tensor

* fix

* fix

* fix

* fix conflict

* fix bug

* fix ci bug

* fix

* delete diag op

* fix conflict

* Fix segment

* fix

* merge

* autotest framework generate input tensor

* autotest framework generate input tensor

* fix bug

* fix impl bug

* refine

* refine

* refine

* fix

* fix

* fix comments

* delete useless

* fix ci error

* fix ci error

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cleanup KernelUtil (#6212)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename flow to oneflow in user hint (#6190)

* style(*): rename flow to oneflow in user hint

* fix(*): fix doctest

* auto format by CI

* remove ddp speed test

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>

* merg and refactor

* refact code

* add io identity for activation checkpointing

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>

* rm useless code

* refien test

* add skip

* Fix bug of Multi-Client src tick output order

* refactor

* test pass

* waning to info

* import os

* fix test

* auto format by CI

* refactor fix pipeline stage id pass algo

* add refine test

* insert identity when do pipeline

* refine test

* fix copy buffer

* refien pipeline demo

* format

* Add input/output ctrl edge to DstSubTick for order io and callback_notify

* add test scripts

* remove note

* add comment for using pipeling

* add comment

* rm useless to

* avoid cpu run cuda test

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: peitingkuan <peitingkuan@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com>
Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@strint strint strint approved these changes

@liujuncheng liujuncheng Awaiting requested review from liujuncheng

@leaves-zwx leaves-zwx Awaiting requested review from leaves-zwx

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
automerge bottleneck blocking another feature/PR feature interface system
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@chengtbf @strint @oneflow-ci-bot