ddp broadcast params and buffers #5913

daquexian · 2021-08-16T12:57:18Z

ddp 初始化时广播 0 号卡的 parameters，每次迭代前广播 0 号卡的 buffers（和 pytorch 对齐）

Signed-off-by: daquexian <daquexian566@gmail.com>

oneflow/core/framework/op_interpreter/eager_mirrored_op_interpreter.cpp

github-actions · 2021-08-18T02:20:43Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 140.5ms (= 7025.5ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 128.1ms (= 6404.6ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.10 (= 140.5ms / 128.1ms)

PyTorch resnet50 time: 84.5ms (= 4226.6ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 74.4ms (= 3719.3ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.14 (= 84.5ms / 74.4ms)

PyTorch resnet50 time: 57.4ms (= 2872.2ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.6ms (= 2380.3ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.21 (= 57.4ms / 47.6ms)

PyTorch resnet50 time: 48.9ms (= 2444.3ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 40.1ms (= 2002.8ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.22 (= 48.9ms / 40.1ms)

PyTorch resnet50 time: 41.8ms (= 2092.2ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 40.5ms (= 2027.3ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.03 (= 41.8ms / 40.5ms)

github-actions · 2021-08-18T06:17:30Z

CI failed, removing label automerge

github-actions · 2021-08-19T14:46:12Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 139.7ms (= 6986.7ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 128.0ms (= 6399.0ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.09 (= 139.7ms / 128.0ms)

PyTorch resnet50 time: 85.8ms (= 4287.9ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 74.5ms (= 3727.5ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.15 (= 85.8ms / 74.5ms)

PyTorch resnet50 time: 57.5ms (= 2872.9ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.8ms (= 2388.4ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.20 (= 57.5ms / 47.8ms)

PyTorch resnet50 time: 46.4ms (= 2318.4ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 40.4ms (= 2018.4ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.15 (= 46.4ms / 40.4ms)

PyTorch resnet50 time: 43.5ms (= 2175.1ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 35.3ms (= 1762.7ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.23 (= 43.5ms / 35.3ms)

* disable backward pass consistent tensor meta check. (#5871) * disable backward pass consistent tensor meta check. * auto format by CI Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * ddp broadcast params and buffers (#5913) * ddp broadcast params and buffers Signed-off-by: daquexian <daquexian566@gmail.com> * auto format by CI Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * add clang tidy target (#5957) * add clang tidy target * fix a bug * refine * refine * reformat Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * cfg: add move assignment operator for performance (#5962) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * add zhangshen op-test (#5600) * add some op-test * fix dims_error in my branch * Fix the bad backward kernel function by using 'cuda::atomic::Add' (#5614) * Test `nn.AdaptiveAvgPoolXd` (#5615) * Fix the bad backward kernel function by using 'cuda::atomic::Add' * Support the 'NoneType' annotation * Support objects of 'collections.abc.Iterable' as 'output_size' * Test with all cases of 'output_size' * Update adaptive_pool_gpu_kernel.cu * Skip testing `nn.AdaptiveAvgPool3d` for the current PyTorch * remove some useless test * Format TODO * Add the assertion messages for 'output_size' * Reformat codes * Remove raw tests for `flow.negative` * Remove unnecessary codes and add the assertion messages * Merge updates for 'generators.py' from master * Remove unnecessary 'random()' * Delete the separate test for `AvgPool2d` * Fix import paths * Fix import problems * Remove the PyTorch import * Denote the annotations for `tile` and `repeat` ops * Add the test for `nn.AvgPool1d` * Choose better generators for `nn.MaxPoolXd` * Randomly choose `dilation` and default values * auto format by CI * Test more kwargs for `nn.AvgPoolXd` * Add tests for `return_indices` * auto format by CI Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * fix wrong names (#5951) * fix wrong names * auto format by CI * refine * auto format by CI Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Enable more checkers for clang-tidy in CI (#5738) * CI: enable more checkers for clang-tidy * .clang-tidy: remove cppcoreguidelines-pro-type-vararg * CI: remove duplicate checkers * CI: remove clang-analyzer-alpha.deadcode.* * .clang-tidy: add performance-* * oneflow/core/eager: remove unnecessary malloc & free * .clang-tidy: add clang-analyzer-cplusplus.* to werror * user_kernel: remove useless move * quantization_aware_training: fix move return * .clang-tidy: add google-* * CI: fix clang tidy command * CI: fix test Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Feat grad mode classes (#5956) * feat(no_grad): support no_grad decorator * feat(AutogradMode): export flow.autograd_mode * feat(GradMode): export some grad_mode class * docs(GradMode): export documents * refine * docs(GradMode): export document for is_grad_enabled * auto format by CI * fix(GradMode): fix single client bug * fix bug Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * extract_consistent_to_consistent_op_expr (#5870) * abstract_consistent_to_consistent_op_expr * fix compiler complaint * refactor consistent-to-consistent eager consisitent op interpreter * fix compiler complaint * refactor ConsistentToConsistentOpExpr * lazy interpreter (#5903) * fix bugs about consistent_id * refactor functional::ToConsistent * refactor GetNdSbp * Update eager_consistent_op_interpreter.cpp * Update eager_mirrored_op_interpreter.cpp * fix error * fix error * auto format by CI * Update nd_sbp.h * refine identity boxing * fix sync checkmeta error * avoid consistent id check in lazy Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache (#5970) * add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache * skip test targets of re2 Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * check: fix clang-tidy-diff commands (#5972) * check: fix clang-tidy-diff commands * CI: fix step names Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cpu mpi (#5865) * cuda base cpu mpi boxing * cpu_mpi * fix conflicts * add cpu mpi unittests * more checks and unittests * abstract_consistent_to_consistent_op_expr * fix compiler complaint * refactor consistent-to-consistent eager consisitent op interpreter * fix compiler complaint * refactor ConsistentToConsistentOpExpr * lazy interpreter (#5903) * fix bugs about consistent_id * more test_consistent_cast unittests * refactor functional::ToConsistent * refactor GetNdSbp * fix compiler complaints * refactor GetDevice4CurrentProcessCtx * fix error Co-authored-by: clackhan <han_binbin@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * fix_bug_test_tensor_str (#5958) * fix bug int test_tensor_str * format * fix comment * fix bug to(cuda) is unavailable in cpu env Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * common/error: fix build error in mac (#5971) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Prevent running oneflow in forked subprocess (#5976) * prevent_running_oneflow_in_forked_subprocess * add line change * IsFork => IsForkedSubProcess * auto format by CI Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * refine randint Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Peihong Liu <mosout@qq.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com> Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com> Co-authored-by: liufengwei0103 <2472937968@qq.com>

* add randint * add * add doc test * Update randint_kernel.cu * Update randint_kernel.cpp * Update randint_op.cpp * reconstrcut * refine the code * add test * add test * add test * format * Dev randint refine (#5981) * disable backward pass consistent tensor meta check. (#5871) * disable backward pass consistent tensor meta check. * auto format by CI Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * ddp broadcast params and buffers (#5913) * ddp broadcast params and buffers Signed-off-by: daquexian <daquexian566@gmail.com> * auto format by CI Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * add clang tidy target (#5957) * add clang tidy target * fix a bug * refine * refine * reformat Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * cfg: add move assignment operator for performance (#5962) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * add zhangshen op-test (#5600) * add some op-test * fix dims_error in my branch * Fix the bad backward kernel function by using 'cuda::atomic::Add' (#5614) * Test `nn.AdaptiveAvgPoolXd` (#5615) * Fix the bad backward kernel function by using 'cuda::atomic::Add' * Support the 'NoneType' annotation * Support objects of 'collections.abc.Iterable' as 'output_size' * Test with all cases of 'output_size' * Update adaptive_pool_gpu_kernel.cu * Skip testing `nn.AdaptiveAvgPool3d` for the current PyTorch * remove some useless test * Format TODO * Add the assertion messages for 'output_size' * Reformat codes * Remove raw tests for `flow.negative` * Remove unnecessary codes and add the assertion messages * Merge updates for 'generators.py' from master * Remove unnecessary 'random()' * Delete the separate test for `AvgPool2d` * Fix import paths * Fix import problems * Remove the PyTorch import * Denote the annotations for `tile` and `repeat` ops * Add the test for `nn.AvgPool1d` * Choose better generators for `nn.MaxPoolXd` * Randomly choose `dilation` and default values * auto format by CI * Test more kwargs for `nn.AvgPoolXd` * Add tests for `return_indices` * auto format by CI Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * fix wrong names (#5951) * fix wrong names * auto format by CI * refine * auto format by CI Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Enable more checkers for clang-tidy in CI (#5738) * CI: enable more checkers for clang-tidy * .clang-tidy: remove cppcoreguidelines-pro-type-vararg * CI: remove duplicate checkers * CI: remove clang-analyzer-alpha.deadcode.* * .clang-tidy: add performance-* * oneflow/core/eager: remove unnecessary malloc & free * .clang-tidy: add clang-analyzer-cplusplus.* to werror * user_kernel: remove useless move * quantization_aware_training: fix move return * .clang-tidy: add google-* * CI: fix clang tidy command * CI: fix test Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Feat grad mode classes (#5956) * feat(no_grad): support no_grad decorator * feat(AutogradMode): export flow.autograd_mode * feat(GradMode): export some grad_mode class * docs(GradMode): export documents * refine * docs(GradMode): export document for is_grad_enabled * auto format by CI * fix(GradMode): fix single client bug * fix bug Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * extract_consistent_to_consistent_op_expr (#5870) * abstract_consistent_to_consistent_op_expr * fix compiler complaint * refactor consistent-to-consistent eager consisitent op interpreter * fix compiler complaint * refactor ConsistentToConsistentOpExpr * lazy interpreter (#5903) * fix bugs about consistent_id * refactor functional::ToConsistent * refactor GetNdSbp * Update eager_consistent_op_interpreter.cpp * Update eager_mirrored_op_interpreter.cpp * fix error * fix error * auto format by CI * Update nd_sbp.h * refine identity boxing * fix sync checkmeta error * avoid consistent id check in lazy Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache (#5970) * add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache * skip test targets of re2 Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * check: fix clang-tidy-diff commands (#5972) * check: fix clang-tidy-diff commands * CI: fix step names Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cpu mpi (#5865) * cuda base cpu mpi boxing * cpu_mpi * fix conflicts * add cpu mpi unittests * more checks and unittests * abstract_consistent_to_consistent_op_expr * fix compiler complaint * refactor consistent-to-consistent eager consisitent op interpreter * fix compiler complaint * refactor ConsistentToConsistentOpExpr * lazy interpreter (#5903) * fix bugs about consistent_id * more test_consistent_cast unittests * refactor functional::ToConsistent * refactor GetNdSbp * fix compiler complaints * refactor GetDevice4CurrentProcessCtx * fix error Co-authored-by: clackhan <han_binbin@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * fix_bug_test_tensor_str (#5958) * fix bug int test_tensor_str * format * fix comment * fix bug to(cuda) is unavailable in cpu env Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * common/error: fix build error in mac (#5971) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Prevent running oneflow in forked subprocess (#5976) * prevent_running_oneflow_in_forked_subprocess * add line change * IsFork => IsForkedSubProcess * auto format by CI Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * refine randint Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Peihong Liu <mosout@qq.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com> Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com> Co-authored-by: liufengwei0103 <2472937968@qq.com> * refine * refine * auto format by CI * refine * Update functional_api.yaml * Update functional_api.yaml * refine the code * auto format by CI * refine * fix ci error * fix test * auto format by CI * fixtest * refine code * auto format by CI * refine code * auto format by CI * fix ci fail * remove redefination api * fix ci test * auto format by CI * fix consistency with torch * auto format by CI * unittest fixed * fix doctest Co-authored-by: Bowen Chen <bob2420083992@gmail.com> Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Peihong Liu <mosout@qq.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com> Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com> Co-authored-by: liufengwei0103 <2472937968@qq.com>

* add randint * add * add doc test * Update randint_kernel.cu * Update randint_kernel.cpp * Update randint_op.cpp * reconstrcut * refine the code * add test * add test * add test * format * Dev randint refine (#5981) * disable backward pass consistent tensor meta check. (#5871) * disable backward pass consistent tensor meta check. * auto format by CI Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * ddp broadcast params and buffers (#5913) * ddp broadcast params and buffers Signed-off-by: daquexian <daquexian566@gmail.com> * auto format by CI Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * add clang tidy target (#5957) * add clang tidy target * fix a bug * refine * refine * reformat Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * cfg: add move assignment operator for performance (#5962) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * add zhangshen op-test (#5600) * add some op-test * fix dims_error in my branch * Fix the bad backward kernel function by using 'cuda::atomic::Add' (#5614) * Test `nn.AdaptiveAvgPoolXd` (#5615) * Fix the bad backward kernel function by using 'cuda::atomic::Add' * Support the 'NoneType' annotation * Support objects of 'collections.abc.Iterable' as 'output_size' * Test with all cases of 'output_size' * Update adaptive_pool_gpu_kernel.cu * Skip testing `nn.AdaptiveAvgPool3d` for the current PyTorch * remove some useless test * Format TODO * Add the assertion messages for 'output_size' * Reformat codes * Remove raw tests for `flow.negative` * Remove unnecessary codes and add the assertion messages * Merge updates for 'generators.py' from master * Remove unnecessary 'random()' * Delete the separate test for `AvgPool2d` * Fix import paths * Fix import problems * Remove the PyTorch import * Denote the annotations for `tile` and `repeat` ops * Add the test for `nn.AvgPool1d` * Choose better generators for `nn.MaxPoolXd` * Randomly choose `dilation` and default values * auto format by CI * Test more kwargs for `nn.AvgPoolXd` * Add tests for `return_indices` * auto format by CI Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * fix wrong names (#5951) * fix wrong names * auto format by CI * refine * auto format by CI Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Enable more checkers for clang-tidy in CI (#5738) * CI: enable more checkers for clang-tidy * .clang-tidy: remove cppcoreguidelines-pro-type-vararg * CI: remove duplicate checkers * CI: remove clang-analyzer-alpha.deadcode.* * .clang-tidy: add performance-* * oneflow/core/eager: remove unnecessary malloc & free * .clang-tidy: add clang-analyzer-cplusplus.* to werror * user_kernel: remove useless move * quantization_aware_training: fix move return * .clang-tidy: add google-* * CI: fix clang tidy command * CI: fix test Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Feat grad mode classes (#5956) * feat(no_grad): support no_grad decorator * feat(AutogradMode): export flow.autograd_mode * feat(GradMode): export some grad_mode class * docs(GradMode): export documents * refine * docs(GradMode): export document for is_grad_enabled * auto format by CI * fix(GradMode): fix single client bug * fix bug Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * extract_consistent_to_consistent_op_expr (#5870) * abstract_consistent_to_consistent_op_expr * fix compiler complaint * refactor consistent-to-consistent eager consisitent op interpreter * fix compiler complaint * refactor ConsistentToConsistentOpExpr * lazy interpreter (#5903) * fix bugs about consistent_id * refactor functional::ToConsistent * refactor GetNdSbp * Update eager_consistent_op_interpreter.cpp * Update eager_mirrored_op_interpreter.cpp * fix error * fix error * auto format by CI * Update nd_sbp.h * refine identity boxing * fix sync checkmeta error * avoid consistent id check in lazy Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache (#5970) * add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache * skip test targets of re2 Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * check: fix clang-tidy-diff commands (#5972) * check: fix clang-tidy-diff commands * CI: fix step names Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cpu mpi (#5865) * cuda base cpu mpi boxing * cpu_mpi * fix conflicts * add cpu mpi unittests * more checks and unittests * abstract_consistent_to_consistent_op_expr * fix compiler complaint * refactor consistent-to-consistent eager consisitent op interpreter * fix compiler complaint * refactor ConsistentToConsistentOpExpr * lazy interpreter (#5903) * fix bugs about consistent_id * more test_consistent_cast unittests * refactor functional::ToConsistent * refactor GetNdSbp * fix compiler complaints * refactor GetDevice4CurrentProcessCtx * fix error Co-authored-by: clackhan <han_binbin@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * fix_bug_test_tensor_str (#5958) * fix bug int test_tensor_str * format * fix comment * fix bug to(cuda) is unavailable in cpu env Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * common/error: fix build error in mac (#5971) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Prevent running oneflow in forked subprocess (#5976) * prevent_running_oneflow_in_forked_subprocess * add line change * IsFork => IsForkedSubProcess * auto format by CI Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> * refine randint Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Peihong Liu <mosout@qq.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com> Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com> Co-authored-by: liufengwei0103 <2472937968@qq.com> * refine * refine * auto format by CI * refine * Update functional_api.yaml * Update functional_api.yaml * refine the code * auto format by CI * refine * fix ci error * replace np randint with flow.randint * add tensor.item tensor.tolist * add test case * refine * auto format by CI * fix ci fail * revert change * auto format by CI * refine * auto format by CI Co-authored-by: Kevin-XiongC <kevin_xiong1997@outlook.com> Co-authored-by: Bowen Chen <bob2420083992@gmail.com> Co-authored-by: Li Xinqi <lixinqi2010@gmail.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Peihong Liu <mosout@qq.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com> Co-authored-by: Tianyu Zhao <guikarist@gmail.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: Xinqi Li <lixinqi0703106@163.com> Co-authored-by: leaves-zwx <kunta0932@gmail.com> Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com> Co-authored-by: liufengwei0103 <2472937968@qq.com>

ddp broadcast params and buffers

21bb91c

Signed-off-by: daquexian <daquexian566@gmail.com>

Ldpe2G reviewed Aug 17, 2021

View reviewed changes

oneflow/core/framework/op_interpreter/eager_mirrored_op_interpreter.cpp Show resolved Hide resolved

Ldpe2G approved these changes Aug 17, 2021

View reviewed changes

daquexian added automerge bug python system labels Aug 18, 2021

Merge branch 'master' into ddp_broadcast

0898a1d

daquexian requested a review from oneflow-ci-bot August 18, 2021 01:09

Merge branch 'master' into ddp_broadcast

4a19c47

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 18, 2021 01:09

auto format by CI

54e4111

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 18, 2021 01:12

oneflow-ci-bot removed their request for review August 18, 2021 02:22

Merge branch 'master' into ddp_broadcast

c70fcb8

oneflow-ci-bot self-requested a review August 18, 2021 02:59

Merge branch 'master' into ddp_broadcast

47b4360

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 18, 2021 03:53

Merge branch 'master' into ddp_broadcast

93337b0

oneflow-ci-bot self-requested a review August 18, 2021 04:57

github-actions bot removed the automerge label Aug 18, 2021

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 18, 2021 06:19

oneflow-ci-bot removed their request for review August 18, 2021 07:58

Merge branch 'master' into ddp_broadcast

649f02b

oneflow-ci-bot self-requested a review August 19, 2021 03:40

Merge branch 'master' into ddp_broadcast

f7d3dbb

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 19, 2021 04:39

Merge branch 'master' into ddp_broadcast

5b28c4e

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 19, 2021 05:49

Merge branch 'master' into ddp_broadcast

fa71e4c

oneflow-ci-bot self-requested a review August 19, 2021 07:17

Merge branch 'master' into ddp_broadcast

91cc184

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 19, 2021 08:14

Merge branch 'master' into ddp_broadcast

f34fcda

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 19, 2021 09:33

Merge branch 'master' into ddp_broadcast

54d235f

oneflow-ci-bot self-requested a review August 19, 2021 11:14

Merge branch 'master' into ddp_broadcast

35032b7

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 19, 2021 13:44

oneflow-ci-bot removed their request for review August 19, 2021 14:53

oneflow-ci-bot merged commit 71e2623 into master Aug 19, 2021

oneflow-ci-bot deleted the ddp_broadcast branch August 19, 2021 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddp broadcast params and buffers #5913

ddp broadcast params and buffers #5913

daquexian commented Aug 16, 2021

github-actions bot commented Aug 18, 2021

github-actions bot commented Aug 18, 2021

github-actions bot commented Aug 19, 2021

ddp broadcast params and buffers #5913

ddp broadcast params and buffers #5913

Conversation

daquexian commented Aug 16, 2021

github-actions bot commented Aug 18, 2021

github-actions bot commented Aug 18, 2021

github-actions bot commented Aug 19, 2021