【PIR Dist Op Reg No.28】 reg dgc #62781

xiaoyewww · 2024-03-16T15:56:46Z

PR Category

Execute Infrastructure

PR Types

Devs

Description

#60436
注册算子dgc

paddle-bot · 2024-03-16T15:56:51Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

kangguangli · 2024-03-21T03:05:17Z

paddle/fluid/pir/dialect/operator/ir/ops.yaml

@@ -454,6 +454,13 @@
  optional : in_accum, in_state, out_scale, out_accum, out_state
  inplace : (scale -> out_scale, in_accum -> out_accum, in_state -> out_state)

+- op : dgc
+  args : (Tensor u, Tensor v, Tensor grad, Tensor param, Tensor current_step, Tensor nranks, float[] sparsity, , float m=0.9, bool use_nesterov=true, float rampup_begin_step=0.0, float rampup_step=0.0, float regular_coeff=0.0, int regular_type=0)


多打了逗号

感谢已修改，请问一下新的报错是注册算子的时候还有地方写错了吗

kangguangli

从代码来看，

Paddle/paddle/fluid/operators/CMakeLists.txt

Lines 134 to 135 in 65126fa

    
           if (WITH_DGC) 
        
               op_library(dgc_op DEPS dgc)

这里应该是对应流水线没有开启WITH_DGC选项，需要修改CMakeLists，
在第30行前加入这样的判断：

if(NOT WITH_DGC)
  list(REMOVE_ITEM TEST_INTERP_CASES test_dgc_translate)
endif()

xiaoyewww · 2024-03-22T09:19:47Z

从代码来看，

Paddle/paddle/fluid/operators/CMakeLists.txt

Lines 134 to 135 in 65126fa

if (WITH_DGC)

op_library(dgc_op DEPS dgc)

这里应该是对应流水线没有开启WITH_DGC选项，需要修改CMakeLists，在第30行前加入这样的判断：
if(NOT WITH_DGC)
  list(REMOVE_ITEM TEST_INTERP_CASES test_dgc_translate)
endif()

多谢，已修改

test/ir/pir/translator/CMakeLists.txt

xiaoyewww · 2024-03-26T08:01:17Z

从代码来看，

Paddle/paddle/fluid/operators/CMakeLists.txt

Lines 134 to 135 in 65126fa

if (WITH_DGC)

op_library(dgc_op DEPS dgc)

这里应该是对应流水线没有开启WITH_DGC选项，需要修改CMakeLists，在第30行前加入这样的判断：
if(NOT WITH_DGC)
  list(REMOVE_ITEM TEST_INTERP_CASES test_dgc_translate)
endif()

这里这样修改后似乎还是遇到老问题

kangguangli · 2024-03-26T08:24:32Z

paddle/fluid/operators/CMakeLists.txt

+if(NOT WITH_DGC)
+  list(REMOVE_ITEM TEST_INTERP_CASES test_dgc_translator)
+endif()
+


加错地方了，加在test/ir/pir/translator/CMakeLists.txt里

抱歉，不知道怎么加在那边了

xiaoyewww · 2024-03-27T14:52:52Z

@kangguangli 这个麻烦再看一下，还是原来的问题

kangguangli · 2024-03-28T07:32:10Z

现在的报错是在翻译过程中报错：

2024-03-26 19:29:43 ======================================================================
2024-03-26 19:29:43 ERROR: test_translator (test_dgc_translator.TestDgcOpTranslator)
2024-03-26 19:29:43 ----------------------------------------------------------------------
2024-03-26 19:29:43 Traceback (most recent call last):
2024-03-26 19:29:43 File "/mnt/paddle/build/test/ir/pir/translator/test_dgc_translator.py", line 73, in test_translator
2024-03-26 19:29:43 self.check()
2024-03-26 19:29:43 File "/mnt/paddle/build/test/ir/pir/translator/test_op_translator.py", line 42, in check
2024-03-26 19:29:43 pir_program = pir.translate_to_pir(self.main_program.desc)
2024-03-26 19:29:43 ValueError:
2024-03-26 19:29:43
2024-03-26 19:29:43 --------------------------------------
2024-03-26 19:29:43 C++ Traceback (most recent call last):
2024-03-26 19:29:43 --------------------------------------
2024-03-26 19:29:43 0 paddle::TranslateLegacyProgramToProgram(paddle::framework::ProgramDesc const&)
2024-03-26 19:29:43 1 paddle::translator::ProgramTranslator::Translate()
2024-03-26 19:29:43 2 paddle::translator::ProgramTranslator::TranslateBlock(paddle::framework::BlockDesc const&, unsigned long, unsigned long, paddle::translator::TranslationContext*, pir::Block*)
2024-03-26 19:29:43 3 paddle::translator::ProgramTranslator::TranslateGeneralOperation(paddle::framework::OpDesc const*, paddle::translator::TranslationContext*, pir::Block*)
2024-03-26 19:29:43 4 std::_Function_handler<pir::Operation* (pir::IrContext*, paddle::translator::TranslationContext*, paddle::framework::OpDesc const&, pir::Block*), paddle::translator::OpTranscriber>::_M_invoke(std::_Any_data const&, pir::IrContext*&&, paddle::translator::TranslationContext*&&, paddle::framework::OpDesc const&, pir::Block*&&)
2024-03-26 19:29:43 5 paddle::translator::OpTranscriber::operator()(pir::IrContext*, paddle::translator::TranslationContext*, paddle::framework::OpDesc const&, pir::Block*)
2024-03-26 19:29:43 6 common::enforce::EnforceNotMet::EnforceNotMet(common::ErrorSummary const&, char const*, int)
2024-03-26 19:29:43 7 common::enforce::GetCurrentTraceBackStringabi:cxx11
2024-03-26 19:29:43
2024-03-26 19:29:43 ----------------------
2024-03-26 19:29:43 Error Message Summary:
2024-03-26 19:29:43 ----------------------
2024-03-26 19:29:43 InvalidArgumentError: Input param not found when parsing op dgc
2024-03-26 19:29:43 [Hint: Expected legacy_input_vars.size() == 1UL, but received legacy_input_vars.size():0 != 1UL:1.] (at /paddle/paddle/fluid/ir_adaptor/translator/op_translator.cc:560)

这个错误是说翻译的时候在旧IR算子中没有发现 param 这个输入，一般是由于param为空导致的，原因是旧IR下有些输入没有正确的声明optional，后续遇到此类问题，可以在算子的yaml里添加下面的声明试试。

   optional: param

xiaoyewww · 2024-03-28T14:53:08Z

现在的报错是在翻译过程中报错：

2024-03-26 19:29:43 ======================================================================
2024-03-26 19:29:43 ERROR: test_translator (test_dgc_translator.TestDgcOpTranslator)
2024-03-26 19:29:43 ----------------------------------------------------------------------
2024-03-26 19:29:43 Traceback (most recent call last):
2024-03-26 19:29:43 File "/mnt/paddle/build/test/ir/pir/translator/test_dgc_translator.py", line 73, in test_translator
2024-03-26 19:29:43 self.check()
2024-03-26 19:29:43 File "/mnt/paddle/build/test/ir/pir/translator/test_op_translator.py", line 42, in check
2024-03-26 19:29:43 pir_program = pir.translate_to_pir(self.main_program.desc)
2024-03-26 19:29:43 ValueError:
2024-03-26 19:29:43
2024-03-26 19:29:43 --------------------------------------
2024-03-26 19:29:43 C++ Traceback (most recent call last):
2024-03-26 19:29:43 --------------------------------------
2024-03-26 19:29:43 0 paddle::TranslateLegacyProgramToProgram(paddle::framework::ProgramDesc const&)
2024-03-26 19:29:43 1 paddle::translator::ProgramTranslator::Translate()
2024-03-26 19:29:43 2 paddle::translator::ProgramTranslator::TranslateBlock(paddle::framework::BlockDesc const&, unsigned long, unsigned long, paddle::translator::TranslationContext*, pir::Block*)
2024-03-26 19:29:43 3 paddle::translator::ProgramTranslator::TranslateGeneralOperation(paddle::framework::OpDesc const*, paddle::translator::TranslationContext*, pir::Block*)
2024-03-26 19:29:43 4 std::_Function_handler<pir::Operation* (pir::IrContext*, paddle::translator::TranslationContext*, paddle::framework::OpDesc const&, pir::Block*), paddle::translator::OpTranscriber>::_M_invoke(std::_Any_data const&, pir::IrContext*&&, paddle::translator::TranslationContext*&&, paddle::framework::OpDesc const&, pir::Block*&&)
2024-03-26 19:29:43 5 paddle::translator::OpTranscriber::operator()(pir::IrContext*, paddle::translator::TranslationContext*, paddle::framework::OpDesc const&, pir::Block*)
2024-03-26 19:29:43 6 common::enforce::EnforceNotMet::EnforceNotMet(common::ErrorSummary const&, char const*, int)
2024-03-26 19:29:43 7 common::enforce::GetCurrentTraceBackStringabi:cxx11
2024-03-26 19:29:43
2024-03-26 19:29:43 ----------------------
2024-03-26 19:29:43 Error Message Summary:
2024-03-26 19:29:43 ----------------------
2024-03-26 19:29:43 InvalidArgumentError: Input param not found when parsing op dgc
2024-03-26 19:29:43 [Hint: Expected legacy_input_vars.size() == 1UL, but received legacy_input_vars.size():0 != 1UL:1.] (at /paddle/paddle/fluid/ir_adaptor/translator/op_translator.cc:560)

这个错误是说翻译的时候在旧IR算子中没有发现 param 这个输入，一般是由于param为空导致的，原因是旧IR下有些输入没有正确的声明optional，后续遇到此类问题，可以在算子的yaml里添加下面的声明试试。
   optional: param

谢谢，已这样修改，但有点困惑，哪里可以看到旧IR算子中缺少这个输入呢

kangguangli · 2024-03-29T02:53:44Z

从报错提示中可以看出旧IR中缺少这个参数，如果对相关代码感兴趣，可以看看这个目录下的代码paddle/fluid/ir_adaptor/translator，PFCC仓库应该也有一份文档。

另外，PR模版现在有变更，麻烦按照 https://github.com/PaddlePaddle/Paddle/wiki/PULL-REQUEST-TEMPLATE--REFERENCE 修改一下，我们的任务应该属于执行架构（Execute Infrastructure）和框架本身开发的改动。

xiaoyewww · 2024-03-30T02:22:10Z

Details

多谢已修改

kangguangli · 2024-04-01T06:30:49Z

paddle/fluid/pir/dialect/operator/ir/ops.yaml

+  args : (Tensor u, Tensor v, Tensor grad, Tensor param, Tensor current_step, Tensor nranks, float[] sparsity, float m=0.9, bool use_nesterov=true, float rampup_begin_step=0.0, float rampup_step=0.0, float regular_coeff=0.0, int regular_type=0)
+  output : Tensor(u_out), Tensor(v_out), Tensor(encode_grad), Tensor(grad_out), Tensor(k), Tensor(gather_buff)
+  kernel :
+    func : dgc


Suggested change

func : dgc

func : dgc

param: [u, v, ...]

在kernel里面需要指定传给dgc_kernel参数，顺序和kernel参数一致，其他地方不用修改

多谢，已修改，请问一下这里为什么要添加呢

xiaoyewww · 2024-04-02T15:11:25Z

@kangguangli 请问这个pr模版有什么问题吗，按照格式修改过了？

kangguangli

PR模版需要把原来的注释部分删除

kangguangli · 2024-04-03T03:38:19Z

paddle/fluid/pir/dialect/operator/ir/ops.yaml

+  output : Tensor(u_out), Tensor(v_out), Tensor(encode_grad), Tensor(grad_out), Tensor(k), Tensor(gather_buff)
+  kernel :
+    func : dgc
+    param : [u, v, grad, param, current_step, nranks, sparsity, m, use_nesterov, rampup_begin_step, rampup_step, regular_coeff, regular_type]


这个顺序和kernel顺序不完全一致，再和paddle/phi/kernels/gpu/dgc_kernel.cu里的DGCKernel对照下

已修改，因为这里我看sparsity没有提到默认值所以提前了一下

xiaoyewww · 2024-04-08T14:07:24Z

@kangguangli 这个麻烦review一下，windows ci挂了

zyfncg · 2024-04-09T09:43:55Z

test/ir/pir/translator/test_dgc_translator.py

@@ -0,0 +1,77 @@
+# Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.


2023 -> 2024

* feat(pir): reg dgc * feat(pir): dgc * feat(pir): dgc * feat(pir): reg dgc * feat(pir): reg dgc * feat(pir): reg dgc * feat(pir): reg dgc * feat(pir): reg dgc * feat(pir): reg dgc

paddle-bot bot added the contributor External developers label Mar 16, 2024

xiaoyewww force-pushed the pir/dgc branch 2 times, most recently from 5b3b1bd to 2dc683d Compare March 17, 2024 04:36

luotao1 added the HappyOpenSource 快乐开源活动issue与PR label Mar 18, 2024

luotao1 mentioned this pull request Mar 18, 2024

【PIR】PIR下的分布式算子注册 #60436

Closed

luotao1 assigned luotao1 and kangguangli Mar 18, 2024

kangguangli reviewed Mar 21, 2024

View reviewed changes

kangguangli reviewed Mar 22, 2024

View reviewed changes

xiaoyewww force-pushed the pir/dgc branch from 84a95f4 to 036336a Compare March 22, 2024 14:42

kangguangli reviewed Mar 25, 2024

View reviewed changes

test/ir/pir/translator/CMakeLists.txt Show resolved Hide resolved

kangguangli reviewed Mar 26, 2024

View reviewed changes

kangguangli reviewed Apr 1, 2024

View reviewed changes

kangguangli reviewed Apr 3, 2024

View reviewed changes

xiaoyewww force-pushed the pir/dgc branch 3 times, most recently from f5be4ad to 68bf75c Compare April 4, 2024 02:59

xiaoyewww added 4 commits April 7, 2024 11:06

feat(pir): reg dgc

92b713c

feat(pir): dgc

2022d10

feat(pir): dgc

7c144f1

feat(pir): reg dgc

9fea079

xiaoyewww added 4 commits April 7, 2024 11:06

feat(pir): reg dgc

f25835e

feat(pir): reg dgc

213562d

feat(pir): reg dgc

90c1063

feat(pir): reg dgc

5010351

xiaoyewww force-pushed the pir/dgc branch from 68bf75c to 5010351 Compare April 7, 2024 11:06

feat(pir): reg dgc

0c91378

kangguangli approved these changes Apr 9, 2024

View reviewed changes

zyfncg approved these changes Apr 9, 2024

View reviewed changes

kangguangli merged commit 6adee41 into PaddlePaddle:develop Apr 9, 2024
30 checks passed

xiaoyewww deleted the pir/dgc branch May 10, 2024 15:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

【PIR Dist Op Reg No.28】 reg dgc #62781

【PIR Dist Op Reg No.28】 reg dgc #62781

xiaoyewww commented Mar 16, 2024 •

edited by kangguangli

Loading

paddle-bot bot commented Mar 16, 2024

kangguangli Mar 21, 2024

xiaoyewww Mar 21, 2024

kangguangli left a comment

xiaoyewww commented Mar 22, 2024

xiaoyewww commented Mar 26, 2024

kangguangli Mar 26, 2024

xiaoyewww Mar 26, 2024

xiaoyewww commented Mar 27, 2024

kangguangli commented Mar 28, 2024

xiaoyewww commented Mar 28, 2024

kangguangli commented Mar 29, 2024

xiaoyewww commented Mar 30, 2024

kangguangli Apr 1, 2024

xiaoyewww Apr 2, 2024

xiaoyewww commented Apr 2, 2024

kangguangli left a comment

kangguangli Apr 3, 2024

xiaoyewww Apr 4, 2024

xiaoyewww commented Apr 8, 2024

zyfncg Apr 9, 2024

		@@ -0,0 +1,77 @@
		# Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.

【PIR Dist Op Reg No.28】 reg dgc #62781

【PIR Dist Op Reg No.28】 reg dgc #62781

Conversation

xiaoyewww commented Mar 16, 2024 • edited by kangguangli Loading

PR Category

PR Types

Description

paddle-bot bot commented Mar 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kangguangli left a comment

Choose a reason for hiding this comment

xiaoyewww commented Mar 22, 2024

xiaoyewww commented Mar 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaoyewww commented Mar 27, 2024

kangguangli commented Mar 28, 2024

xiaoyewww commented Mar 28, 2024

kangguangli commented Mar 29, 2024

xiaoyewww commented Mar 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaoyewww commented Apr 2, 2024

kangguangli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaoyewww commented Apr 8, 2024

Choose a reason for hiding this comment

xiaoyewww commented Mar 16, 2024 •

edited by kangguangli

Loading