【New IR】backward code of new ir #55957
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e/Paddle into vjp_for_new_ir
… vjp_for_new_ir
xiaoguoguo626807
changed the title
cxxly
reviewed
Aug 9, 2023
python/paddle/ir/__init__.py
Outdated
| @@ -40,4 +41,7 @@ | |||
| 'OpResult', | |||
| 'Type', | |||
| 'translate_to_new_ir', | |||
| 'grad', | |||
| 'calc_gradient', | |||
| 'calc_gradient_helper', | |||
cxxly
reviewed
Aug 9, 2023
python/paddle/ir/backward.py
Outdated
| from collections.abc import Sequence | ||
|
|
||
| import paddle.ir | ||
| from paddle.fluid.core import call_vjp, has_vjp |
There was a problem hiding this comment.
使用paddle.framework.core 吧，fluid core逻辑已经迁移到 framework下了
cxxly
reviewed
Aug 9, 2023
python/paddle/ir/backward.py
Outdated
| return complete_outputs, complete_gradoutputs, backward_ops | ||
|
|
||
|
|
||
| def some_in_set(value_list, value_set): |
There was a problem hiding this comment.
该算法语义貌似本质上是判断两个集合交集是否为空，用集合操作可能更直观一些
if set(operand2value(src_operand) ) & set(operand2value(dst_operand)):
return True
return FalseThere was a problem hiding this comment.
good suggestion
cxxly
reviewed
Aug 9, 2023
python/paddle/ir/backward.py
Outdated
| def inverse_sort_op(ops): | ||
| ''' | ||
| if topo graph is op1 -> op2 -> op3 | ||
| return [ops, op2, op1] |
cxxly
reviewed
Aug 9, 2023
python/paddle/ir/backward.py
Outdated
| return sorted_list | ||
|
|
||
|
|
||
| def append_backward_ops( |
|
backward.py相关逻辑放到autograd目录下是否更合适？个人认为，自动微分和IR不是强绑定关系 |
Aurelius84
reviewed
Aug 9, 2023
python/paddle/ir/backward.py
Outdated
| for input in inputs: | ||
| if input.get_defining_op().get_parent_block() != block: | ||
| raise ValueError( | ||
| "all inputts must be in the same block with outputs" |
There was a problem hiding this comment.
Suggested change
| "all inputts must be in the same block with outputs" | |
| "all inputs must be in the same block with outputs" |
python/paddle/ir/backward.py
Outdated
| ) | ||
|
|
||
|
|
||
| def update_no_grad_set_stopgradient(block, no_grad_set): |
There was a problem hiding this comment.
Suggested change
| def update_no_grad_set_stopgradient(block, no_grad_set): | |
| def update_no_grad_set(block, no_grad_set): |
There was a problem hiding this comment.
there are two update_no_grad_set function, so add postfix to distinguish details
python/paddle/ir/backward.py
Outdated
| no_grad_set.add(value) | ||
|
|
||
|
|
||
| def update_all_structure(backward_ops, op_to_opgrad_list, gradop): |
There was a problem hiding this comment.
grad_op 是不是比gradop 更可读一些？
python/paddle/ir/backward.py
Outdated
| ): | ||
| """ | ||
| if grad_outputs is none, add fill_1 op to create grad_outputs | ||
| else check if outputs shape and dtype is same to grad_outputs, else raise error |
There was a problem hiding this comment.
Suggested change
| else check if outputs shape and dtype is same to grad_outputs, else raise error | |
| else check whether outputs shape and dtype is same to grad_outputs, otherwise raise error |
python/paddle/ir/backward.py
Outdated
|
|
||
| if len(grad_outputs) != len(outputs): | ||
| raise ValueError( | ||
| "Should have the same number of grad_outputs as outputs" |
There was a problem hiding this comment.
Suggested change
| "Should have the same number of grad_outputs as outputs" | |
| "grad_outputs should have the same length as outputs." |
python/paddle/ir/backward.py
Outdated
| from outputs to inputs add value not in the path to no_grad_set, | ||
| ''' | ||
| inputs_set = set(inputs) | ||
| if inputs_set != []: |
There was a problem hiding this comment.
Suggested change
| if inputs_set != []: | |
| if inputs_set: |
python/paddle/ir/backward.py
Outdated
| no_grad_set.add(value) | ||
|
|
||
| outputs_set = set(outputs) | ||
| no_grad_set_tmp = set() |
There was a problem hiding this comment.
这里为什么要单独定义一个tmp，for 循环里直接 no_grad_set.add 就可以?
There was a problem hiding this comment.
no_grad_set 在循环中会被用到，不能边修改边使用
python/paddle/ir/backward.py
Outdated
| add grad_op in order of topological sort | ||
| ''' | ||
| for op in effective_forward_op: | ||
| if has_vjp(op): |
There was a problem hiding this comment.
一个函数体不易过大，可以把if-else分支的逻辑封装成单独的函数
There was a problem hiding this comment.
此处逻辑会配合多输入多输出算子做调整，后续统一整理
python/paddle/ir/backward.py
Outdated
| 0.0, | ||
| dtype=value.dtype, | ||
| ) | ||
| fillop = block.ops[len(block.ops) - 1] |
python/paddle/ir/backward.py
Outdated
| if state.value_to_valuegrad[output] != []: | ||
| inputs_set.add(state.value_to_valuegrad[output][0][0]) | ||
|
|
||
| no_gradvar_set = set() # no_grad_set 中前向对应的反向变量 |
Ligoml
approved these changes
Aug 9, 2023
tianshuo78520a
approved these changes
Aug 9, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Description
修改block.get_ops() 函数调用接口为 block.ops
新ir 自动微分模块初步开发：
完成x -> tanh -> mean -> out 子图的反向构建过程。初步验证检查逻辑；反向组网逻辑。
接口：
临时调用接口paddle.ir.grad 待验证完备后合并至paddle.grad接口
TODO:
buildin combine + concat op. 对应concat vjp = concat gradop + split
split op + buildin split. 对应split vjp = combine + split gradop
3.待 验证剪枝逻辑 + no_grad_set用法
Others
pcard-67164