Add fused_rope forward op #54351

niuliling123 · 2023-06-05T06:37:08Z

PR types

Others

PR changes

Others

Description

Pcard-70458
Others

paddle-bot · 2023-06-05T06:37:13Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Xreki

加一下单测

Xreki · 2023-06-19T01:52:39Z

paddle/phi/kernels/gpu/fused_rope_grad_kernel.cu

@@ -0,0 +1,160 @@
+// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.


融合的算子实现到phi/kernels/fusion/gpu目录下吧

Xreki · 2023-06-19T01:53:10Z

paddle/phi/kernels/gpu/fused_rope_grad_kernel.cu

+template <typename T, int VecSize>
+struct alignas(sizeof(T) * VecSize) VectorType {
+  T val[VecSize];
+};


为何不直接使用AlignedVector呢？

已经修改

Xreki · 2023-06-19T01:55:06Z

paddle/phi/kernels/gpu/fused_rope_grad_kernel.cu

+                   phi::dtype::bfloat16) {
+  kernel->InputAt(0).SetBackend(phi::Backend::ALL_BACKEND);
+  kernel->InputAt(1).SetBackend(phi::Backend::ALL_BACKEND);
+  kernel->InputAt(2).SetBackend(phi::Backend::ALL_BACKEND);


这里需要设置ALL_BACKEND吗？

已经删除

这里是删除后又加回来了？还有前向也是。

Xreki · 2023-06-19T01:55:41Z

paddle/phi/kernels/gpu/fused_rope_kernel.cu

+#include "paddle/phi/kernels/funcs/aligned_vector.h"
+
+namespace phi {
+template <typename T, int VecSize>


直接使用AlignedVector？

已经修改

Xreki · 2023-06-19T01:56:26Z

paddle/phi/kernels/gpu/fused_rope_kernel.cu

+                                          int C,
+                                          int main_offset,
+                                          phi::Array<T*, 3> outs_data,
+                                          int break_iter,


break_iter -> num_inputs？

Xreki · 2023-06-19T01:58:32Z

paddle/phi/kernels/gpu/fused_rope_kernel.cu

+  auto N = q.dims()[0];
+  auto H = q.dims()[1];
+  auto W = q.dims()[2];
+  auto C = q.dims()[3];


q是序列，它的四个维度含义分别是[batch_size, seq_len, num_heads, head_dim]，用维度含义来命名变量

已经修改

Xreki · 2023-06-19T01:59:40Z

python/paddle/tensor/math.py

@@ -621,6 +621,11 @@ def add(x, y, name=None):
        return _elementwise_op(LayerHelper('elementwise_add', **locals()))


+def fused_rope(q, k, v):
+    if in_dynamic_mode():
+        return _C_ops.fused_rope(q, k, v)


fused_rope API应该加到paddle.incubate.nn.functional下面比较合适吧，API名使用完整的rotary_position_embedding

已经修改

YuanRisheng · 2023-06-27T11:59:27Z

paddle/phi/api/yaml/legacy_ops.yaml

@@ -422,6 +422,17 @@
  optional : skip_update, master_params
  inplace : (params -> params_out), (moments1 -> moments1_out), (moments2 -> moments2_out), (beta1_pows -> beta1_pows_out), (beta2_pows -> beta2_pows_out), (master_params -> master_params_out)

+- op : fused_rope


这个可以放在fused_ops.yaml里

下个PR 再改

zyfncg · 2023-06-27T14:39:01Z

paddle/phi/infermeta/backward.h

@@ -459,5 +459,11 @@ void IndexAddGradInferMeta(const MetaTensor& index,
                           int axis,
                           MetaTensor* x_grad,
                           MetaTensor* add_tensor_grad);
+void FusedRopeGradInferMeta(const MetaTensor& dout_q,


函数按字典序放置

zyfncg · 2023-06-27T14:39:10Z

paddle/phi/infermeta/multiary.cc

@@ -3489,5 +3489,33 @@ void WeightedSampleNeighborsInferMeta(const MetaTensor& row,
  out_count->set_dims({-1});
  out_count->set_dtype(DataType::INT32);
 }
+


zyfncg · 2023-06-27T14:40:00Z

paddle/phi/kernels/fused_rope_grad_kernel.h

+namespace phi {
+
+template <typename T, typename Context>
+void FusedRopeGradKernel(const Context& dev_ctx,


fuse类型的kernel不用写头文件声明

下个PR统一修改

zyfncg · 2023-06-27T14:40:12Z

paddle/phi/kernels/fused_rope_kernel.h

+namespace phi {
+
+template <typename T, typename Context>
+void FusedRopeKernel(const Context& dev_ctx,


下个PR统一修改

jzhang533

LGTM

ZzSean

LGTM for skipIf

Xreki

LGTM. 一些review建议下个PR再改下

Xreki · 2023-06-29T05:12:31Z

paddle/phi/infermeta/backward.cc

+  PADDLE_ENFORCE_EQ(input_dims.size(),
+                    4,
+                    phi::errors::InvalidArgument(
+                        "Input should be a 4-D tensor of format [N, C, H, W] "


这里的N、C、H、W也统一改成实际含义吧，下同

Xreki · 2023-06-29T05:17:30Z

paddle/phi/kernels/gpu/fused_rope_grad_kernel.cu

+
+template <typename T, typename Context>
+void FusedRopeGradKernel(const Context& dev_ctx,
+                         const DenseTensor& dout_q,


我有点不太理解dout_q是啥意思，是前向的计算结果out_q吗？

就是反向传递过来的dout

Xreki · 2023-06-29T05:19:21Z

paddle/phi/kernels/gpu/fused_rope_grad_kernel.cu

+                   phi::dtype::bfloat16) {
+  kernel->InputAt(0).SetBackend(phi::Backend::ALL_BACKEND);
+  kernel->InputAt(1).SetBackend(phi::Backend::ALL_BACKEND);
+  kernel->InputAt(2).SetBackend(phi::Backend::ALL_BACKEND);


这里是删除后又加回来了？还有前向也是。

Xreki · 2023-06-29T05:22:12Z

python/paddle/incubate/nn/functional/fused_rotary_position_embedding.py

+    Fused rotary position embedding.
+
+    Args:
+        q (Tensor): The input tensor. The data type is bfloat16, float16, float32 or float64.


文档里面加一下输入Tensor shape的描述吧。

好的，下个PR 修改

Xreki · 2023-06-29T05:26:05Z

test/legacy_test/test_fused_rotary_position_embedding.py

+    indices = 1 / 10000 ** (indices / q.shape[3])
+    sinusoid_inp = pos_seq.unsqueeze(1) * indices.unsqueeze(0)
+
+    sin_sin = np.empty((q.shape[2] * q.shape[3]), dtype=np.float32)


为啥一部分计算用Paddle API、一部分计算用Numpy API呢？

因为要不一样呀

* style * more * update ctest * Update legacy_backward.yaml * Update legacy_ops.yaml * Update legacy_ops.yaml * update * update * update for move

* Add fused_rope forward op (#54351) * style * more * update ctest * Update legacy_backward.yaml * Update legacy_ops.yaml * Update legacy_ops.yaml * update * update * update for move * Update the rope op according to the comments (#54985) * Update multiary.cc * Update __init__.py * for int64_t and assert * more * remove useless assert first --------- Co-authored-by: sneaxiy <sneaxiy@126.com>

…addlePaddle#55931) * Add fused_rope forward op (PaddlePaddle#54351) * style * more * update ctest * Update legacy_backward.yaml * Update legacy_ops.yaml * Update legacy_ops.yaml * update * update * update for move * Update the rope op according to the comments (PaddlePaddle#54985) * Update multiary.cc * Update __init__.py * for int64_t and assert * more * remove useless assert first --------- Co-authored-by: sneaxiy <sneaxiy@126.com>

niuliling123 force-pushed the fuse_broadcast branch 2 times, most recently from 0b1089d to 8e174d1 Compare June 13, 2023 06:45

Xreki reviewed Jun 19, 2023

View reviewed changes

Xreki changed the title ~~Add fused_fope forward op~~ Add fused_rope forward op Jun 19, 2023

niuliling123 force-pushed the fuse_broadcast branch 4 times, most recently from a7e40fd to 80c112d Compare June 20, 2023 04:16

style

115cc40

niuliling123 force-pushed the fuse_broadcast branch from 80c112d to 115cc40 Compare June 20, 2023 04:19

more

4281348

niuliling123 force-pushed the fuse_broadcast branch from 731faf0 to 4281348 Compare June 20, 2023 04:52

update ctest

b143bd0

niuliling123 force-pushed the fuse_broadcast branch from 72829f9 to b143bd0 Compare June 25, 2023 06:21

niuliling123 and others added 6 commits June 26, 2023 13:33

Merge branch 'develop' into fuse_broadcast

48275da

Update legacy_backward.yaml

c19cb0d

Update legacy_ops.yaml

96a5dc6

Update legacy_ops.yaml

070966a

update

ecb663a

update

bba1968

YuanRisheng previously approved these changes Jun 27, 2023

View reviewed changes

zyfncg reviewed Jun 27, 2023

View reviewed changes

jzhang533 previously approved these changes Jun 28, 2023

View reviewed changes

ZzSean previously approved these changes Jun 28, 2023

View reviewed changes

zyfncg previously approved these changes Jun 28, 2023

View reviewed changes

Merge branch 'develop' into fuse_broadcast

7703138

niuliling123 dismissed stale reviews from zyfncg, ZzSean, and jzhang533 via 7703138 June 28, 2023 06:14

niuliling123 dismissed YuanRisheng’s stale review via 7703138 June 28, 2023 06:14

update for move

46ae391

YuanRisheng approved these changes Jun 28, 2023

View reviewed changes

ZzSean approved these changes Jun 28, 2023

View reviewed changes

jzhang533 approved these changes Jun 28, 2023

View reviewed changes

zyfncg approved these changes Jun 28, 2023

View reviewed changes

lanxianghit approved these changes Jun 29, 2023

View reviewed changes

Xreki approved these changes Jun 29, 2023

View reviewed changes

sneaxiy approved these changes Jun 29, 2023

View reviewed changes

sneaxiy merged commit a215c46 into PaddlePaddle:develop Jun 29, 2023
27 of 28 checks passed

		@@ -0,0 +1,160 @@
		// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.

Add fused_rope forward op #54351

Add fused_rope forward op #54351

Conversation

niuliling123 commented Jun 5, 2023 • edited

PR types

PR changes

Description

paddle-bot bot commented Jun 5, 2023

Xreki left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki Jun 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niuliling123 Jun 28, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niuliling123 Jun 28, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

ZzSean left a comment

Choose a reason for hiding this comment

Xreki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niuliling123 Jun 29, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niuliling123 commented Jun 5, 2023 •

edited

Xreki left a comment •

edited

Xreki Jun 19, 2023 •

edited

niuliling123 Jun 28, 2023 •

edited

niuliling123 Jun 28, 2023 •

edited

niuliling123 Jun 29, 2023 •

edited