【PIR API adaptor No.51, 60】Migrate some ops into pir #58684

longranger2 · 2023-11-04T02:29:39Z

PR types

Others

PR changes

APIs

Description

PIR API 推全升级
将如下算子迁移升级至 pir，并更新单测

decode_jpeg：暂不支持
distribute_fpn_proposals（13/16）：

目前继承自 OpTest 的单测，在 pir 模式下暂不支持设置输入的 lod_level，因此TestDistributeFPNProposalsOp等三个单测可以先跳过
test_detection.py里的test_error 性质的单测目前暂时无法适配
新IR Python API适配升级 #58067

paddle-bot · 2023-11-04T02:29:49Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

paddle-ci-bot · 2023-11-12T03:23:09Z

Sorry to inform you that 3157aa1's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

MarioLulab · 2023-11-14T03:06:51Z

麻烦 merge 一下最新的分支~

test/legacy_test/test_distribute_fpn_proposals_op.py

@@ -142,7 +143,7 @@ def setUp(self):
        self.set_data()

    def test_check_output(self):
-        self.check_output(check_dygraph=False)
+        self.check_output(check_dygraph=False, check_pir=True)



longranger2 · 2023-11-21T15:18:09Z

decode_jpeg 和 deform_conv2d 的单测还没添加

done

MarioLulab

deform_conv2d 还缺少 test/legacy_test/test_deformable_conv_op.py 和 test/legacy_test/test_deformable_conv_v1_op.py 文件里相关单测的适配
distribute_fpn_proposals 还缺少 test/legacy_test/test_detection.py 文件里相关单测的适配

test/legacy_test/test_deform_conv2d.py

@@ -108,6 +109,7 @@ def out_size(

        self.mask = np.random.uniform(-1, 1, self.mask_shape).astype(self.dtype)

+    @test_with_pir_api


test/legacy_test/test_deform_conv2d.py

@@ -320,6 +322,7 @@ def out_size(

        self.mask = np.random.uniform(-1, 1, self.mask_shape).astype(self.dtype)

+    @test_with_pir_api


test/legacy_test/test_deform_conv2d.py

test/legacy_test/test_deformable_conv_op.py

@@ -390,6 +392,7 @@ def init_test_case(self):


 class TestModulatedDeformableConvInvalidInput(unittest.TestCase):
+    @test_with_pir_api
    def test_error(self):


test/legacy_test/test_deformable_conv_op.py

@@ -459,6 +462,7 @@ def test_invalid_groups():


 class TestDeformConv2DAPI(unittest.TestCase):
+    @test_with_pir_api


test/legacy_test/test_detection.py


        for res_stat, res_dy in zip(output_stat_np, output_dy_np):
            np.testing.assert_array_equal(res_stat, res_dy)

+    @test_with_pir_api


test/legacy_test/test_detection.py

        np.testing.assert_array_equal(np.array(rois_stat), rois_dy)
        np.testing.assert_array_equal(np.array(roi_probs_stat), roi_probs_dy)
        np.testing.assert_array_equal(np.array(rois_num_stat), rois_num_dy)


 class TestMulticlassNMS2(unittest.TestCase):
+    @test_with_pir_api


MarioLulab · 2023-11-30T02:44:41Z

test/legacy_test/test_detection.py

@@ -62,6 +63,7 @@ def static_graph(self):
            base.default_main_program().random_seed = self.seed
            yield

+    @test_with_pir_api


这里适配的原因是？

MarioLulab · 2023-11-30T02:45:06Z

test/legacy_test/test_detection.py

-
+    @test_with_pir_api
+    def static_generate_proposals(
+        self, scores_np, bbox_deltas_np, im_info_np, anchors_np, variances_np


这里是适配的原因是？该单测并没有测试 distribute_fpn_proposals

MarioLulab · 2023-11-30T02:45:13Z

test/legacy_test/test_detection.py


+    def dynamic_generate_proposals(


MarioLulab · 2023-11-30T02:45:23Z

test/legacy_test/test_detection.py

@@ -155,13 +156,34 @@ def test_generate_proposals(self):
            rois_dy = rois.numpy()
            roi_probs_dy = roi_probs.numpy()
            rois_num_dy = rois_num.numpy()
+        return rois_dy, roi_probs_dy, rois_num_dy
+
+    def test_generate_proposals(self):


MarioLulab

nice work !
还有一些地方需要修改

test/legacy_test/test_detection.py

test/legacy_test/test_deform_conv2d.py

+            y_v1 = paddle.vision.ops.deform_conv2d(
                input=x,
                offset=offset,
                mask=None,


test/legacy_test/test_deform_conv2d.py

@@ -159,7 +161,7 @@ def static_graph_case_dcn(self):
                modulated=False,
            )

-            y_v2 = paddle.static.nn.common.deformable_conv(
+            y_v2 = paddle.vision.ops.deform_conv2d(


test/legacy_test/test_deform_conv2d.py

+            y_v1 = paddle.vision.ops.deform_conv2d(
                input=x,
                offset=offset,
                mask=None,


test/legacy_test/test_deform_conv2d.py

@@ -371,7 +374,7 @@ def static_graph_case_dcn(self):
                modulated=False,
            )

-            y_v2 = paddle.static.nn.common.deformable_conv(
+            y_v2 = paddle.vision.ops.deform_conv2d(
                input=x,
                offset=offset,


test/legacy_test/test_deformable_conv_op.py

@@ -155,7 +156,7 @@ def deform_conv2d_wrapper(
 class TestModulatedDeformableConvOp(OpTest):
    def setUp(self):
        self.python_api = deform_conv2d_wrapper
-        self.op_type = "deformable_conv"
+        self.op_type = "deform_conv2d"


test/legacy_test/test_deformable_conv_op.py

                input, offset, mask, 1, 1, padding=1, groups=0
            )

        self.assertRaises(ValueError, test_invalid_groups)


 class TestDeformConv2DAPI(unittest.TestCase):
+    @test_with_pir_api
    def test_api(self):


test/legacy_test/test_deform_conv2d.py

@@ -728,6 +732,7 @@ def setUp(self):

 class TestDeformConv2DError(unittest.TestCase):
    def test_input_error(self):
+        @test_with_pir_api
        def test_input_rank_error():


longranger2 · 2023-12-07T09:11:37Z

done

MarioLulab

paddle.vision.ops.deform_conv2d 需要输入 weight 参数。麻烦再仔细检查一下入参是否正确吧~ deform_conv2d 的迁移比较麻烦，最好用本地环境先测试一下

MarioLulab · 2023-12-08T07:43:55Z

test/legacy_test/test_detection.py

-    def test_distribute_fpn_proposals(self):
-        rois_np = np.random.rand(10, 4).astype('float32')
-        rois_num_np = np.array([4, 6]).astype('int32')
+    @test_with_pir_api


@test_with_pir_api 应该用来修饰下面的 test_distribute_fpn_proposals，test_distribute_fpn_proposals 才是用来进行结果比较的

longranger2 · 2023-12-08T07:59:17Z

paddle.vision.ops.deform_conv2d 需要输入 weight 参数。麻烦再仔细检查一下入参是否正确吧~ deform_conv2d 的迁移比较麻烦，最好用本地环境先测试一下

哈哈好的，很久没编译了，周末再来调试下~

MarioLulab · 2023-12-19T06:03:27Z

paddle.vision.ops.deform_conv2d 需要输入 weight 参数。麻烦再仔细检查一下入参是否正确吧~ deform_conv2d 的迁移比较麻烦，最好用本地环境先测试一下

哈哈好的，很久没编译了，周末再来调试下~

long 师傅最近编译了吗 ~ 如果没有的话 deform_conv2d 的适配就由我来做吧

longranger2 · 2023-12-19T06:13:06Z

paddle.vision.ops.deform_conv2d 需要输入 weight 参数。麻烦再仔细检查一下入参是否正确吧~ deform_conv2d 的迁移比较麻烦，最好用本地环境先测试一下

哈哈好的，很久没编译了，周末再来调试下~

long 师傅最近编译了吗 ~ 如果没有的话 deform_conv2d 的适配就由我来做吧

好的

MarioLulab · 2023-12-20T04:43:59Z

deform_conv2d 的 pir 适配将会在此 pr 里推进 #60168
long 师傅有空的时候可以关注下~ 然后顺便修改一下当前 pr 的标题和复原 deform_conv2d 的更改吧

MarioLulab · 2023-12-22T07:15:13Z

辛苦解决一下冲突~

… pir4

MarioLulab · 2023-12-25T03:23:37Z

test/legacy_test/test_read_file.py

辛苦复原一下 read_file 的单测~

longranger2 · 2023-12-25T07:15:08Z

done

MarioLulab

LGTM
decode_jpeg api 已在 #58955 里完成适配，麻烦更新一下当前 pr 的单测覆盖率和 pr 标题吧~

)

migrate decode_jpeg, deform_conv2d, distribute_fpn_proposals into pir

3157aa1

paddle-bot bot added the contributor External developers label Nov 4, 2023

longranger2 changed the title ~~【PIR API adaptor No.54-56】Migrate some ops into pir~~ 【PIR API adaptor No.51-52, 60】Migrate some ops into pir Nov 5, 2023

luotao1 mentioned this pull request Nov 6, 2023

新IR Python API适配升级 #58067

Closed

luotao1 added the HappyOpenSource 快乐开源活动issue与PR label Nov 6, 2023

luotao1 assigned luotao1, 0x45f, YuanRisheng and MarioLulab Nov 6, 2023

Merge branch 'PaddlePaddle:develop' into pir4

b5f3048

MarioLulab reviewed Nov 20, 2023

View reviewed changes

test/legacy_test/test_distribute_fpn_proposals_op.py

@@ -142,7 +143,7 @@ def setUp(self):

self.set_data()

def test_check_output(self):

self.check_output(check_dygraph=False)

self.check_output(check_dygraph=False, check_pir=True)

This comment was marked as resolved.

Sign in to view

This comment was marked as resolved.

Sign in to view

fix bug

17ce133

This comment was marked as resolved.

Sign in to view

fix bug

14d80d8

MarioLulab reviewed Nov 29, 2023

View reviewed changes

fix bug

bfecca8

MarioLulab reviewed Nov 30, 2023

View reviewed changes

fix bug

9504955

MarioLulab reviewed Dec 7, 2023

View reviewed changes

fix bug

bc5b3f0

MarioLulab reviewed Dec 8, 2023

View reviewed changes

longranger2 changed the title ~~【PIR API adaptor No.51-52, 60】Migrate some ops into pir~~ 【PIR API adaptor No.51, 60】Migrate some ops into pir Dec 22, 2023

longranger2 added 2 commits December 24, 2023 21:34

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

e226b89

… pir4

Update test_detection.py

ab55c11

MarioLulab reviewed Dec 25, 2023

View reviewed changes

test/legacy_test/test_read_file.py Outdated

Copy link

Contributor

MarioLulab Dec 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦复原一下 read_file 的单测~

Update test_read_file.py

fea6f20

MarioLulab approved these changes Dec 26, 2023

View reviewed changes

0x45f approved these changes Dec 26, 2023

View reviewed changes

0x45f merged commit a134bf9 into PaddlePaddle:develop Dec 26, 2023
29 checks passed

Wanglongzhi2001 pushed a commit to Wanglongzhi2001/Paddle that referenced this pull request Jan 7, 2024

【PIR API adaptor No.51, 60】Migrate some ops into pir (PaddlePaddle#58684

53368f2

)

		@@ -108,6 +109,7 @@ def out_size(

		self.mask = np.random.uniform(-1, 1, self.mask_shape).astype(self.dtype)

		@test_with_pir_api

		@@ -320,6 +322,7 @@ def out_size(

		self.mask = np.random.uniform(-1, 1, self.mask_shape).astype(self.dtype)

		@test_with_pir_api

		@@ -459,6 +462,7 @@ def test_invalid_groups():


		class TestDeformConv2DAPI(unittest.TestCase):
		@test_with_pir_api

【PIR API adaptor No.51, 60】Migrate some ops into pir #58684

【PIR API adaptor No.51, 60】Migrate some ops into pir #58684

Conversation

longranger2 commented Nov 4, 2023 • edited

PR types

PR changes

Description

paddle-bot bot commented Nov 4, 2023

paddle-ci-bot bot commented Nov 12, 2023

MarioLulab commented Nov 14, 2023

This comment was marked as resolved.

This comment was marked as resolved.

longranger2 commented Nov 21, 2023

This comment was marked as resolved.

MarioLulab left a comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

This comment was marked as resolved.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as resolved.

This comment was marked as off-topic.

MarioLulab Nov 30, 2023

Choose a reason for hiding this comment

MarioLulab Nov 30, 2023

Choose a reason for hiding this comment

MarioLulab Nov 30, 2023

Choose a reason for hiding this comment

MarioLulab Nov 30, 2023

Choose a reason for hiding this comment

MarioLulab left a comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

longranger2 commented Dec 7, 2023

MarioLulab left a comment

Choose a reason for hiding this comment

MarioLulab Dec 8, 2023

Choose a reason for hiding this comment

longranger2 commented Dec 8, 2023

MarioLulab commented Dec 19, 2023

longranger2 commented Dec 19, 2023

MarioLulab commented Dec 20, 2023

MarioLulab commented Dec 22, 2023

MarioLulab Dec 25, 2023

Choose a reason for hiding this comment

longranger2 commented Dec 25, 2023

MarioLulab left a comment

Choose a reason for hiding this comment

longranger2 commented Nov 4, 2023 •

edited