[Frontend][PaddlePaddle] Add 10+ operators for PaddlePaddle #9126

jiangjiajun · 2021-09-26T11:40:47Z

This pull request is part of #9102 hope this will bring some help to review

@AndrewZhaoLuo
Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

merge to newest code

upate

python/tvm/relay/frontend/paddlepaddle.py

mbrookhart

This is really big, so it's hard to catch all of the edge cases in a review, but it looks okay, with the standard caveats that dynamic support might not be fully there yet. Not seeing anything I disapprove of, but I'd like to get more eyes on it before approving.

mbrookhart · 2021-09-27T17:11:10Z

python/tvm/relay/frontend/paddlepaddle.py

+            except Exception as e:
+                msg = "Dynamic shape is not supported in SAME padding algorithm while stride!=1"
+                raise tvm.error.OpAttributeInvalid(msg) from e


Just as a heads up, I supported SAME padding in the ONNX frontend with dynamic shapes here:

tvm/python/tvm/relay/frontend/onnx.py

Lines 412 to 472 in d0c6ca5

def autopad(

data,

strides,

kernel_shape,

dilations,

ndim,

pad_type="constant",

deconv=False,

mode="SAME_UPPER",

pad_value=0.0,

):

"""

Perform autopadding with dynamic input shapes

"""

# get attributes as constants

strides = _op.const(np.array(strides), dtype="int64")

dilated_kernel_shape = _op.const(

np.array(

[(kernel - 1) * dilation + 1 for kernel, dilation in zip(kernel_shape, dilations)]

),

dtype="int64",

)

# get input shape

shape = _op.strided_slice(shape_of(data, dtype="int64"), [2], [ndim])

# set up integer constants

zero = _op.const(0, dtype="int64")

one = _op.const(1, dtype="int64")

two = _op.const(2, dtype="int64")

# Calculate total padding

mod = _op.mod(shape, strides)

left = _op.maximum(dilated_kernel_shape - strides, zero)

right = _op.maximum(dilated_kernel_shape - mod, zero)

total_pad = _op.where(_op.equal(mod, zero), left, right)

if deconv:

total_pad = _op.const(np.array(kernel_shape), dtype="int64") - one - total_pad

# split total padding into before and after

pad_before = _op.floor_divide(total_pad, two)

pad_after = total_pad - pad_before

# combine

if "LOWER" in mode:

pad = _op.concatenate(

[_op.reshape(pad_after, [-1, 1]), _op.reshape(pad_before, [-1, 1])], axis=1

)

else:

pad = _op.concatenate(

[_op.reshape(pad_before, [-1, 1]), _op.reshape(pad_after, [-1, 1])], axis=1

)

# pad N and C with zeros

pad = _op.concatenate([_op.const(np.zeros([2, 2], dtype="int64"), dtype="int64"), pad], axis=0)

if isinstance(pad_value, (float, int)):

pad_value = _op.const(pad_value)

return _op.nn.pad(data, fold_constant(pad), pad_value, pad_type)

It's fairly complicated, I'm totally cool if you want to punt on that until you need it.

Thanks, this will solve a big problem! But to avoid making this pull request more complicated to review, let's left this for the next pull request.

AndrewZhaoLuo

Some initial comments. I've gotten up to convert_fill_constant will review the rest later. Agree with mbrookhart. This is pretty big so we might need more eyes.

Can you send the most up to date english docs btw?

Also, does PaddlePaddle support operator versioning? How will you handle API changes in the future?

AndrewZhaoLuo · 2021-09-27T17:42:53Z

python/tvm/relay/frontend/paddlepaddle.py

+            inputs[i] = input_op.astype(max_dtype)
+    return inputs
+
+
 def shape_of(x, dtype="int32"):


Can you use python/tvm/relay/frontend/common.py::infer_shape?

We have referred ONNX frontend, this function also comes from there https://github.com/apache/tvm/blob/main/python/tvm/relay/frontend/onnx.py#L1411
It's a little different from common::infer_shape

AndrewZhaoLuo · 2021-09-27T17:43:14Z

python/tvm/relay/frontend/paddlepaddle.py

    return _op.shape_of(x, dtype)


-def _get_pad_size(in_size, dilated_kernel_size, stride_size):
-    """calculate the paddings size"""
+def _infer_value(x, params):


Can you use python/tvm/relay/frontend/common.py::infer_value?

Done. I just found there's try_infer_value in common.py, this function is removed.

AndrewZhaoLuo · 2021-09-27T17:45:38Z

python/tvm/relay/frontend/paddlepaddle.py

@@ -40,28 +41,108 @@
 __all__ = ["from_paddle"]


+def _get_pad_size(in_size, dilated_kernel_size, stride_size):
+    """calculate the paddings size"""


In general Docstrings should be complete sentences. End with a period and capitalize the first letter.

E.g. "Calculate the paddings size."

Please fix the other docstrings

python/tvm/relay/frontend/paddlepaddle.py

AndrewZhaoLuo · 2021-09-27T17:56:16Z

python/tvm/relay/frontend/paddlepaddle.py

+    ipt1 = g.get_node(op.input("Y")[0])
+    op_func = get_relay_op(op.type)
+    out = op_func(ipt0, ipt1)
+    g.add_node(op.output("Out")[0], out)


 def convert_arg_max(g, op, block):


Do you have a link to the english docs?

https://www.paddlepaddle.org.cn/documentation/docs/en/1.8/api/layers/argmax.html

Doesn't seem to have some of the attributes listed in the op

Now the latest version is 2.1, API documents: https://www.paddlepaddle.org.cn/documentation/docs/en/api/paddle/argmax_en.html#argmax

Follow the API definition code, we can find there's some attributes not list in the API's parameters
https://github.com/PaddlePaddle/Paddle/blob/release/2.1/python/paddle/tensor/search.py#L179

attrs['keepdims'] = keepdim attrs['axis'] = axis attrs['flatten'] = flatten attrs['dtype'] = var_dtype helper.append_op( type='arg_max', inputs={'X': x}, outputs={'Out': [out]}, attrs=attrs) out.stop_gradient = True

AndrewZhaoLuo · 2021-09-27T18:08:53Z

python/tvm/relay/frontend/paddlepaddle.py

+    ipt1 = g.get_node(op.input("Y")[0])
+    op_func = get_relay_op(op.type)
+    out = op_func(ipt0, ipt1)
+    g.add_node(op.output("Out")[0], out)


 def convert_arg_max(g, op, block):


A lot of the logic in argmin and argmax is similar. Refactor to combine the two.

AndrewZhaoLuo · 2021-09-27T18:11:51Z

python/tvm/relay/frontend/paddlepaddle.py

+    axis = op.attr("axis")
+    descending = op.attr("descending")
+
+    out = _op.sort(x, axis, not descending)


consider using _op.gather on the out_indices

AndrewZhaoLuo · 2021-09-27T18:11:58Z

python/tvm/relay/frontend/paddlepaddle.py

+    descending = op.attr("descending")
+
+    out = _op.sort(x, axis, not descending)
+    out_indice = _op.argsort(x, axis, not descending, dtype="int64")


nit: out_indices

AndrewZhaoLuo · 2021-09-27T18:22:53Z

python/tvm/relay/frontend/paddlepaddle.py

+    x = g.get_node(op.input("X")[0])
+    y = g.get_node(op.input("Y")[0])
+
+    out = _op.sum(_op.multiply(x, y), axis=[-1], keepdims=True)


You might want to note the semantics of paddle paddle's dot operator. Namely how it also operates on 2d-vectors (and hence why axis=[-1]).

I have not seen this elsewhere

paddle.dot : https://www.paddlepaddle.org.cn/documentation/docs/en/api/paddle/dot_en.html

It's similar with torch.dot, while torch.dot only supports 1D tensor.
In PaddlePaddle, inputs should be both 1D or 2D tensor. When it is 2d, the first dimension of this matrix is the batch dimension.

For clarify, I also put this explanation in code

jiangjiajun · 2021-09-28T02:41:00Z

Hi, @AndrewZhaoLuo @mbrookhart
Thanks for reviewing this PR.

PaddlePaddle now provides English document for API , there's no document describe operators, we are supporting this operators mostly by refer to its cpp code or test code, like crop_op.h or test_crop_tensor.py, but I think it's necessary to provide such documents, I'll try to push this within the PaddlePaddle team in the next quarter. For now, if you have any question about the operators, just comment in this pr, I'll try to make explanation here.
Like other framework, PaddlePaddle has needs to upgrade operators, this will add or delete some parameters for the operator, and also bring a new operator name, like squeeze2 or multiclass_nms3. Currently, we create a convert function for all the different versions of operator, but different versions of operator are both list in the _convert_map.

remove unreviewed code

jiangjiajun · 2021-09-29T12:14:46Z

This PR is still too big I think considering most work here is unrelated to each other. Can you remove operators from this PR until you are down to ~+300 loc?

Just so you know, all ops above convert_fill_constant I have taken a look at so if you reduce this PR down to those changes only the review process can go a lot faster

Hi, @AndrewZhaoLuo
All the modifications under convert_fill_constant are removed .
Still lack of lots of pull requests to finish my work, I'll try to classify these pull requests to make reviewing faster

AndrewZhaoLuo

Coming together well. I think I am ok with merging this in the current state as part of a more experimental frontend, but would like another pair of eyes on this.

Still a few comments which will make this better:

Can you add more substantial test cases to your tests? E.g. different input shapes and those of different ranks at least for some of the relevant ops. I feel that some issues might come to light from this. Do not worry about PR size anymore
Please add type annotations to functions for this and future PRs. e.g.

def myFunc(a: int, b: List[string]) -> None:
...

AndrewZhaoLuo · 2021-09-30T06:46:58Z

python/tvm/relay/frontend/paddlepaddle.py

-    out = act_func(g.get_node(op.input("X")[0]))
+    x = g.get_node(op.input("X")[0])
+    target_shape = op.attr("target_shape")
+    out = _op.broadcast_to(x, target_shape)


I believe you might run into a similar issue as https://github.com/apache/tvm/blob/main/python/tvm/relay/frontend/onnx.py#L2257

PaddlePaddle's expand_as doesn't support multi-directional broadcasting, so this problem will not happen in PaddlePaddle frontend

AndrewZhaoLuo · 2021-09-30T06:51:22Z

python/tvm/relay/frontend/paddlepaddle.py

    new_var,
 )

 __all__ = ["from_paddle"]


+def _get_pad_size(in_size, dilated_kernel_size, stride_size):
+    """Calculate the paddings size."""


Should describe padding size for what

add more cases for tests

jiangjiajun · 2021-10-04T08:44:37Z

Coming together well. I think I am ok with merging this in the current state as part of a more experimental frontend, but would like another pair of eyes on this.

Still a few comments which will make this better:

Can you add more substantial test cases to your tests? E.g. different input shapes and those of different ranks at least for some of the relevant ops. I feel that some issues might come to light from this. Do not worry about PR size anymore

Please add type annotations to functions for this and future PRs. e.g.

def myFunc(a: int, b: List[string]) -> None: ...

More cases have been added in tests, but only for the new operators in this pull request. I will send another pull request for the previous operators.
Type annotation is a good code habit, but for the function like convert_dot(g : GraphProto, op : paddle.fluid.framework.operators, block: paddle.fluid.framework.Block), the type annotation will bring dependency of paddlepaddle for TVM, I noticed that all the frontends putting framework importing in from_xxx function to avoid strong dependency for TVM.

AndrewZhaoLuo

Shame about the typing, you can do forward references like:

def g(f:"paddle.paddleblahblah.blah"): -->:

But eh it's not the end of the world to be untyped since the rest of the frontends are like that. Just one comment about test case sizes.

We'll need another approver though. @mbrookhart ?

AndrewZhaoLuo · 2021-10-04T22:22:26Z

tests/python/frontend/paddlepaddle/test_forward.py

+        "relu",
+        "tanh",
+    ]
+    input_shapes = [[128], [2, 256], [1000, 128, 32], [7, 3, 256, 256]]


Please reduce the size of your test cases to something smaller e.g. less than 256 total elements (totally arbitrary, just as small as possible while still accomplishing the test)

Done. I guess the limit on the number of elements is to reduce the cost time of testing?

AndrewZhaoLuo

LGTM

jiangjiajun · 2021-10-08T02:08:15Z

@junrushao1994 Hi, could you help to merge this pull request?

junrushao

Thanks @AndrewZhaoLuo for the review! Thanks @jiangjiajun for the PR!

…Paddle (apache#9126)" This reverts commit c980db3.

…pache#9126) * add part of operators * remove part of operators * add lookup * add test * Update paddlepaddle.py * modify error message for SAME padding * Remove some function and old version operator * Remove some function and old version operator * Remove some function and old version operator * Remove some function and old version operator * add dot test * modify doc * remove unreviewed code * Update paddlepaddle.py * Update test_forward.py * Update paddlepaddle.py * Update paddlepaddle.py * Update test_forward.py * Update test_forward.py * add more cases for tests * add more cases for tests * remove annotation * reduce test case sizes

jiangjiajun and others added 6 commits August 3, 2021 17:14

Merge pull request #2 from apache/main

5b39c79

merge to newest code

Merge pull request #8 from apache/main

74cc942

upate

Merge branch 'apache:main' into main

e0420bd

Merge branch 'apache:main' into main

f181b0a

Merge branch 'apache:main' into main

800b187

add part of operators

87c6d3d

jiangjiajun requested review from areusch, comaniac, Huyuwei, jroesch, junrushao, jwfromm, kazum, mbrookhart, merrymercy, siju-samuel, srkreddy1238, tqchen and yzhliu as code owners September 26, 2021 11:40

jiangjiajun added 2 commits September 26, 2021 11:54

remove part of operators

39b96fc

add lookup

50e3c41

jiangjiajun commented Sep 26, 2021

View reviewed changes

python/tvm/relay/frontend/paddlepaddle.py Outdated Show resolved Hide resolved

jiangjiajun and others added 3 commits September 26, 2021 13:02

add test

555406e

Update paddlepaddle.py

75956db

modify error message for SAME padding

a3aa170

mbrookhart reviewed Sep 27, 2021

View reviewed changes

AndrewZhaoLuo reviewed Sep 27, 2021

View reviewed changes

jiangjiajun added 2 commits September 28, 2021 08:05

Remove some function and old version operator

326383a

Remove some function and old version operator

6e275c2

jiangjiajun added 5 commits September 29, 2021 19:46

Update paddlepaddle.py

ef7a003

Update paddlepaddle.py

43ae5ab

Update test_forward.py

8d0af49

Update test_forward.py

509e023

Merge pull request #57 from jiangjiajun/unreviewed

4139fb3

remove unreviewed code

AndrewZhaoLuo reviewed Sep 30, 2021

View reviewed changes

jiangjiajun and others added 4 commits October 4, 2021 15:17

Merge branch 'apache:main' into pr001

2a5e30d

add more cases for tests

3c34b5a

add more cases for tests

e600036

Merge pull request #60 from jiangjiajun/add-more-cases

4887e96

add more cases for tests

remove annotation

ef4c84b

AndrewZhaoLuo reviewed Oct 4, 2021

View reviewed changes

reduce test case sizes

5d1aa7c

AndrewZhaoLuo approved these changes Oct 5, 2021

View reviewed changes

junrushao approved these changes Oct 8, 2021

View reviewed changes

junrushao merged commit c980db3 into apache:main Oct 8, 2021

masahi mentioned this pull request Oct 8, 2021

[Bug] PaddlePaddle integration tests busted at main #9231

Closed

masahi pushed a commit to masahi/tvm that referenced this pull request Oct 8, 2021

Revert "[Frontend][PaddlePaddle][Part1] Add 100+ operators for Paddle…

9956e7b

…Paddle (apache#9126)" This reverts commit c980db3.

masahi mentioned this pull request Oct 8, 2021

Revert "[Frontend][PaddlePaddle][Part1] Add 100+ operators for Paddle… #9232

Closed

masahi pushed a commit to masahi/tvm that referenced this pull request Oct 9, 2021

Revert "[Frontend][PaddlePaddle][Part1] Add 100+ operators for Paddle…

d45f474

…Paddle (apache#9126)" This reverts commit c980db3.

jiangjiajun mentioned this pull request Oct 28, 2021

[RFC][Tracking][RFC-0019] Add PaddlePaddle Frontend #8751

Open

6 tasks

jiangjiajun changed the title ~~[Frontend][PaddlePaddle][Part1] Add 100+ operators for PaddlePaddle~~ [Frontend][PaddlePaddle][Part1] Add 10+ operators for PaddlePaddle Oct 28, 2021

jiangjiajun changed the title ~~[Frontend][PaddlePaddle][Part1] Add 10+ operators for PaddlePaddle~~ [Frontend][PaddlePaddle] Add 10+ operators for PaddlePaddle Oct 28, 2021

jiangjiajun mentioned this pull request Nov 2, 2021

Apache TVM v0.8 Release Note Candidate #9416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Frontend][PaddlePaddle] Add 10+ operators for PaddlePaddle #9126

[Frontend][PaddlePaddle] Add 10+ operators for PaddlePaddle #9126

jiangjiajun commented Sep 26, 2021 •

edited

mbrookhart left a comment

mbrookhart Sep 27, 2021 •

edited

jiangjiajun Sep 28, 2021

AndrewZhaoLuo left a comment

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

AndrewZhaoLuo Sep 27, 2021

jiangjiajun Sep 28, 2021

jiangjiajun commented Sep 28, 2021 •

edited

jiangjiajun commented Sep 29, 2021 •

edited

AndrewZhaoLuo left a comment

AndrewZhaoLuo Sep 30, 2021

jiangjiajun Oct 4, 2021

AndrewZhaoLuo Sep 30, 2021

jiangjiajun Oct 4, 2021

jiangjiajun commented Oct 4, 2021

AndrewZhaoLuo left a comment

AndrewZhaoLuo Oct 4, 2021

jiangjiajun Oct 5, 2021

AndrewZhaoLuo left a comment

jiangjiajun commented Oct 8, 2021

junrushao left a comment

	def autopad(
	data,
	strides,
	kernel_shape,
	dilations,
	ndim,
	pad_type="constant",
	deconv=False,
	mode="SAME_UPPER",
	pad_value=0.0,
	):
	"""
	Perform autopadding with dynamic input shapes
	"""
	# get attributes as constants
	strides = _op.const(np.array(strides), dtype="int64")
	dilated_kernel_shape = _op.const(
	np.array(
	[(kernel - 1) * dilation + 1 for kernel, dilation in zip(kernel_shape, dilations)]
	),
	dtype="int64",
	)
	# get input shape
	shape = _op.strided_slice(shape_of(data, dtype="int64"), [2], [ndim])

	# set up integer constants
	zero = _op.const(0, dtype="int64")
	one = _op.const(1, dtype="int64")
	two = _op.const(2, dtype="int64")

	# Calculate total padding
	mod = _op.mod(shape, strides)

	left = _op.maximum(dilated_kernel_shape - strides, zero)
	right = _op.maximum(dilated_kernel_shape - mod, zero)

	total_pad = _op.where(_op.equal(mod, zero), left, right)
	if deconv:
	total_pad = _op.const(np.array(kernel_shape), dtype="int64") - one - total_pad

	# split total padding into before and after
	pad_before = _op.floor_divide(total_pad, two)
	pad_after = total_pad - pad_before

	# combine
	if "LOWER" in mode:
	pad = _op.concatenate(
	[_op.reshape(pad_after, [-1, 1]), _op.reshape(pad_before, [-1, 1])], axis=1
	)
	else:
	pad = _op.concatenate(
	[_op.reshape(pad_before, [-1, 1]), _op.reshape(pad_after, [-1, 1])], axis=1
	)

	# pad N and C with zeros
	pad = _op.concatenate([_op.const(np.zeros([2, 2], dtype="int64"), dtype="int64"), pad], axis=0)

	if isinstance(pad_value, (float, int)):
	pad_value = _op.const(pad_value)

	return _op.nn.pad(data, fold_constant(pad), pad_value, pad_type)

[Frontend][PaddlePaddle] Add 10+ operators for PaddlePaddle #9126

[Frontend][PaddlePaddle] Add 10+ operators for PaddlePaddle #9126

Conversation

jiangjiajun commented Sep 26, 2021 • edited

mbrookhart left a comment

Choose a reason for hiding this comment

mbrookhart Sep 27, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewZhaoLuo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiangjiajun commented Sep 28, 2021 • edited

jiangjiajun commented Sep 29, 2021 • edited

AndrewZhaoLuo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiangjiajun commented Oct 4, 2021

AndrewZhaoLuo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewZhaoLuo left a comment

Choose a reason for hiding this comment

jiangjiajun commented Oct 8, 2021

junrushao left a comment

Choose a reason for hiding this comment

jiangjiajun commented Sep 26, 2021 •

edited

mbrookhart Sep 27, 2021 •

edited

jiangjiajun commented Sep 28, 2021 •

edited

jiangjiajun commented Sep 29, 2021 •

edited