[FRONTEND][ONNX] 'Shape' operator not implemented

[rajh619]

I have an Alexnet-ONNX model converted from Tensorflow which contains 'Shape' operator.
Unfortunately while loading in nnvm , it fails with NotImplementedError: Operator Shape not implemented.
and i couldn't find the same in nnvm.symbol (nnvm tensor operator api).
Basically 'Shape' operator (as specified in onnx operator List ) is to infer the shape of input tensor as 1D int64 output tensor .
Is there any alternative way to output the shape in 1D int64 tensor in onnx frontend ?

[srkreddy1238]

I have an active PR in tensor flow for Shape.

You could take ref and implement for ONNX.

[rajh619]

could you please share the link of your PR .

[srkreddy1238]

#1232

[rajh619]

@srkreddy1238

I have converted only a fragment of alexnet tf model into onnx, containing the 'Shape' operator for easy debugging.
Below is my ONNX graph :

graph tf2onnx (
  %Placeholder:0[FLOAT, 1x227x227x3]
) initializers (
  %conv1/weights:0[FLOAT, 96x3x11x11]
  %conv1/biases:0[FLOAT, 96]
) {
  %Conv2D__2:0 = Transpose[perm = [0, 3, 1, 2]](%Placeholder:0)
  %Conv2D:0 = Conv[dilations = [1, 1], kernel_shape = [11, 11], strides = [4, 4]](%Conv2D__2:0, %conv1/weights:0)
  %Conv2D__3:0 = Transpose[perm = [0, 2, 3, 1]](%Conv2D:0)
  %BiasAdd:0 = Add(%Conv2D__3:0, %conv1/biases:0)
  %Shape:0 = Shape(%Conv2D__3:0)
  %Reshape__4:0 = Cast[to = 7](%Shape:0)
  %Reshape:0 = Reshape(%BiasAdd:0, %Reshape__4:0)
  %conv1_1:0 = Relu(%Reshape:0)
  return %conv1_1:0
}

def _shape():
    def _impl(inputs, attr, params):
        input_shapes = attr['_input_shapes'][inputs[0]]

        # Fix the -1 dimensions to 1
        input_shapes[0] = [1 if x == -1 else x for x in input_shapes[0]]
        params[attr['_node_name']] = tvm.nd.array(input_shapes[0])

        return _sym.Variable(name=attr['_node_name'],
                             shape=params[attr['_node_name']].shape)
    return _impl

In my graph Attributes for 'Shape' is empty{} . I see no attr['_input_shapes'] for op_type 'Shape'.
So i get following error :

, in _convert_operator
    sym = convert_map[op_name](inputs, attrs, self._params)
  File "C:\Users\HRG\AppData\Roaming\Python\Python36\site-packages\nnvm-0.8.0-py3.6.egg\nnvm\frontend\onnx.py", line 419, in _impl
    input_shapes = attr['_input_shapes'][inputs[0]]
KeyError: '_input_shapes'

So what does " %Shape:0 = Shape(%Conv2D__3:0)" mean ? how to feed Conv2D__3:0 as input for 'Shape'.

[srkreddy1238]

You can't copy the function as it is, it's only a reference.

Can you share the ONNX graph ?

[rajh619]

Hi please find the attached onnx file
alexnet_fragment.zip

[srkreddy1238]

@rajh619

I checked the graph and it's a bit different from tensorflow.
Tensorflow frontend has all the out_shapes for all nodes available hence the Shape operator is computed at frontend and made it as a param to graph.

We may need Shape as an operator from compiler / topi in this case.

@tqchen
Earlier assumption of output_shapes availability doesn't hold true for all frontends.
Do we implement new operator Shape here ?

[srkreddy1238]

@rajh619

I further checked and here is my observation.

Tensorflow uses Shape operator to derive shape as a Tensor.
This Tensor is passed to next operators like Reshape as a Tensor argument for shape.

TVM Reshape operator doesn't support Tensor argument for shape (it's a tuple instead).
This shape operator is specific to tensorflow and onnx (not sure if onnx inherited from tensorflow for compatibility).

At this point I advice to use tensorflow frontend directly which handle these issues with _output_shapes until we have "Shape" and extended "Reshape" operator support in TVM.

[rajh619]

@srkreddy1238 Thanks for the explanation .
I will try tensorflow frontend .But am mostly interested in ONNX front end ,as it is the open format for any deep learning model .Also i think it helps users to convert their DL models implemented in DL frameworks where there is no frontend support available in NNVM (like Caffe2/CNTK/Chainer models etc.) to ONNX using converters

So maybe i have to wait for the shape operator to be available in TVM to try out this ONNX model !

[srkreddy1238]

I will PR the shape and reshape_like soon. I could enhance the tensor flow too with these.

[zwhdang]

@srkreddy1238
Thanks for your explanation.
I just export the super_resolution.onnx by the latest version of pytorch(ebae3f) according to "http://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html". When I try the "from_onnx.py" in tutorials of nnvm, it fails with NotImplementedError: Operator Shape not implemented too. So the 'Shape' operator may by a common demand for several ONNX using frameworks.

[srkreddy1238]

@zwhdang
I further analyzed and here are some challenges around shape.
Implementing Shape is not a challenge but, the next operation after Shape is tricky.

For example Reshape receiving the "to shape" as an input Tensor cannot pass through compilation process in current framework.

I have seen Shape operator in Tensorflow and may be inherited into ONNX to keep the compatibility. But with TVM it's a challenge.

I was able to workaround in tensorflow frontend as I have the output_shapes and input_shapes in the graph definition.

[zwhdang]

@srkreddy1238
I just try the pretrained model of pytorch, such as AlexNet and ResNet, and the "Shape problem" appears multiply in tvm-onnx-frontend. So do you have any good suggestion for loading pytorch-onnx models? Thanks.

[tqchen]

Seems most of the shape requirement was to achieve reshape_like, which we cold have a specialization path

[srkreddy1238]

I am thinking on below two approaches.

• We could have a an extra function in nnvm build_module some where after infer_shape and around optimize to convert Shape followed by Resize to drop Shape and update attributes for Reshape.

Or

• A new pass backed by a new node API like
  FConversion: which indicates converting this node to param node.
  FInferAttributes: which is called on dependent nodes to alter the attributes and drop this input.
  ** This is just an initial though to standardize this requirement, I am yet to check the feasibility for this. request to comment on these options.

[srkreddy1238]

Ref. #1333 much simpler and better approach.

[zwhdang]

@srkreddy1238
I try this commit and the Shape problem disappears. However, another error occurs:
NotImplementedError: Operator Slice not implemented.
Below is my ONNX graph :

$ python super_resolution_with_caffe2.py 
graph(%0 : Float(1, 1, 224, 224)
      %1 : Float(64, 1, 5, 5)
      %2 : Float(64)
      %3 : Float(64, 64, 3, 3)
      %4 : Float(64)
      %5 : Float(32, 64, 3, 3)
      %6 : Float(32)
      %7 : Float(9, 32, 3, 3)
      %8 : Float(9)) {
  %9 : Float(1, 64, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[5, 5], pads=[2, 2, 2, 2], strides=[1, 1]](%0, %1, %2), scope: SuperResolutionNet/Conv2d[conv1]
  %10 : Float(1, 64, 224, 224) = onnx::Relu(%9), scope: SuperResolutionNet/ReLU[relu]
  %11 : Float(1, 64, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1]](%10, %3, %4), scope: SuperResolutionNet/Conv2d[conv2]
  %12 : Float(1, 64, 224, 224) = onnx::Relu(%11), scope: SuperResolutionNet/ReLU[relu]
  %13 : Float(1, 32, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1]](%12, %5, %6), scope: SuperResolutionNet/Conv2d[conv3]
  %14 : Float(1, 32, 224, 224) = onnx::Relu(%13), scope: SuperResolutionNet/ReLU[relu]
  %15 : Float(1, 9, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1]](%14, %7, %8), scope: SuperResolutionNet/Conv2d[conv4]
  %16 : Dynamic = onnx::Shape(%15), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %17 : Dynamic = onnx::Slice[axes=[0], ends=[1], starts=[0]](%16), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %18 : Long() = onnx::Squeeze[axes=[0]](%17), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %19 : Dynamic = onnx::Shape(%15), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %20 : Dynamic = onnx::Slice[axes=[0], ends=[2], starts=[1]](%19), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %21 : Long() = onnx::Squeeze[axes=[0]](%20), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %22 : Dynamic = onnx::Shape(%15), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %23 : Dynamic = onnx::Slice[axes=[0], ends=[3], starts=[2]](%22), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %24 : Long() = onnx::Squeeze[axes=[0]](%23), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %25 : Dynamic = onnx::Shape(%15), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %26 : Dynamic = onnx::Slice[axes=[0], ends=[4], starts=[3]](%25), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %27 : Long() = onnx::Squeeze[axes=[0]](%26), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %28 : Dynamic = onnx::Constant[value={9}](), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %29 : Long() = onnx::Div[broadcast=1](%21, %28), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %30 : Dynamic = onnx::Constant[value={3}](), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %31 : Long() = onnx::Mul[broadcast=1](%24, %30), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %32 : Dynamic = onnx::Constant[value={3}](), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %33 : Long() = onnx::Mul[broadcast=1](%27, %32), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %34 : Long() = onnx::Constant[value={3}](), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %35 : Long() = onnx::Constant[value={3}](), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %36 : Dynamic = onnx::Unsqueeze[axes=[0]](%18), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %37 : Dynamic = onnx::Unsqueeze[axes=[0]](%29), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %38 : Dynamic = onnx::Unsqueeze[axes=[0]](%34), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %39 : Dynamic = onnx::Unsqueeze[axes=[0]](%35), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %40 : Dynamic = onnx::Unsqueeze[axes=[0]](%24), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %41 : Dynamic = onnx::Unsqueeze[axes=[0]](%27), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %42 : Dynamic = onnx::Concat[axis=0](%36, %37, %38, %39, %40, %41), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %43 : Float(1, 1, 3, 3, 224, 224) = onnx::Reshape(%15, %42), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %44 : Float(1, 1, 224, 3, 224, 3) = onnx::Transpose[perm=[0, 1, 4, 2, 5, 3]](%43), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %45 : Dynamic = onnx::Unsqueeze[axes=[0]](%18), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %46 : Dynamic = onnx::Unsqueeze[axes=[0]](%29), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %47 : Dynamic = onnx::Unsqueeze[axes=[0]](%31), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %48 : Dynamic = onnx::Unsqueeze[axes=[0]](%33), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %49 : Dynamic = onnx::Concat[axis=0](%45, %46, %47, %48), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  %50 : Float(1, 1, 672, 672) = onnx::Reshape(%44, %49), scope: SuperResolutionNet/PixelShuffle[pixel_shuffle]
  return (%50);
}

[srkreddy1238]

@zwhdang

This model need Slice , Squeeze and Unsqueeze (expand_dims) to get through.
recent days there are few operators added to topi but frontends are not updated.

I am working for tensorflow and onnx to start with. I will try to push these operators first to unblock you.
You may need to wait for a while :)

[srkreddy1238]

#1339 for Squeeze and Unsqueeze.