Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Yolact][Pytorch Frontend][Paritioning Issue] Yolact - Instance segmentation model compilation issue #85

Open
abdulazizm opened this issue Jan 20, 2022 · 6 comments
Open

[Yolact][Pytorch Frontend][Paritioning Issue] Yolact - Instance segmentation model compilation issue #85

abdulazizm opened this issue Jan 20, 2022 · 6 comments

Comments

@abdulazizm
Copy link

https://github.com/Ma-Dan/yolact/tree/onnx - Ma-Dan version of yolact is onnx exportable (hope its good to JIT trace), tried this with pytorch frontend but getting segmentation fault at partition_for_vitis_ai(mod, params, dpu=target) as below (debug logs enabled - can post full log in file if needed)

DEBUG:pyxir:-- -- Sweep transpose: bX: ['moved_moved_nn_conv2d_NHWC-NCHW-94241211375472'], X: nn.relu-94240959460784, tX: ['nn_conv2d-94241211375472']
DEBUG:pyxir:-- Visit: moved_moved_moved_moved_transpose-94241201609008
-- -- for opt: SweepTransposesFlowDirection
DEBUG:pyxir:-- Visit: nn_bias_add-94241211132064
-- -- for opt: SweepTransposesFlowDirection
DEBUG:pyxir:-- -- Sweep transpose: bX: ['moved_moved_moved_moved_transpose-94241201609008'], X: nn_bias_add-94241211132064, tX: ['nn.relu-94241201608720']
INFO:pyxir:Writing graph visualization to after_partitioning_sweep_transposes.png
Fatal Python error: Segmentation fault

Current thread 0x00007f8944951740 (most recent call first):
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tvm-0.8.dev1859+g627e92e7c-py3.6-linux-x86_64.egg/tvm/ir/transform.py", line 161 in __call__
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tvm-0.8.dev1859+g627e92e7c-py3.6-linux-x86_64.egg/tvm/relay/op/contrib/vitis_ai.py", line 191 in partition_for_vitis_ai
  File "compile_yoloact_vitisai.py", line 226 in <module>
Segmentation fault (core dumped)

To JIT trace yoloact model:

def do_trace(model, in_size=100):
    model_trace = torch.jit.trace(model, torch.rand(1, 3, in_size, in_size))
    model_trace.eval()
    return model_trace

from yolact import Yolact
from data import cfg, set_cfg, set_dataset
set_cfg('yolact_resnet50' + '_config')
in_size = 550
inp = torch.rand(1, 3, in_size, in_size)

model = Yolact()
model.load_weights("weights/yolact_resnet50_54_800000.pth")
model.eval()
with torch.no_grad():
     out = model(inp)
     script_module = do_trace(model, in_size=550)

Please refer: https://discuss.tvm.apache.org/t/frontend-pytorch-compile-yolact-for-xilinx-hardware-zcu104/11909/5?u=abdulazizm
@abdulazizm abdulazizm changed the title Yolact Instance segmentation model with pytorch frontend [Yolact][Pytorch Frontend][Paritioning Issue] Yolavt - Instance segmentation model compilation issue Jan 20, 2022
@abdulazizm abdulazizm changed the title [Yolact][Pytorch Frontend][Paritioning Issue] Yolavt - Instance segmentation model compilation issue [Yolact][Pytorch Frontend][Paritioning Issue] Yolact - Instance segmentation model compilation issue Jan 20, 2022
@abdulazizm abdulazizm mentioned this issue Jan 20, 2022
Open
@abdulazizm
Copy link
Author

Hi @jornt-xilinx ,

Please help us with your suggestion for improving the performance. Did you find any irregularities in this mod['main']?

FYI:

  1. All nn.conv2d has common 256 output channel but not falls into dpu HW constraint. Anyway to bring those onto dpu?
  2. Noticed 1 image.resize2d surrounded by layout transform. Not sure why, I have added it to desired layouts (tried to register with level 10 - but it seems already registered internally), no changed observed in mod['main'] even if I change image.resize2d to 'NCHW' or 'NHWC' layout
fn (%data: Tensor[(1, 3, 550, 550), float32]) -> (Tensor[(1, 19173, 4), float32], Tensor[(1, 19173, 81), float32], Tensor[(1, 19173, 32), float32], Tensor[(19173, 4), float32]) {
  %0 = layout_transform(%data, src_layout="NCHW", dst_layout="NHWC") /* ty=Tensor[(1, 550, 550, 3), float32] */;
  %1 = @tvmgen_default_vitis_ai_main_0(%0) /* ty=(Tensor[(1, 35, 35, 256), float32], Tensor[(1, 69, 69, 256), float32], Tensor[(1, 35, 35, 12), float32], Tensor[(1, 18, 18, 12), float32], Tensor[(1, 9, 9, 12), float32], Tensor[(1, 35, 35, 243), float32], Tensor[(1, 18, 18, 243), float32], Tensor[(1, 9, 9, 243), float32], Tensor[(1, 35, 35, 96), float32], Tensor[(1, 18, 18, 96), float32], Tensor[(1, 9, 9, 96), float32]) */;
  %2 = %1.0;
  %3 = layout_transform(%2, src_layout="NHWC", dst_layout="NCHW") /* ty=Tensor[(1, 256, 35, 35), float32] */;
  %4 = %1.1;
  %5 = image.resize2d(%3, size=[69, 69], roi=[0f, 0f, 0f, 0f], rounding_method="", cubic_alpha=-0.75f) /* C.graph: aten::upsample_bilinear2d, jit._trace.TopLevelTracedModule: __module.fpn */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %6 = layout_transform(%4, src_layout="NHWC", dst_layout="NCHW") /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %7 = add(%5, %6) /* C.graph: aten::add, jit._trace.TopLevelTracedModule: __module.fpn */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %8 = nn.conv2d(%7, meta[relay.Constant][0] /* ty=Tensor[(256, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=256, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.fpn/__module.fpn.pred_layers.1_PART_0 */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %9 = add(%8, meta[relay.Constant][1] /* ty=Tensor[(1, 256, 1, 1), float32] */) /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %10 = nn.relu(%9) /* C.graph: aten::relu, jit._trace.TopLevelTracedModule: __module.fpn */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %11 = nn.conv2d(%10, meta[relay.Constant][2] /* ty=Tensor[(256, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=256, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.upfeature/__module.prediction_layers.0.upfeature.0_PART_2 */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %12 = add(%11, meta[relay.Constant][3] /* ty=Tensor[(1, 256, 1, 1), float32] */) /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %13 = nn.relu(%12) /* C.graph: aten::relu_, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.upfeature/__module.prediction_layers.0.upfeature.1 */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %14 = nn.conv2d(%13, meta[relay.Constant][4] /* ty=Tensor[(12, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=12, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.bbox_layer_PART_0 */ /* ty=Tensor[(1, 12, 69, 69), float32] */;
  %15 = add(%14, meta[relay.Constant][5] /* ty=Tensor[(1, 12, 1, 1), float32] */) /* ty=Tensor[(1, 12, 69, 69), float32] */;
  %16 = transpose(%15, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 12), float32] */;
  %17 = %1.2;
  %18 = transpose(%17, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 12), float32] */;
  %19 = %1.3;
  %20 = transpose(%19, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 12), float32] */;
  %21 = %1.4;
  %22 = transpose(%21, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 12), float32] */;
  %23 = reshape(%16, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 4), float32] */;
  %24 = reshape(%18, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 4), float32] */;
  %25 = reshape(%20, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 4), float32] */;
  %26 = reshape(%22, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 4), float32] */;
  %27 = (%23, %24, %25, %26);
  %28 = nn.conv2d(%13, meta[relay.Constant][6] /* ty=Tensor[(243, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=243, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.conf_layer_PART_0 */ /* ty=Tensor[(1, 243, 69, 69), float32] */;
  %29 = add(%28, meta[relay.Constant][7] /* ty=Tensor[(1, 243, 1, 1), float32] */) /* ty=Tensor[(1, 243, 69, 69), float32] */;
  %30 = transpose(%29, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 243), float32] */;
  %31 = %1.5;
  %32 = transpose(%31, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 243), float32] */;
  %33 = %1.6;
  %34 = transpose(%33, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 243), float32] */;
  %35 = %1.7;
  %36 = transpose(%35, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 243), float32] */;
  %37 = reshape(%30, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 81), float32] */;
  %38 = reshape(%32, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 81), float32] */;
  %39 = reshape(%34, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 81), float32] */;
  %40 = reshape(%36, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 81), float32] */;
  %41 = (%37, %38, %39, %40);
  %42 = concatenate(%41, axis=-2) /* C.graph: aten::cat, warning: no trace info 1 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %43 = nn.conv2d(%13, meta[relay.Constant][8] /* ty=Tensor[(96, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=96, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.mask_layer_PART_0 */ /* ty=Tensor[(1, 96, 69, 69), float32] */;
  %44 = add(%43, meta[relay.Constant][9] /* ty=Tensor[(1, 96, 1, 1), float32] */) /* ty=Tensor[(1, 96, 69, 69), float32] */;
  %45 = transpose(%44, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 96), float32] */;
  %46 = reshape(%45, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %47 = %1.8;
  %48 = transpose(%47, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 96), float32] */;
  %49 = reshape(%48, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %50 = %1.9;
  %51 = transpose(%50, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 96), float32] */;
  %52 = reshape(%51, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %53 = %1.10;
  %54 = transpose(%53, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 96), float32] */;
  %55 = reshape(%54, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %56 = tanh(%46) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %57 = tanh(%49) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %58 = tanh(%52) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %59 = tanh(%55) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %60 = (%56, %57, %58, %59);
  %61 = concatenate(%27, axis=-2) /* C.graph: aten::cat, warning: no trace info 0 */ /* ty=Tensor[(1, 19173, 4), float32] */;
  %62 = nn.softmax(%42) /* C.graph: aten::softmax, warning: no trace info 4 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %63 = concatenate(%60, axis=-2) /* C.graph: aten::cat, warning: no trace info 2 */ /* ty=Tensor[(1, 19173, 32), float32] */;
  (%61, %62, %63, meta[relay.Constant][10] /* ty=Tensor[(19173, 4), float32] */)
}

@jtuyls
Copy link

jtuyls commented Mar 1, 2022

@abdulazizm I think the issue is the bilinear resize2d layers. From the Vitis AI user guide:

If the mode of the resize is 'BILINEAR', align_corner=false, half_pixel_centers = false, size = 2, 4, 8; align_corner=false, half_pixel_centers = true, size = 2, 4 can be transformed to DPU implementations (pad +depthwisetransposed conv2d). Ifthe mode of the resize is 'NEAREST' and the size is an integer, the resize would be mapped to DPU implementations.

I created a small test case and the DPU compiler couldn't handle these resize layers. I believe the issue is the floating scale from [35, 35] to [69, 69].

@abdulazizm
Copy link
Author

Thanks for the quick reply, @jornt-xilinx . Completely makes sense. Are there any suggestions to improve this further by any chance?

@abdulazizm
Copy link
Author

@jtuyls aten::view, aten::permute operators are running on cpu it seems. DPU is not supporting these?

Tried changing aten::view -> aten::reshape , seems its also pulled on CPU.

Is this expected?

@abdulazizm
Copy link
Author

@abdulazizm I think the issue is the bilinear resize2d layers. From the Vitis AI user guide:

If the mode of the resize is 'BILINEAR', align_corner=false, half_pixel_centers = false, size = 2, 4, 8; align_corner=false, half_pixel_centers = true, size = 2, 4 can be transformed to DPU implementations (pad +depthwisetransposed conv2d). Ifthe mode of the resize is 'NEAREST' and the size is an integer, the resize would be mapped to DPU implementations.

I created a small test case and the DPU compiler couldn't handle these resize layers. I believe the issue is the floating scale from [35, 35] to [69, 69].

@jtuyls I don't think the issue is with scaling_factor. Refer below mod['main'] -> this has resize2d [69,69] to [138,138] in CPU after partition.

fn (%data: Tensor[(1, 3, 550, 550), float32]) -> (Tensor[(1, 19173, 4), float32], Tensor[(1, 19173, 81), float32], Tensor[(1, 19173, 32), float32], Tensor[(19173, 4), float32], Tensor[(1, 138, 138, 32), float32]) {
  %0 = layout_transform(%data, src_layout="NCHW", dst_layout="NHWC") /* ty=Tensor[(1, 550, 550, 3), float32] */;
  %1 = @tvmgen_default_vitis_ai_main_0(%0) /* ty=(Tensor[(1, 69, 69, 12), float32], Tensor[(1, 35, 35, 12), float32], Tensor[(1, 18, 18, 12), float32], Tensor[(1, 9, 9, 12), float32], Tensor[(1, 69, 69, 243), float32], Tensor[(1, 35, 35, 243), float32], Tensor[(1, 18, 18, 243), float32], Tensor[(1, 9, 9, 243), float32], Tensor[(1, 69, 69, 96), float32], Tensor[(1, 35, 35, 96), float32], Tensor[(1, 18, 18, 96), float32], Tensor[(1, 9, 9, 96), float32], Tensor[(1, 69, 69, 256), float32]) */;
  %2 = %1.0;
  %3 = transpose(%2, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 12), float32] */;
  %4 = %1.1;
  %5 = transpose(%4, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 12), float32] */;
  %6 = %1.2;
  %7 = transpose(%6, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 12), float32] */;
  %8 = %1.3;
  %9 = transpose(%8, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 12), float32] */;
  %10 = reshape(%3, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 4), float32] */;
  %11 = reshape(%5, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 4), float32] */;
  %12 = reshape(%7, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 4), float32] */;
  %13 = reshape(%9, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 4), float32] */;
  %14 = (%10, %11, %12, %13);
  %15 = %1.4;
  %16 = transpose(%15, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 243), float32] */;
  %17 = %1.5;
  %18 = transpose(%17, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 243), float32] */;
  %19 = %1.6;
  %20 = transpose(%19, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 243), float32] */;
  %21 = %1.7;
  %22 = transpose(%21, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 243), float32] */;
  %23 = reshape(%16, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 81), float32] */;
  %24 = reshape(%18, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 81), float32] */;
  %25 = reshape(%20, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 81), float32] */;
  %26 = reshape(%22, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 81), float32] */;
  %27 = (%23, %24, %25, %26);
  %28 = concatenate(%27, axis=-2) /* C.graph: aten::cat, warning: no trace info 4 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %29 = %1.8;
  %30 = transpose(%29, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 96), float32] */;
  %31 = reshape(%30, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %32 = %1.9;
  %33 = transpose(%32, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 96), float32] */;
  %34 = reshape(%33, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %35 = %1.10;
  %36 = transpose(%35, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 96), float32] */;
  %37 = reshape(%36, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %38 = %1.11;
  %39 = transpose(%38, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 96), float32] */;
  %40 = reshape(%39, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %41 = tanh(%31) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %42 = tanh(%34) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %43 = tanh(%37) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %44 = tanh(%40) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %45 = (%41, %42, %43, %44);
  %46 = %1.12;
  %47 = layout_transform(%46, src_layout="NHWC", dst_layout="NCHW") /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %48 = image.resize2d(%47, size=[138, 138], roi=[0f, 0f, 0f, 0f], rounding_method="", cubic_alpha=-0.75f) /* C.graph: aten::upsample_bilinear2d, jit._trace.TopLevelTracedModule: __module.proto_net/__module.proto_net.6 */ /* ty=Tensor[(1, 256, 138, 138), float32] */;
  %49 = nn.relu(%48) /* C.graph: aten::relu_, jit._trace.TopLevelTracedModule: __module.proto_net/__module.proto_net.7 */ /* ty=Tensor[(1, 256, 138, 138), float32] */;
  %50 = nn.conv2d(%49, meta[relay.Constant][1] /* ty=Tensor[(32, 256, 1, 1), float32] */, padding=[0, 0, 0, 0], channels=32, kernel_size=[1, 1]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.proto_net/__module.proto_net.8_PART_0 */ /* ty=Tensor[(1, 32, 138, 138), float32] */;
  %51 = add(%50, meta[relay.Constant][2] /* ty=Tensor[(1, 32, 1, 1), float32] */) /* ty=Tensor[(1, 32, 138, 138), float32] */;
  %52 = nn.relu(%51) /* C.graph: aten::relu_, warning: no trace info 0 */ /* ty=Tensor[(1, 32, 138, 138), float32] */;
  %53 = concatenate(%14, axis=-2) /* C.graph: aten::cat, warning: no trace info 3 */ /* ty=Tensor[(1, 19173, 4), float32] */;
  %54 = nn.softmax(%28) /* C.graph: aten::softmax, warning: no trace info 7 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %55 = concatenate(%45, axis=-2) /* C.graph: aten::cat, warning: no trace info 5 */ /* ty=Tensor[(1, 19173, 32), float32] */;
  %56 = transpose(%52, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, warning: no trace info 1 */ /* ty=Tensor[(1, 138, 138, 32), float32] */;
  (%53, %54, %55, meta[relay.Constant][0] /* ty=Tensor[(19173, 4), float32] */, %56)
}

Is there any timeline for supporting bilinear interpolation with this tvm flow?

@jtuyls
Copy link

jtuyls commented Mar 18, 2022

@abdulazizm I think that this (69, 69) -> (128, 128) resize could be offloaded to the DPU indeed, however, the tanh and reshape ops in between are prohibiting putting resize2d in the DPU subgraph (we only support one DPU subgraph).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jtuyls @abdulazizm