Batch Support Error Triton ONNX Backend

[sarperkilic]

Description
Hello,

I have an ONNX model. I am sharing the input and output dimensions of this model below.

I need to deploy this model with Triton Inference Server.

Below is my config file:

name: "segmentation_model"
platform: "onnxruntime_onnx"
max_batch_size: 8
input [
{
name: "input"
data_type: TYPE_FP32
dims: [3, -1, -1 ]
}
]
output [
{
name: "output"
data_type: TYPE_INT64
dims: [ -1,-1,-1]
}
]

When I try to deploy the model with this configfile, I got below error and never solved.

Invalid argument: model 'segmentation_model', tensor 'output': for the model to support batching the shape should have at least 1 dimension and the first dimension must be -1; but shape expected by the model is [1,-1,-1,-1]

I need to batching inference (for example 7 images in one inference), when I try to start tritonserver without config file with this command:

tritonserver --strict-model-config=false --model-repository=triton_model_repository/

I can only inference single image, not batch size>1.

How can I solve this problem?

Thanks

Triton Information
What version of Triton are you using?
tritonserver:21.08-py3

Are you using the Triton container or did you build it yourself?
Docker container

[tanmayv25]

Looks similar to #109

For Triton to support dynamic batching the output shape should be [-1, <dim0>, <dim1>, <dim2>], however, when reading the onnx model the "output" shape was [1, -1, -1, -1]. Note, the first dimension should have been -1.

Is there a way to make batch in [batch, Unsqueezeoutput_dim_1, height, width] to be treated as -1?
@sarperkilic Are you able to run a batch size>1 request on the model outside triton on your model.onnx using onnx runtime?

[mayani-nv]

@tanmayv25 is the batch can be changed as -1 using polygraphy?

$ python3 -m pip install polygraphy
$ polygraphy surgeon sanitize <inut_model.onnx>  -o <output_model.onnx> --override-input-shapes input:[-1,3,height,width]

[sarperkilic]

@tanmayv25 is the batch can be changed as -1 using polygraphy?

$ python3 -m pip install polygraphy
$ polygraphy surgeon sanitize <inut_model.onnx>  -o <output_model.onnx> --override-input-shapes input:[-1,3,height,width]

Hello,

I tried the polygraphy

First, I inspect my model as follow:

$ polygraphy inspect model model.onnx
[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[I] Loading model: /home/mert/Documents/model.onnx
[I] ==== ONNX Model ====
Name: torch-jit-export | ONNX Opset: 11

---- 1 Graph Input(s) ----
{input [dtype=float32, shape=('batch', 3, 'height', 'width')]}

---- 1 Graph Output(s) ----
{output [dtype=int64, shape=('batch', 'Unsqueezeoutput_dim_1', 'height', 'width')]}

---- 74 Initializer(s) ----

---- 165 Node(s) ----

And then, I tried to can the input shape as you said. But I failed. It says no matches found.

$ polygraphy surgeon sanitize model.onnx -o output_model.onnx --override-input-shapes input:[-1,3,height,width]
zsh: no matches found: input:[-1,3,height,width]

What do you suggest me?

Thanks

[sarperkilic]

Looks similar to triton-inference-server/onnxruntime_backend#109

For Triton to support dynamic batching the output shape should be [-1, <dim0>, <dim1>, <dim2>], however, when reading the onnx model the "output" shape was [1, -1, -1, -1]. Note, the first dimension should have been -1.

Is there a way to make batch in [batch, Unsqueezeoutput_dim_1, height, width] to be treated as -1? @sarperkilic Are you able to run a batch size>1 request on the model outside triton on your model.onnx using onnx runtime?

Hello,

I have converted my model input and output tensor shape as follows:

And then I tried to inference using onnx-runtime. It works.

I share input and output logs of onnx-runtime below:

batch array shape: (7, 3, 524, 870)
output shape: (1, 1, 7, 524, 870)

But, when I try to load the model to triton server I got the same error.

tensor 'output': for the model to support batching the shape should have at least 1 dimension and the first dimension must be -1; but shape expected by the model is [1,1,-1,-1]

I also tried to load model to triton server with tritonserver --strict-model-config=false command and run the
curl localhost:8000/v2/models/segmentation_model/config command. Here is the output.

"input":[{"name":"input","data_type":"TYPE_FP32","format":"FORMAT_NONE","dims":[-1,3,-1,-1],
"output":[{"name":"output","data_type":"TYPE_INT64","dims":[1,1,-1,-1]

How can I solve this problem?
Thanks

[tanmayv25]

Triton is still reading the output tensor shape as [1, 1, -1, -1]

I don't understand these shapes in your comment:

batch array shape: (7, 3, 524, 870)
output shape: (1, 1, 7, 524, 870)

Why is there an extra dimension in the output tensor? And why the batch size 7 is present in 3rd dim instead of first.

And then I tried to inference using onnx-runtime. It works.

I presume onnx runtime doesn't apply strict output validation as needed by Triton. Something is wrong with the model, the generated tensor (1, 1, 7, 524, 870) is definitely not compliant with [-1, 1, height, width].

[sarperkilic]

Triton is still reading the output tensor shape as [1, 1, -1, -1]

I don't understand these shapes in your comment:

batch array shape: (7, 3, 524, 870)
output shape: (1, 1, 7, 524, 870)

Why is there an extra dimension in the output tensor? And why the batch size 7 is present in 3rd dim instead of first.

And then I tried to inference using onnx-runtime. It works.

I presume onnx runtime doesn't apply strict output validation as needed by Triton. Something is wrong with the model, the generated tensor (1, 1, 7, 524, 870) is definitely not compliant with [-1, 1, height, width].

I dont know why the output tensor shape has extra dimension.

My model was trained with PyTorch. I still have .pth file.

If you suggest me the way, I can re-convert from .pth to .onnx

When I try batch-size=1 inference on Triton, the output tensor shape is (1,524,870)

[tanmayv25]

Unfortunately, I don't know why it would be happening either. Have you asked it here.

We obtain the dimension for the tensor in onnxruntime_backend here. This is giving [1,1, -1, -1].

I am transferring the issue to onnxruntime_backend project. But I think you may be better off working with pytorch and onnx teams to fix your model to generate outputs in expected format.