How to do batch inference with onnx model?

[MasterGG]

when i do some test for a batchSize inference by onnxruntime, i got error:
InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid rank for input: input.1 Got: 5 Expected: 4 Please fix either the inputs or the model.

the input shape is [1, 3, 224, 224].

how can i do for batch inference? i would be appreciate it , if some examples can show for me?

[tianleiwu]

You will need set dynamic axes during exporting ONNX model. Search "dynamic_axes" in https://pytorch.org/docs/stable/onnx.html.

[MasterGG]

You will need set dynamic axes during exporting ONNX model. Search "dynamic_axes" in https://pytorch.org/docs/stable/onnx.html.

Thanks for your guidance，That's exactly what I was wondering.

[K-prog]

@MasterGG were you able to implement batch inferencing?
I exported my model with a dynamic batch input size.

I used this code to get inference from model

image_path= '001311.jpg'
image = cv2.imread(image_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
img_resized = cv2.resize(image, (256, 256), interpolation=cv2.INTER_CUBIC).astype(np.float32)
image_list = [img_resized, img_resized]
result_array = np.stack(image_list, axis=0)
input_dict = {"image":result_array}
ort_outs = ort_session.run(None, input_dict)

I am still getting dimension error, It is working fine if I pass a single image and the shape of input_dict is (1,256,256,3) but when I stack multiple images and the of input_dict becomes (2,256,256,3), I get the dimension error.

Can you help me out?

[shubhamgoel27]

I'm facing the same issue. I too exported my model with a dynamic batch input size, but it doesn't seem to be helping. Not sure how to do batch inference.

[kopyl]

@shubhamgoel27 i also have no idea how to do the batched inference after exporting an onnx model with dynamic_axes param (tells me it can't handle shape of (10...) (where 10 is amount of batches).

Although i found that you can set a fixed batch size on exporting like this:

dummy_input = torch.randn(10, 3, 300, 300).cuda() 
model = model_efficientnet
input_names = [ "actual_input" ]
output_names = [ "output" ]

torch.onnx.export(model,
                 dummy_input,
                 "model.onnx",
                 verbose=False,
                 input_names=input_names,
                 output_names=output_names,
                 export_params=True,
                 )

look at dummy_input = torch.randn(10, 3, 300, 300).cuda(). It got (10, 3, 300, 300) shape and 10 is the batch size.

And then you can do the inference like this:

def transform_images(image_paths):
    image_list = []
    for image_path in image_paths:
        img = cv2.imread(image_path)
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        
        resized_image = cv2.resize(img, (300, 300))
        image_for_prediction = resized_image.transpose(2, 0, 1)
        image_for_prediction = image_for_prediction.astype('float32') / 255.0
        image_for_prediction = image_for_prediction[np.newaxis, :]
        image_list.append(image_for_prediction)

    result_array = np.stack(image_list, axis=0).squeeze()
    return result_array


def predict_array_of_images(image_paths_array):
    images = transform_images(image_paths_array)
    result = session.run([output_name], {input_name: images})
    predictions = [int(np.argmax(np.array(r))) for r in result[0]]
    labels = [class_names[prediction] for prediction in predictions]
    probabilities = [np.exp(r) / np.sum(np.exp(r)) for r in result[0]]
    scores = [np.max(p) for p in probabilities]

    return labels, scores


res = predict_array_of_images((['download (15).png'] + ['download (17).png']) * 5)
print(res)

Hope it helps.

[TruongNoDame]

Hi @kopyl , your export onnx code with batch is great, but I have a problem: if my batch doesn't have 10 items it gives an error:

index: 0 Got: 2 Expected: 10
Please fix either the inputs or the model.

I look forward to receiving your support on this matter.

[tianleiwu]

@TruongNoDame, please search "dynamic_axes" in torch onnx export document:
https://pytorch.org/docs/stable/onnx_torchscript.html