Support dynamic batch inference with onnx/onnxruntime

[zhiqwang]

🚀 Feature

Support dynamic batch inference with onnx/onnxruntime.

Motivation

As @makaveli10 pointed out in #39 (comment), the current implementation of onnx/onnxruntime mechanism only supports dynamic shapes inference, not dynamic batch size.

I didn't know how to implement the dynamic batch inference, any help is welcome here.

[makaveli10]

Thanks. I'll run some experiments and see how it goes.

[timmh]

I followed the ONNX deployment walkthrough and run export_onnx(..., skip_preprocess=True). However, during the inference using PredictorORT(weights).predict(inputs) I get the following error when trying to use a batch size different from 1:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Split node. Name:'Split_48' Status Message: Cannot split using values in 'split' attribute. Axis=0 Input shape={10,3,450,600} NumOutputs=1 Num entries in 'split' (must equal number of outputs) was 1 Sum of sizes in 'split' (must equal size of selected axis) was 1

This error occcurs independently of the batch_size parameter used in tracing. The model I export is just a vanilla YOLOv5 model.

[zhiqwang]

Hi @timmh , The default example only supports ONNX models with preprocessing, could you please open a new ticket about inferencing without preprocessing for easier to track this issue?