How to run multi models at the same time with GPU?

[xinsuinizhuan]

System information

• OS Platform and Distribution:windows10
• ONNX Runtime installed from (source or binary):1.9.0
• ONNX Runtime version:1.9.0
• Python version:
• Visual Studio version (if applicable):vs2019
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: cuda11.4
• GPU model and memory: cuda11.4
  with the C++，i create three models in one dll with GPU model, but the foward is so slow. with one model it so fast.

[xinsuinizhuan]

I have three camera, and i create the threes threads, different camera have different models, but they run at same time. How to run thems at same time? With one camera, one thread and one model, it is so fast. But when i run three, it is so slow at the same time.

[hariharans29]

Can you elaborate on what you mean by "how to run them at same time " ? From your first sentence, I am guessing they are running at the same time. Are they currently all running on the same card ? What is the card you are using ? With three models running at the same time on the same card, does resource contention mean there will be some noticeable perf drop ?

[xinsuinizhuan]

I have three cameras, every camare i use a thread and load itself onnx models, then trigger the three camaras at the same time, my card is one nvidia 3080ti, so the three models run on one card at the same time.

[hariharans29]

Hard to say without knowing how heavy your models are, but I think this is expected that if you run multiple heavy models at the same time on one card, there is going to be resource contention that all models have to do contend with. We do not have anything special in our code that slows things down when there are multiple models running at the same time.

[xinsuinizhuan]

Oh, thanks. Other question is, how to forward with bash of iamges at same time, C++?

[xinsuinizhuan]

Add another problem, I run the model on nvidia 1060 killtime 30ms， but when i run it on nvdia 3080, it still 30ms， i think it should faster, but now it same as?

[hariharans29]

Oh, thanks. Other question is, how to forward with bash of iamges at same time, C++?

Your model needs to support batching. Usually the input shape will have a dynamic shape in it (if you open it in Netron). It will be something like [batch, 3, 256, 256] and if this is the case you can send in an input tensor containing any batch of images. If your model doesn't batching, ORT can't support batching.

[hariharans29]

Add another problem, I run the model on nvidia 1060 killtime 30ms， but when i run it on nvdia 3080, it still 30ms， i think it should faster, but now it same as?

Where did you get the model from (i.e.) what is the base framework ? What is the base framework's model's perf on 3080 ?

[hariharans29]

Closing this for now. If you have a model that has better perf in the base framework than in ORT, please open a new issue with relevant details (code to repro, model, build flavor, platform, etc.)