dynamic shape

[Qiancj0605]

@ttyio Hi，If the input batchsize is 50，what is the batchsize during inference？

Set as follows：dynamic shape([-1,4561])
auto profile = builder->createOptimizationProfile();
profile->setDimensions(“inputname”, OptProfileSelector::kMIN, Dims{8, 4561});
profile->setDimensions(“inputname”,, OptProfileSelector::kOPT, Dims{32, 4561});
profile->setDimensions(“inputname”,, OptProfileSelector::kMAX, Dims{64, 4561});

[ttyio]

Hi @Qiancj0605 , you need set_binding_shape before inference.
https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Core/ExecutionContext.html#tensorrt.IExecutionContext.set_binding_shape

reference code https://github.com/NVIDIA/TensorRT/blob/master/demo/BERT/inference.py#L150

[Qiancj0605]

Maybe I didn't describe it clearly. I mean, after setDimensions, if the input batchsize(50) is greater than kOPT(batchsize:32) and less than kMAX(batchsize:64), what is the actual batchsize(32 or 50 or other?) in the calculation? @ttyio

[ttyio]

@Qiancj0605 , trt has symbolic system, and the actual batchsize is the same as the one you specified in set_binding_shape.

[Qiancj0605]

I don't understand? If I call createOptimizationProfile，The purpose is to automatically adapt the following batch(kMIN,kOPT,kMAX) during inference based on uncertain input. And during inference, I will set setOptimizationProfile。
What I want to express is not how to do it, but assuming that my code is written correctly, what is the specific batchsize during inference？
Thanks@ttyio

[ttyio]

Hello @Qiancj0605 , during inference, the input shape is determined before enqueue, right? You need call set_binding_shape whenever the input shape changed. Else the behavior is undefined.
(kMIN,kOPT,kMAX) is just build time hint, used to filter out invalid tactics, and select the optimal one, but we still need setup the tactic using the correct batch size before enqueue.

[Qiancj0605]

Thanks，Maybe I understand that there is still a problem, I will look at the relevant documents。

[Qiancj0605]

@ttyio Thanks,I figured it out。
I still have a small question。How to understand "kernel selection" here？Thanks

[ttyio]

Hi @Qiancj0605 ,
we will first filter all the valid algorithm using [kMIN, kMAX], then setup the problem size to kOPT in the filter out algorithms. finally choose the fastest one.

So here kernel selection means that the kOPT is used in the step to select the fastest one.

[ttyio]

I will close this since the question is answered, thanks!