GPU deployment unable to get the performance benefits

[srinivaskotamraju]

In the Benchmarking section, there are improvements when the inference server is deployed against a GPU infrastructure. We are using Tesla V100-PCIE-16GB, however we are not seeing any improvements in the inference time as published in the benchmarks. NOT looking for similar numbers, however expecting to see some improvements in the numbers because of the GPU infrastructure.

We are using the deployment command as provided "sudo NV_GPU=0 nvidia-docker......".

Do we need to enable any other options in the GPU machine?

[JoeSL]

Hello @srinivaskotamraju.

I would recommend to monitor the GPU utilization before and after running the docker-container to see if the GPU is being used.
For that use the following command: watch -n 0.1 nvidia-smi
This will refresh the GPU utilization data at a rate of 0.1 seconds.

This way you'll get a better insight on whether the GPU is being used and at what rate and also if other processes are using the GPU thus leaving the API with little amount of memory left.

In case you have multiple gpus on the machine, try replacing the gpu number in the NV_GPU variable to the number of the GPU you wish to use to check if the problem persists