the docker can't isolate mulitple gpu cards on one machine

[caimouli]

1. Issue or feature description

There is a nvidia docker decoder and cuda function performance issue on mutiple cards we have meet. We have test on Geforce 2080 Ti and Tesla T4 cards. We used 2 cards on one server to test. At beginning, we just use nvidia docker work on cenos 7, we found that when we use 2 cards working at the same time, the performance reduced almost to half. Then we use vmware to create 2 virtual machines on one server with gpu passthrough technology, each vm bound with one card, the result show each card arrive it's best performance.The test case and snaps as follow:
The program we use nvidia Video_Codec_SDK_11.1.5 to decode , then we write cuda code transform nv12 to bgr:

2. Steps to reproduce the issue

1. we just run one docker cantainer on one gpu card.we put 30 channels of rtsp video to test, the performance is good. we can get valid picture from the program, the fps kept on more than 25.
   
   nvidia-smi pmon
   
   cuda function elapsed time
   

2. then we run one docker container with 2 gpu cards to test. we put 60 channels of rtsp video to test, the result reduce to half
   
   nvidia-smi pmon
   
   cuda function elapsed time
   

3. after that , we run two docker containers with 2 gpu cards, each container bound each card. we put 60 channels of rtsp video to test. we make sure that each container have 30 channels. the result is the same as on docker container with 2 cards.

4. so we think the container can't isolate the gpu card very well, then we use vmware to test.we created 2 virtual machines to test. we passthrough 2 cards to each machine.then run one container in each vm.The performance is well, each card is the same as the first case.

as the result, we get the issue: the docker can't isolate mulitple gpu cards, the performance reduced very quickly on mutiple gpu cards.

[klueska]

Can you show me your docker command line options?

[caimouli]

Can you show me your docker command line options?

docker run -d --runtime=nvidia --restart=always -v /etc/localtime:/etc/localtime -e NVIDIA_VISIBLE_DEVICES=N -e NVIDIA_DRIVER_CAPABILITIES=video,compute,utility -e LANG=C.UTF-8 -p xxx:xxx -v /tmp:/tmp -e DISPLAY=$DISPLAY --cap-add=SYS_PTRACE --security-opt seccomp=unconfined city_manager:v2.0