GPU memory is not released after prediction

[shuo-chen-us]

Hello, I just realized that the GPU memory seems not to be released each time after prediction. I tried paddle.device.cuda.empty_cache(), but it does not work. The GPU memory just keeps increasing as more predictions happen till it reaches 100%. Is there any suggestions? Thanks in advance!

[Evezerest]

hi, there are several reasons that could cause the problem. You can check according to the following steps:

1. The version of PaddlePaddle, we recommend you install the version above 2.3.0.

2. Repeatedly predicting the same image while observing whether the memory consumption will increase (Expected consumption remains unchanged)

3. If there are images with large resolutions, it could be a possible reason for increasing GPU memory consumption.

[shuo-chen-us]

@Evezerest Thanks so much for your reply and your suggestions.

Just provide more details here. We are using a NVIDIA T4 Tensor Core GPU with 16GB memory. We allow maximum 8 PaddleOCR workers running during the peak time.

• When we used MobileNetV3 as OCR detection backbone, the GPU memory usage seemed healthy. The GPU usage kept between 45% - 55% within a week.

• But when we switched to ResNet18 as OCR detection backbone, the GPU memory usage increased from 50% to over 80% in a couple days.

So, based on our observation, we suspect there is a memory leak inside Paddle when using ResNet18. Circling back to your suggestions:

1. We are trying to upgrade PaddlePaddle to v2.3.0. We are using 2.2.1 for now. Does 2.2.1 have any GPU issue as far as you are aware?
2. As I explained above, the GPU consumption of MobileNetV3 seemed unchanged as expected, but ResNet18 just kept increasing.
3. So, question here, even if there are images with large resolutions, should the GPU memory increase when predicting the large image and decrease after the processing is done?

[littletomatodonkey]

1. ResNet18 is much larger than MobileNetV3 model in PP-OCR(100+M vs 10M), you might decresae the workers num for large model inference.
2. paddle2.2.1 does not have any GPU issue from our test result.
3. Paddle needs to apply some gpu memories for further inference process, which is might smaller than resnet18 inference occupation but larger than mobilenet.
4. Actually, if you want to release the gpu memory during inference, you can try to use paddle.inference.Predictor.try_shrink_memory() (link).