VideoReader - how to deallocate pipeline (release the GPU memory)?

[luizgh]

Hi,

I am trying to use a pipeline with VideoReader for an activity detection detection task.

I want to run training and validation in the same script (validation after each epoch), and since the pipeline takes a lot of GPU memory, I cannot initialize both training and validation pipelines at the same time.

I want to initialize the training pipeline in the beginning of the epoch, and release it at the end (so that I can initialize the validation pipeline). However, I can't find a way for the pipeline to release the GPU memory (I saw issue #1048, but the person didn't say how he got it to work). Example code, using GC to delete the pipeline object:

from nvidia.dali.pipeline import Pipeline
import nvidia.dali.ops as ops
import nvidia.dali.types as types
import gc


class VideoPipe(Pipeline):
    def __init__(self, file_list, sequence_length, batch_size, device_id, shuffle):
        super(VideoPipe, self).__init__(batch_size, num_threads=2, device_id=device_id, seed=16, exec_async=False, exec_pipelined=False)

        self.input = ops.VideoReader(device="gpu", file_list=file_list, sequence_length=sequence_length,
                                     shard_id=0, num_shards=1, file_list_frame_num=True, enable_frame_num=True,
                                     random_shuffle=shuffle, initial_fill=16, step=1, dtype=types.DALIDataType.FLOAT)

    def define_graph(self):
        output = self.input(name="Reader")
        return output


pipe = VideoPipe('/disk/events_dali.txt', sequence_length=25, batch_size=8, device_id=0, shuffle=True)
pipe.build()
data = pipe.run()

del data
del pipe

gc.collect()  # Memory is not released

PS: I am setting exec_async and exec_pipelined to false so that I can use pytorch ops for data augmentation, while these are not natively supported in DALI

[JanuszL]

Strange, doesn't work for me either. You can try to:

pipe._pipe = None

Works in my case. @mzient any idea what is going on there?

[luizgh]

Thanks @JanuszL . This solves my problem. I'll leave the issue open, in case you want to track this (e.g. why setting pipe = None doesn't collect the memory), but feel free to close it.

[JanuszL]

@luizgh - Indeed there is some reference leak. #1845 should fix this problem.

[JanuszL]

0.21 is available and should fix this problem.