Memory leak

[zkdfbb]

When I upgrade to 0.5.0 I found a huge memory usage growth. I checked version 0.2.0(include 0.3), the memory growth as:
4826, 5096, 5302
and 0.5.0(include 0.4.1) growth as:
6754, 8970, 11114

and when run evaluation, the last batch maybe smaller, so every evaluation I re read the datasets(tfrecords), the result under version 0.2.0 is:
4826, 5294, 5510

How can I solve this problem?

[JanuszL]

Hi,
This problem was reported in #328 as well. Now we don't have a good, general solution for that.
Could you tell more about pipeline do you use, what data set and etc?
Maybe this will help us to narrow down the problem you experiencing.
Tracked as DALI-425.
Br,
Janusz

[zkdfbb]

@JanuszL my own dataset, not public, using TFRecordPipeline. I'm afraid there is a bug between 0.3.0 and 0.4.0

[JanuszL]

Can you check this on some publicly available data set, so we have 1005 reproduction here as well?
Otherwise even if we fix something we cannot tell if it works for you.

[mzient]

Regarding host memory usage:
We're pleased to say that we've changed the allocation strategy for (non-pinned) CPU buffers. It reduces the memory consumption in RN50 training in PyTorch by almost 50%. Please check the latest master (or next successul nightly) and see if your issue is resolved.
The memory is now freed when a requested tensor is smaller than a given percentage of actual allocation. You can tweak it by setting the environment variable DALI_HOST_BUFFER_SHRINK_THRESHOLD=0.xx. The default value is 0.9. You can also set it in python using nvidia.dali.backend.SetHostBufferShrinkThreshold(threshold).

[JanuszL]

0.19 is out and should address this. Please reopen if it doesn't work.