Passing dict in datapipe/dataset will have memory leak problem #103581
Labels
module: data
torch.utils.data
triaged
This issue has been looked at a team member, and triaged and prioritized into an appropriate module
🐛 Describe the bug
Passing dict in datapipe or dataset will casuse memory leak
output:
It is clear that is pasing the dict of tensor memory will leak but list of tensor will not.
I used dict of tensor in my model training, and I found the training faied multiple times all since of memory leak. And I tried to used Tensordict(https://pytorch.org/rl/tensordict/), but it cannot contains the string. I need string during my datapipes passing (str to tensor encode in one of datapipes).
copy from: pytorch/data#1183
Versions
Versions
torch version: 2.0.0
torchdata version: 0.6.0
cc @VitalyFedyunin @ejguan @dzhulgakov
The text was updated successfully, but these errors were encountered: