Memory Leak During Training with Large Datasets

[halilberkdergi]

Hello,

Firstly, I want to thank you for your incredible work on this project; it's really impressive.

I am encountering an issue where the memory usage continually increases with every batch during training, especially noticeable when working with large datasets exceeding 5 million sessions. Despite all computations being conducted on CUDA, there seems to be a significant memory leak.

Could you provide any insights or suggestions on why this might be happening and how to resolve it?

Thank you.

Best regards,