Any body has the problem of Memory Explosion?

[zhangzhaocs]

Hi Tim,
When I run this code, the memory usage is getting bigger and finally got an explosion.
I was wondering if you have the same problem and what's the solution?

Looking forward to your kind feedback.
Thanks,
tsingker88

[TimDettmers]

I worked on this problem in the past and could resolve some issues, but it seems that there are still issues with the background loader NumPy pointers that are not released.

Two things that might help in this case is: (1) Use fewer loader_threads for the Streambatcher, (2) play around reducing the samples_per_file variable in the StreamToHDF5 class (I think increasing the samples per file should improve the memory profile, but I am not sure anymore).

Let me know if that helps!

[zhangzhaocs]

Your suggestion really helps.
Thank, Tim.

[1780041410]

How did you solve this problem? Can you email to me?

[1780041410]

1780041410@qq.com