regression task memory error

[geoHeil]

classification is working fine for me. However after 50 iterations the data loader is running out of memory in the case of a 1D regression task:

RuntimeError: DataLoader worker (pid 140809) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

I till try to further reduce the batch size, but so far this did not help.

Interestingly, the GPU as well as host memory do not look occupied at all. No where near their limit.

[oguiza]

Hi @geoHeil,
I've seen that issue pop up in the fastai forums from time to time.
Personally, I haven't seen it. It might be because I always work with data on disk (np.memmap). I'm not sure I'm going to be able to help. I do not believe it's a tsai specific issue. I'll investigate it though. In the meantime, you may want to either try np.memmap arrays, read the forum or raise a question there.

[oguiza]

Hi @geoHeil,
Have you tried using np.memmap? Do you still have this issue?

[geoHeil]

Unfortunately, I was blocked the last couple of days. I will try mmap soon! until then, I can close the issue - and re-open when I know more.