Dataloader stuck

[walkerning]

Occasionally, the data loader in the search process will stuck at some point... Usually the first time when the controller queue is used. This might be related to this issue: pytorch/pytorch/issues/1355.
pytorch/pytorch/issues/1355#issuecomment-308587289 said this issue might be related to shm running out. But there are 32G shm configured, and the actual usage is never close to that.
Try adding some swap space to avoid data handling thread being killed due to running out of memory (not work).

Seems this might be also due to calling iter on the data loader too early, and not used it for a long while.

[walkerning]

Might be mitigated in f0a4d60.
Maybe add a timeout while waiting and reinit the infinite iterator?

[walkerning]

After f0a4d60, I haven't observe any data loader stuck anymore. Maybe it's already resolved.