PicklingError encountered when using multiple GPUs

[TotalVariation]

🐛 Bug

Hello,

I was using Slurm-based HPC with multiple GPUs to train deep learning models.

I believe this pickle error was caused by multiprocessing but I have little knowledge about this. Thank you so much.

Environment

cc @ssnl @VitalyFedyunin @ejguan @NivekT

[ejguan]

Potentially the problem came from your implementation of Dataset based on your trace.

Could you please post more detail about it so we can reproduce the error?

[TotalVariation]

Thank you @ejguan for your quick response.

I believe this error is related to multiprocessing used in Torch. It would force everything to be pickled. But there are some packages that are dependent on sh in my code, which contains a class called GlobResults that is not meant for pickling. Here is the same issue I posted on amoffat/sh#590, which you can refer to.

[ecederstrand]

Something is calling sh.glob(). Assuming this happens in your own code, you need to decide whether you want the globbing to happen before or after spawning the process. Either:

1. Pass around the arguments for sh.glob() and run the command in the spawned process
2. Convert the result of sh.glob() to something that can be pickled, and pass the result around:

>>> import pickle, sh
>>> pickle.dumps(sh.glob('*'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'sh.GlobResults'>: attribute lookup GlobResults on sh failed
>>> pickle.dumps(list(sh.glob('*')))
b'...'

[TotalVariation]

@ecederstrand Thank you again for your solution. This is the exact reason why there would be a pickling error. I have solved the issue by following your code snippet.

[NivekT]

Seems like this issue is resolved, so I will close this.

@TotalVariation let us know if there is any further issue or suggestion for changes.