fix for multiprocess dataloader

[peterdudfield]

Pull Request

Description

Fix bug for mulit process dataloader.

referece: fsspec/gcsfs#379

Fixes issue #

How Has This Been Tested?

-usual unittests

• run a ML model using this new code

• No

• Yes

Checklist:

• My code follows OCF's coding style guidelines
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked my code and corrected any misspellings

[JackKelly]

Looks good!

It's a much better idea to call set_fsspec_for_multiprocess() in worker_init_fn(): I'm not sure why I hadn't thought of that before!

Now that you're calling set_fsspec_for_multiprocess() in worker_init_fn(), it might be good to remove utils.set_fsspec_for_multiprocess() from line 206 in satellite_data_source.py.

I don't think there's any harm in calling set_fsspec_for_multiprocess() multiple times, but it maybe helps keep the code clean and easier to debug in the future, maybe?