Added distributed loading of PyTorch DiskDataset

[arunppsg]

Description

Added support for loading disk dataset in multiple processes.

Type of change

Please check the option that is related to your PR.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • In this case, we recommend to discuss your modification on GitHub issues before creating the PR
• Documentations (modification for documents)

Checklist

• My code follows the style guidelines of this project
  • Run yapf -i <modified file> and check no errors (yapf version must be 0.32.0)
  • Run mypy -p deepchem and check no errors
  • Run flake8 <modified file> --count and check no errors
  • Run python -m doctest <modified file> and check no errors
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New unit tests pass locally with my changes
• I have checked my code and corrected any misspellings

[vsaravind01]

The changes look good to me.

[rbharath]

@arunppsg Can we add a unit test for the changed functionality?

[rbharath]

Documenting offline discussion, this is proving difficult to write. The existing unit tests pass. Will come back with additional tests in future PR

[rbharath]

@arunppsg Is there any risk for breakage with these changes? What code uses TorchDiskDataset already?

[arunppsg]

@arunppsg Is there any risk for breakage with these changes? What code uses TorchDiskDataset already?

DiskDataset.make_pytorch_dataset uses TorchDiskDataset method to make an iterable pytorch dataset. In these code snippets, we are only updating the process of distributing the shards across multiple workers in a single machine, so that each gpu gets a different copy of the dataset.

[rbharath]

LGTM