Skip to content

Conversation

@gsakkis
Copy link
Contributor

@gsakkis gsakkis commented Aug 19, 2022

Bump pytorch to 1.12 to support torchdata introduced in #173:

  • torchdata is available for torch 1.11 or later. This was not caught by the CI when testing against torch 1.10 because torchdata was installed after torch and triggered a reinstallation to the latest version (1.12).
  • the ShardingFilter of torchdata doesn't seem to work at version 0.3 (required for torch 1.11); all data are processed by all workers. In torchdata 0.4 / torch 1.12 sharding works correctly.

Also torch 1.12 has better support than 1.11 for sparse CSR tensors. The pytorch_data_api_tiledb_sparse.ipynb example notebook fails at the last cell on torch 1.11 if TensorKind.SPARSE_CSR (default for 2D sparse arrays) is passed in ArrayParams.

@gsakkis gsakkis requested a review from georgeSkoumas August 19, 2022 16:18
Base automatically changed from gsa/sc-20616/pytorchtiledbdataloader-can-be-iterated-only-once to master August 22, 2022 13:28
@gsakkis gsakkis merged commit 744a7bf into master Aug 22, 2022
@gsakkis gsakkis deleted the gsa/bump-pytorch branch August 22, 2022 13:29
@gsakkis gsakkis mentioned this pull request Sep 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants