Adding back transforms for parallel dataset uploads

[AbhinavTuli]

🚀 🚀 Pull Request

Checklist:

• My code follows the style guidelines of this project and the Contributing document
• I have commented my code, particularly in hard-to-understand areas
• I have kept the coverage-rate up
• I have performed a self-review of my own code and resolved any problems
• I have checked to ensure there aren't any other open Pull Requests for the same change
• I have described and made corresponding changes to the relevant documentation
• New and existing unit tests pass locally with my changes

Changes

• This PR adds support for transforms after support was broken in a previous release. The API for the same has also been tweaked.
• Also fixes extraneous print statements on slicing datasets.

Currently does not support:-

• Corner chunk merging similar to the one we had in previous version of transforms for v2.
• Inplace transforms

Some parts of the code can be further simplified once we get rid of cachable.

[AbhinavTuli]

I'm using this class with interface very similar to Dataset to avoid chunk engine overhead while storing samples during transforms. The append, extend and numpy syntax should be very similar to Dataset currently, but might not work for advanced indexing.

[AbhinavTuli]

Need to add docstrings

[verbose-void]

can we try to avoid Shard naming? it's kind of confusing

[AbhinavTuli]

any suggestions?

[verbose-void]

none come to mind, but i feel like we use Shard and Chunk interchangably

[verbose-void]

what are the differences between TransformDatasetShard and TransformDatasetTensor?

also, maybe we can drop the Dataset from both class names?

[AbhinavTuli]

they're literally basic version of Dataset and Tensor respectively, used for the sample_out argument.
The underlying implementation does not involve any chunking/compression though, raw data is just stored.

[verbose-void]

in that case, is it possible to use Dataset & Tensor?

if i'm a user, a likely thing i will do is print out the variables, and if i see Dataset/Tensor i will know exactly what they are.

[verbose-void]

also it makes more sense to me to call these "View"s even if it's not as technically descriptive as "Shard"

[AbhinavTuli]

yeah but this isn't exactly a Dataset/Tensor, it's just a dummy class representing it. So a lot of the methods/attributes of Dataset won't work.

[AbhinavTuli]

What about TransformDataset and TransformTensor?

[verbose-void]

yep that works well

[verbose-void]

quite a few comments, but most logic looks pretty good. great code organization and docstrings!

[farizrahman4u]

Nice work, will have to refac some the transform utils to base encoder.. but we can do that later.

[verbose-void]

yessir