Improvements to to_tensorflow, multi worker transforms, dynamic tensor slicing

[AbhinavTuli]

• to_tensorflow now supports a new argument that only passes certain tensors to it and speeds up iteration time in case multiple extra tensors are present.
• caching present within to_tensorflow has been improved to tensors with dynamic shapes (earlier it was saving only the current sample in the cache)
• adds the option to specify None as compressor while defining the schema
• adds the ability to slice dynamically shaped tensors and obtain a list instead of iterating over them one by one.
• transform logic has been modified to work properly with multiple workers
• Any dataset copy test that got interrupted midway through the test affected all subsequent test runs. This has now been fixed.

Relevant tests have been added in test_dataset and test_converters that can be used to try out the new features.

[github-actions]

Locust summary

Git references

Initial: 85a8004
Terminal: d88c5fb

---

hub/api/integrations.py

Changes:

• Name: _to_tensorflow
  Type: function
  Changed lines: 25
  Total lines: 122
  Changes:
  • Name: _to_tensorflow._get_active_item
    Type: function
    Changed lines: 2
    Total lines: 14
  • Name: _to_tensorflow.tf_gen
    Type: function
    Changed lines: 2
    Total lines: 24
  • Name: _to_tensorflow.dict_to_tf
    Type: function
    Changed lines: 5
    Total lines: 8
  • Name: _to_tensorflow.dtype_to_tf
    Type: function
    Changed lines: 2
    Total lines: 11
  • Name: _to_tensorflow.get_output_shapes
    Type: function
    Changed lines: 2
    Total lines: 7
  • Name: _to_tensorflow.output_shapes_from_dict
    Type: function
    Changed lines: 5
    Total lines: 8

hub/api/dataset.py

Changes:

• Name: Dataset
  Type: class
  Changed lines: 5
  Total lines: 1089
  Changes:
  • Name: Dataset._get_compressor
    Type: function
    Changed lines: 3
    Total lines: 15
  • Name: Dataset.to_tensorflow
    Type: function
    Changed lines: 2
    Total lines: 14

hub/store/dynamic_tensor.py

Changes:

• Name: DynamicTensor
  Type: class
  Changed lines: 21
  Total lines: 488
  Changes:
  • Name: DynamicTensor.get_real_shape
    Type: function
    Changed lines: 6
    Total lines: 12
  • Name: DynamicTensor.__getitem__
    Type: function
    Changed lines: 14
    Total lines: 18

hub/api/datasetview.py

Changes:

• Name: DatasetView
  Type: class
  Changed lines: 2
  Total lines: 320
  Changes:
  • Name: DatasetView.to_tensorflow
    Type: function
    Changed lines: 2
    Total lines: 13

hub/compute/transform.py

Changes:

• Name: Transform
  Type: class
  Changed lines: 25
  Total lines: 414
  Changes:
  • Name: Transform.upload
    Type: function
    Changed lines: 25
    Total lines: 71
    Changes:

hub/store/metastore.py

Changes:

• Name: MetaStorage
  Type: class
  Changed lines: 1
  Total lines: 133
  Changes:
  • Name: MetaStorage.__setitem__
    Type: function
    Changed lines: 1
    Total lines: 17

hub/api/tests/test_converters.py

Changes:

• Name: test_to_tensorflow_key_list
  Type: function
  Changed lines: 37
  Total lines: 37

hub/api/tests/test_dataset.py

Changes:

• Name: test_dataset_dynamic_shaped_slicing
  Type: function
  Changed lines: 24
  Total lines: 24
• Name: test_tensorview_slicing
  Type: function
  Changed lines: 2
  Total lines: 9

hub/utils.py

Changes:

• Name: batchify
  Type: function
  Changed lines: 4
  Total lines: 11

[codecov]

Codecov Report

Merging #689 (4507364) into master (58c65c4) will decrease coverage by 0.04%.
The diff coverage is 97.46%.

@@            Coverage Diff             @@
##           master     #689      +/-   ##
==========================================
- Coverage   89.14%   89.10%   -0.05%     
==========================================
  Files          58       58              
  Lines        4266     4296      +30     
==========================================
+ Hits         3803     3828      +25     
- Misses        463      468       +5     

Impacted Files	Coverage Δ
hub/api/tensorview.py	90.40% <ø> (-0.10%)	⬇️
hub/api/integrations.py	87.53% <90.90%> (-0.11%)	⬇️
hub/api/dataset.py	89.57% <100.00%> (-0.85%)	⬇️
hub/api/dataset_utils.py	95.31% <100.00%> (+1.02%)	⬆️
hub/api/datasetview.py	91.50% <100.00%> (ø)
hub/compute/transform.py	93.57% <100.00%> (+0.15%)	⬆️
hub/store/dynamic_tensor.py	88.77% <100.00%> (+0.23%)	⬆️
hub/store/metastore.py	90.65% <100.00%> (ø)
hub/utils.py	81.67% <100.00%> (+0.28%)	⬆️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 58c65c4...4507364. Read the comment docs.

[imshashank]

Minor comment. Please update that and ready for merge.