Dataset Filtering

[AbhinavTuli]

Adds filtering to datasets using a dictionary as a filter. Changes underlying implementation of datasetview and tensorview for the same, API remains the same as before.
Filtering can only be applied for keys that have product of max_shape less than 100.

Example usage:-

ds = hub.Dataset("username/dataset_name")

# single filter with multiple keys
dsv_combined = ds.filter({"fname": "Active", "lname": "loop"})

# stacking filters
dsv_1 = ds.filter({"id": 1})
dsv_1_subset = dsv_1[10:50]
dsv_2 = dsv_1_subset.filter({"name": "Active"})

[github-actions]

Locust summary

Git references

Initial: ace631f
Terminal: 9d5db62

---

hub/api/tensorview.py

Changes:

• Name: TensorView
  Type: class
  Changed lines: 301
  Total lines: 301
  Changes:
  • Name: TensorView.__init__
    Type: function
    Changed lines: 56
    Total lines: 56
  • Name: TensorView.numpy
    Type: function
    Changed lines: 34
    Total lines: 34
  • Name: TensorView.compute
    Type: function
    Changed lines: 3
    Total lines: 3
  • Name: TensorView.__getitem__
    Type: function
    Changed lines: 56
    Total lines: 56
  • Name: TensorView.__setitem__
    Type: function
    Changed lines: 50
    Total lines: 50
  • Name: TensorView._combine
    Type: function
    Changed lines: 25
    Total lines: 25
  • Name: TensorView.check_slice_bounds
    Type: function
    Changed lines: 10
    Total lines: 10
  • Name: TensorView.dtype_from_path
    Type: function
    Changed lines: 8
    Total lines: 8
  • Name: TensorView.slice_fill
    Type: function
    Changed lines: 12
    Total lines: 12
  • Name: TensorView.__repr__
    Type: function
    Changed lines: 2
    Total lines: 2
  • Name: TensorView.__str__
    Type: function
    Changed lines: 11
    Total lines: 11
  • Name: TensorView.__iter__
    Type: function
    Changed lines: 8
    Total lines: 8
  • Name: TensorView.chunksize
    Type: function
    Changed lines: 2
    Total lines: 2
  • Name: TensorView.is_dynamic
    Type: function
    Changed lines: 2
    Total lines: 2
  • Name: TensorView.disable_lazy
    Type: function
    Changed lines: 2
    Total lines: 2
  • Name: TensorView.enable_lazy
    Type: function
    Changed lines: 2
    Total lines: 2

hub/api/datasetview.py

Changes:

• Name: DatasetView
  Type: class
  Changed lines: 92
  Total lines: 278
  Changes:
  • Name: DatasetView.__init__
    Type: function
    Changed lines: 11
    Total lines: 36
  • Name: DatasetView.__getitem__
    Type: function
    Changed lines: 21
    Total lines: 80
  • Name: DatasetView.__setitem__
    Type: function
    Changed lines: 33
    Total lines: 51
  • Name: DatasetView.filter
    Type: function
    Changed lines: 17
    Total lines: 17
  • Name: DatasetView.__iter__
    Type: function
    Changed lines: 2
    Total lines: 8
  • Name: DatasetView.__len__
    Type: function
    Changed lines: 1
    Total lines: 2
  • Name: DatasetView.__str__
    Type: function
    Changed lines: 1
    Total lines: 2
  • Name: DatasetView.to_tensorflow
    Type: function
    Changed lines: 1
    Total lines: 3
  • Name: DatasetView.to_pytorch
    Type: function
    Changed lines: 1
    Total lines: 13
  • Name: DatasetView.numpy
    Type: function
    Changed lines: 3
    Total lines: 5

hub/api/dataset.py

Changes:

• Name: Dataset
  Type: class
  Changed lines: 43
  Total lines: 1038
  Changes:
  • Name: Dataset.__init__
    Type: function
    Changed lines: 2
    Total lines: 141
  • Name: Dataset.__getitem__
    Type: function
    Changed lines: 11
    Total lines: 65
  • Name: Dataset.__setitem__
    Type: function
    Changed lines: 7
    Total lines: 26
  • Name: Dataset.filter
    Type: function
    Changed lines: 13
    Total lines: 13
  • Name: Dataset.resize_shape
    Type: function
    Changed lines: 1
    Total lines: 11
  • Name: Dataset.to_pytorch
    Type: function
    Changed lines: 4
    Total lines: 35
  • Name: Dataset.to_tensorflow
    Type: function
    Changed lines: 4
    Total lines: 74
    Changes:
• Name: TorchDataset
  Type: class
  Changed lines: 10
  Total lines: 81
  Changes:
  • Name: TorchDataset.__init__
    Type: function
    Changed lines: 2
    Total lines: 11
  • Name: TorchDataset.__len__
    Type: function
    Changed lines: 1
    Total lines: 3
  • Name: TorchDataset.__getitem__
    Type: function
    Changed lines: 7
    Total lines: 27

hub/api/objectview.py

Changes:

• Name: ObjectView
  Type: class
  Changed lines: 264
  Total lines: 264
  Changes:
  • Name: ObjectView.__init__
    Type: function
    Changed lines: 88
    Total lines: 88
  • Name: ObjectView.num_process
    Type: function
    Changed lines: 19
    Total lines: 19
  • Name: ObjectView.process_path
    Type: function
    Changed lines: 30
    Total lines: 30
  • Name: ObjectView.__getitem__
    Type: function
    Changed lines: 73
    Total lines: 73
  • Name: ObjectView.numpy
    Type: function
    Changed lines: 39
    Total lines: 39
  • Name: ObjectView.compute
    Type: function
    Changed lines: 2
    Total lines: 2
  • Name: ObjectView.__str__
    Type: function
    Changed lines: 6
    Total lines: 6

hub/compute/transform.py

Changes:

• Name: Transform
  Type: class
  Changed lines: 5
  Total lines: 399
  Changes:
  • Name: Transform.__getitem__
    Type: function
    Changed lines: 1
    Total lines: 27
  • Name: Transform.upload
    Type: function
    Changed lines: 4
    Total lines: 56

hub/store/dynamic_tensor.py

Changes:

• Name: DynamicTensor
  Type: class
  Changed lines: 8
  Total lines: 484
  Changes:
  • Name: DynamicTensor.get_shape_samples
    Type: function
    Changed lines: 6
    Total lines: 24
  • Name: DynamicTensor.get_shape
    Type: function
    Changed lines: 2
    Total lines: 16

hub/compute/ray.py

Changes:

• Name: RayTransform
  Type: class
  Changed lines: 2
  Total lines: 218
  Changes:
  • Name: RayTransform._func_argd
    Type: function
    Changed lines: 2
    Total lines: 16

hub/exceptions.py

Changes:

• Name: LargeShapeFilteringException
  Type: class
  Changed lines: 4
  Total lines: 4
  Changes:
  • Name: LargeShapeFilteringException.__init__
    Type: function
    Changed lines: 3
    Total lines: 3

hub/store/shape_detector.py

Changes:

• Name: ShapeDetector
  Type: class
  Changed lines: 1
  Total lines: 155
  Changes:
  • Name: ShapeDetector._get_chunks
    Type: function
    Changed lines: 1
    Total lines: 32

hub/api/tests/test_dataset.py

Changes:

• Name: test_datasetview_repr
  Type: function
  Changed lines: 1
  Total lines: 11
• Name: test_dataset_filtering
  Type: function
  Changed lines: 58
  Total lines: 58
• Name: test_dataset_filtering_2
  Type: function
  Changed lines: 10
  Total lines: 10

hub/api/tests/test_objectview.py

Changes:

• Name: test_objectview
  Type: function
  Changed lines: 8
  Total lines: 64
• Name: test_errors
  Type: function
  Changed lines: 1
  Total lines: 39

hub/api/tests/test_tensorview.py

Changes:

• Name: test_tensorview_setitem
  Type: function
  Changed lines: 1
  Total lines: 4

hub/utils.py

Changes:

• Name: _tuple_product
  Type: function
  Changed lines: 5
  Total lines: 5

[codecov]

Codecov Report

Merging #460 (9bd63d3) into master (4f537c5) will increase coverage by 0.08%.
The diff coverage is 89.52%.

@@            Coverage Diff             @@
##           master     #460      +/-   ##
==========================================
+ Coverage   88.20%   88.29%   +0.08%     
==========================================
  Files          50       52       +2     
  Lines        3664     3724      +60     
==========================================
+ Hits         3232     3288      +56     
- Misses        432      436       +4     

Impacted Files	Coverage Δ
hub/schema/sequence.py	100.00% <ø> (ø)
hub/schema/text.py	100.00% <ø> (ø)
hub/api/datasetview.py	91.39% <88.57%> (-0.04%)	⬇️
hub/api/tensorview.py	88.70% <88.70%> (ø)
hub/api/objectview.py	88.96% <88.96%> (ø)
hub/api/dataset.py	90.17% <90.69%> (+0.29%)	⬆️
hub/compute/ray.py	96.35% <100.00%> (ø)
hub/compute/transform.py	93.42% <100.00%> (ø)
hub/exceptions.py	100.00% <100.00%> (ø)
hub/store/dynamic_tensor.py	88.54% <100.00%> (+0.24%)	⬆️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f537c5...9bd63d3. Read the comment docs.