Fix/circleci

[haiyangdeperci]

This PR introduces Windows and MacOS tests, includes conda and refactored some of the linux test steps.

[github-actions]

Locust summary

Git references

Initial: 06cab25
Terminal: a508eb7

---

hub/api/dataset.py

Changes:

• Name: Dataset
  Type: class
  Changed lines: 174
  Total lines: 1087
  Changes:
  • Name: Dataset._get_dynamic_tensor_dtype
    Type: function
    Changed lines: 7
    Total lines: 7
  • Name: Dataset._get_compressor
    Type: function
    Changed lines: 13
    Total lines: 13
  • Name: Dataset._generate_storage_tensors
    Type: function
    Changed lines: 2
    Total lines: 25
  • Name: Dataset.to_tensorflow
    Type: function
    Changed lines: 2
    Total lines: 14
  • Name: Dataset.from_directory
    Type: function
    Changed lines: 148
    Total lines: 149
    Changes:

hub/api/integrations.py

Changes:

• Name: _to_tensorflow
  Type: function
  Changed lines: 13
  Total lines: 110
  Changes:
  • Name: _to_tensorflow._get_active_item
    Type: function
    Changed lines: 1
    Total lines: 13
  • Name: _to_tensorflow.dict_to_tf
    Type: function
    Changed lines: 2
    Total lines: 5
  • Name: _to_tensorflow.dtype_to_tf
    Type: function
    Changed lines: 2
    Total lines: 11
  • Name: _to_tensorflow.get_output_shapes
    Type: function
    Changed lines: 2
    Total lines: 7
  • Name: _to_tensorflow.output_shapes_from_dict
    Type: function
    Changed lines: 2
    Total lines: 5

hub/api/tests/test_dataset.py

Changes:

• Name: test_dataset_from_directory
  Type: function
  Changed lines: 49
  Total lines: 49
  Changes:
  • Name: test_dataset_from_directory.create_image
    Type: function
    Changed lines: 8
    Total lines: 8
  • Name: test_dataset_from_directory.data_in_dir
    Type: function
    Changed lines: 6
    Total lines: 6
  • Name: test_dataset_from_directory.root_dir_image
    Type: function
    Changed lines: 9
    Total lines: 9
  • Name: test_dataset_from_directory.del_data
    Type: function
    Changed lines: 5
    Total lines: 5
• Name: test_tensorview_slicing
  Type: function
  Changed lines: 2
  Total lines: 9
• Name: test_dataset_copy_s3_local
  Type: function
  Changed lines: 2
  Total lines: 14
• Name: test_dataset_copy_gcs_local
  Type: function
  Changed lines: 2
  Total lines: 14
• Name: test_dataset_copy_azure_local
  Type: function
  Changed lines: 5
  Total lines: 21
• Name: test_dataset_copy_hub_local
  Type: function
  Changed lines: 2
  Total lines: 14
• Name: test_dataset_copy_gcs_s3
  Type: function
  Changed lines: 2
  Total lines: 14
• Name: test_dataset_copy_exception
  Type: function
  Changed lines: 2
  Total lines: 12
• Name: test_dataset_filter_3
  Type: function
  Changed lines: 2
  Total lines: 19

hub/compute/transform.py

Changes:

• Name: Transform
  Type: class
  Changed lines: 12
  Total lines: 399
  Changes:
  • Name: Transform.create_dataset
    Type: function
    Changed lines: 1
    Total lines: 16
  • Name: Transform.upload
    Type: function
    Changed lines: 10
    Total lines: 56
    Changes:
  • Name: Transform.store
    Type: function
    Changed lines: 1
    Total lines: 80

hub/store/dynamic_tensor.py

Changes:

• Name: DynamicTensor
  Type: class
  Changed lines: 18
  Total lines: 484
  Changes:
  • Name: DynamicTensor.__init__
    Type: function
    Changed lines: 1
    Total lines: 114
  • Name: DynamicTensor.__getitem__
    Type: function
    Changed lines: 14
    Total lines: 24
  • Name: DynamicTensor.get_shape
    Type: function
    Changed lines: 3
    Total lines: 16

hub/api/datasetview.py

Changes:

• Name: DatasetView
  Type: class
  Changed lines: 2
  Total lines: 320
  Changes:
  • Name: DatasetView.to_tensorflow
    Type: function
    Changed lines: 2
    Total lines: 13

hub/api/tensorview.py

Changes:

• Name: TensorView
  Type: class
  Changed lines: 7
  Total lines: 335
  Changes:
  • Name: TensorView.numpy
    Type: function
    Changed lines: 7
    Total lines: 56

hub/client/base.py

Changes:

• Name: HubHttpClient
  Type: class
  Changed lines: 5
  Total lines: 104
  Changes:
  • Name: HubHttpClient.request
    Type: function
    Changed lines: 5
    Total lines: 52

hub/schema/class_label.py

Changes:

• Name: ClassLabel
  Type: class
  Changed lines: 14
  Total lines: 161
  Changes:
  • Name: ClassLabel.__init__
    Type: function
    Changed lines: 13
    Total lines: 84

hub/schema/image.py

Changes:

• Name: Image
  Type: class
  Changed lines: 6
  Total lines: 75
  Changes:
  • Name: Image.__init__
    Type: function
    Changed lines: 1
    Total lines: 48

hub/store/metastore.py

Changes:

• Name: MetaStorage
  Type: class
  Changed lines: 1
  Total lines: 133
  Changes:
  • Name: MetaStorage.__setitem__
    Type: function
    Changed lines: 1
    Total lines: 17

hub/api/tests/test_converters.py

Changes:

• Name: test_to_tensorflow
  Type: function
  Changed lines: 1
  Total lines: 10

hub/cli/auth.py

Changes:

• Name: reporting
  Type: function
  Changed lines: 1
  Total lines: 11

hub/schema/bbox.py

Changes:

• Name: BBox
  Type: class
  Changed lines: 1
  Total lines: 57

hub/schema/text.py

Changes:

• Name: Text
  Type: class
  Changed lines: 1
  Total lines: 49

hub/store/tests/test_dynamic_tensor.py

Changes:

• Name: test_dynamic_tensor_2
  Type: function
  Changed lines: 1
  Total lines: 15

hub/utils.py

Changes:

• Name: batchify
  Type: function
  Changed lines: 2
  Total lines: 9

[haiyangdeperci]

Fixed issues with Mac, all the tests pass. As for Windows, I think the tests have encountered a bug with dataset copying that is exclusive to windows. I'll investigate and I'll write more about it.

[haiyangdeperci]

It seems like there is an issue with dataset copying on Windows. This inline python code would print [0 1 2 3 4] on Linux/Mac and [0 0 0 0 0] on Windows:

python -c "import hub; from hub.cli.auth import login_fn; import numpy as np; login_fn('testingacc', os.getenv('ACTIVELOOP_HUB_PASSWORD')); ds = hub.Dataset('./dataset_copying', shape=(5,), schema={'num': 'uint8'}); ds['num'] = np.arange(5); ds2 = ds.copy('testingacc/dataset_copying'); print(ds2['num'].compute())"

[haiyangdeperci]

Since the only issue with this PR is actually the bug with hub on Windows (which is a separate issue) I propose that it is merged.

[verbose-void]

windows complex test fails?

[haiyangdeperci]

@McCrearyD this PR introduces Windows Complex Test :) as explained above, it fails due to the same reason as #702

[verbose-void]

ok