Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: multiple image loader function for extended folder structure in image classification #799

Merged

Conversation

sparkingdark
Copy link
Contributor

Some image classification dataset has the following data structure:

--root
        |__test
        |__train
        |__val

To deal with this kind of dataset I implemented a function which can return a hub transform object list in the format corresponding to test,train and val.

─    debo@debo-Lenovo-ideapad-130-15IKB    ~/Hub     feature/multiple_image_loader      
╰─ /usr/bin/python3 /home/debo/Hub/test.py
['test_set', 'training_set']
parsing image classification dataset: 100%|███████████████████████████████████████████████| 2/2 [00:00<00:00,  9.35it/s]
{'image': Image(shape=(None, None, None), dtype='uint8', max_shape=(500, 500, 3)), 'label': ClassLabel(shape=(), dtype='uint16', names=['cats', 'dogs'], num_classes=2)}
parsing image classification dataset: 100%|███████████████████████████████████████████████| 2/2 [00:00<00:00,  2.53it/s]
{'image': Image(shape=(None, None, None), dtype='uint8', max_shape=(768, 1050, 3)), 'label': ClassLabel(shape=(), dtype='uint16', names=['cats', 'dogs'], num_classes=2)}
[<hub.compute.transform.Transform object at 0x7f3b14c71b50>, <hub.compute.transform.Transform object at 0x7f3b14bf86a0>]

a sample output above.

To do:

  • [] add tests
  • [] Add exception handling and direct data upload feature to the hub.

[Work in Progress]

Need review and feedback @davidbuniat @McCrearyD

Copy link
Contributor

@verbose-void verbose-void left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! but could you please write some tests? just add a (very small custom) dummy_dataset in this folder: https://github.com/activeloopai/Hub/tree/master/hub/auto/tests/dummy_data/image_classification

you can use the other datasets in that folder as a reference. & you don't need to make a new test file, you can place your tests in here: https://github.com/activeloopai/Hub/blob/master/hub/auto/tests/test_image_classification.py

@sparkingdark
Copy link
Contributor Author

@McCrearyD tests added can you check it.

@verbose-void verbose-void self-assigned this May 5, 2021
@verbose-void verbose-void added the enhancement New feature or request label May 5, 2021
Copy link
Contributor

@verbose-void verbose-void left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks awesome! note: your dummy_data was missing a subdirectory "root" (to conform to the other tests) & the naming of the dummy_data subdirectory was changed to be folder_class_sample.

i will merge to master when the tests finish running.

@verbose-void verbose-void merged commit 3500ec6 into activeloopai:master May 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants