final from directory function for categorical image data

[sparkingdark]

🚀 🚀 Pull Request

All Submissions:

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• yes final from directory function for categorical image data #459 is linked to an issue Template Schema Generator For Categorical Image Data #293 followed by schema generator #301 and new schema generator branch #370

New Feature Submissions:

• Does your submission pass all tests?
• Does your submission not lower the coverage-rate?
• Have you lint your code locally prior to submission; including black style?
• Have you describe your new feature in the docs?

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?

• Have you written new tests for your core changes, as applicable?

• Have you successfully ran tests with your changes locally?

• Have you keep the coverage-rate constant?

• Have you keep the coverage-rate up?

• Are the docs up to date?

• change labels

• added a static method from_directory

please review @davidbuniat @AbhinavTuli @mynameisvinn

[github-actions]

Locust summary

Git references

Initial: ace631f
Terminal: f23cd0a

---

hub/api/tests/test_dataset.py

Changes:

• Name: test_dataset_from_directory
  Type: function
  Changed lines: 49
  Total lines: 49
  Changes:
  • Name: test_dataset_from_directory.create_image
    Type: function
    Changed lines: 8
    Total lines: 8
  • Name: test_dataset_from_directory.data_in_dir
    Type: function
    Changed lines: 6
    Total lines: 6
  • Name: test_dataset_from_directory.root_dir_image
    Type: function
    Changed lines: 9
    Total lines: 9
  • Name: test_dataset_from_directory.del_data
    Type: function
    Changed lines: 5
    Total lines: 5

hub/api/dataset.py

Changes:

• Name: Dataset.from_directory
  Type: function
  Changed lines: 147
  Total lines: 147
  Changes:
  • Name: Dataset.from_directory.get_max_shape
    Type: function
    Changed lines: 48
    Total lines: 48
  • Name: Dataset.from_directory.make_schema
    Type: function
    Changed lines: 18
    Total lines: 18
  • Name: Dataset.from_directory.upload_data
    Type: function
    Changed lines: 19
    Total lines: 19
• Name: Dataset
  Type: class
  Changed lines: 148
  Total lines: 1192
  Changes:
  • Name: Dataset.from_directory
    Type: function
    Changed lines: 147
    Total lines: 147
    Changes:

[mynameisvinn]

This is not a helpful message.

[mynameisvinn]

This doc string needs to be improved.

[mynameisvinn]

What about tiff? Or bmp? There are many image file types.

[sparkingdark]

@mynameisvinn now check that i correct it with autopep8 for proper formatting.

[mynameisvinn]

I tried this:

from hub import Dataset

mnist = Dataset("activeloop/mnist")
mnist.from_directory(path_to_dir="images/")

and got the following exception:

>>> check your data for fix

Since path_to_dir is the only required argument, I don't understand why it threw this exception.

A few suggestions:

• Please make exceptions more descriptive. "Check your data for fix" does not help me understand the root problem.
• Don't throw generic exceptions. You can subclass existing exceptions.

[sparkingdark]

You doing it wrong this is for uploading datasets to hub not to load a preloaded dataset for that hub.load is enough.

…

On Wed, Feb 3, 2021, 19:40 v ***@***.***> wrote: I tried this: from hub import Dataset mnist = Dataset("activeloop/mnist")mnist.from_directory(path_to_dir="images/") and got the following exception: >>> check your data for fix Since path_to_dir is the only required argument, I don't understand why it threw this exception. A few suggestions: - Please make exceptions more description. "Check your data for fix" means nothing. - Don't throw generic exceptions. You can subclass existing exceptions <https://github.com/sparkingdark/Hub/blob/final_from_directory/hub/exceptions.py> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL7XHCB4S73FPW2II32PIW3S5FKLVANCNFSM4WE7FYIA> .

[sparkingdark]

@mynameisvinn any update?

[mikayelh]

Hey @sparkingdark - @mynameisvinn was out for the Chinese new year - sorry for that. He'll review it shortly :)

[mynameisvinn]

Looks good, merging!