Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[datasets] Add a classification dataset #6

Closed
frgfm opened this issue Sep 28, 2019 · 0 comments
Closed

[datasets] Add a classification dataset #6

frgfm opened this issue Sep 28, 2019 · 0 comments
Assignees
@MateoLostanlen @frgfm @Akilditu
Labels
help wanted Extra attention is needed module: datasets Related to datasets
Milestone
0.1.0

Comments

@frgfm
Copy link
Member

frgfm commented Sep 28, 2019

🚀 Feature

Add a classification dataset to the datasetsmodule

Motivation

Since the project needs to provide models for wildfire detection and allow external contributors to train their own, making datasets available is a priority.

Pitch

  • similarly to torchvision datasets module, each dataset needs to be a class inheriting from torchvision.datasets.vision.VisionDataset or at least torch.utils.data.Dataset.
  • she download URL needs to be accessible by the class.
@frgfm frgfm added help wanted Extra attention is needed module: datasets Related to datasets labels Sep 28, 2019
@frgfm frgfm added this to the 0.1.0 milestone Sep 28, 2019
frgfm referenced this issue Oct 1, 2019
@frgfm
feat: Added OpenFire dataset
9236354 
Inherits from torchvision.datasets.vision.VisionDataset
frgfm added a commit that referenced this issue Oct 2, 2019
@frgfm
refactor: Refactored OpenFire for consistency
81d66e9 
Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes #6
@frgfm frgfm closed this as completed in cc09e72 Oct 2, 2019
frgfm added a commit that referenced this issue Oct 4, 2019
@frgfm
feat: Added datasets module (#11)
3156905 
* feat: Added datasets module

Added URL download function in pyronear.datasets.utils.

* test: Added datasets.utils unittests

* chore: Added __init__ to datasets

* docs: Added datasets docs

* docs: Updated doc theme

* docs: Updated color theme

* chore: Fixed datasets __init__

* refactor: Refactored download_url

Reduced dependencies and sped up execution

* feat: Added OpenFire dataset

Inherits from torchvision.datasets.vision.VisionDataset

* style: Updated docstrings

* docs: Added OpenFire to datasets

* chore: Updated requirements.txt

* refactor: Refactored OpenFire for consistency

Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes #6

* test: Updated datasets.utils test

Sped up basename resolution

* test: Added unittests for datasets.OpenFire

* test: Fixed OpenFire unittests

As per #10, a few files can be unavailable for download temporarily.

* style: Improved warning for OpenFire download

Improved warning message that are thrown by OpenFire.download() to display the error that was caught.

* test: Refactored datasets unittests

* fix: Fixed datasets.OpenFire
@frgfm frgfm mentioned this issue Oct 8, 2019
Merged
JoaoFdC pushed a commit to JoaoFdC/PyroNear that referenced this issue Oct 9, 2019
@frgfm @JoaoFdC
feat: Added datasets module (pyronear#11)
6e01a44 
* feat: Added datasets module

Added URL download function in pyronear.datasets.utils.

* test: Added datasets.utils unittests

* chore: Added __init__ to datasets

* docs: Added datasets docs

* docs: Updated doc theme

* docs: Updated color theme

* chore: Fixed datasets __init__

* refactor: Refactored download_url

Reduced dependencies and sped up execution

* feat: Added OpenFire dataset

Inherits from torchvision.datasets.vision.VisionDataset

* style: Updated docstrings

* docs: Added OpenFire to datasets

* chore: Updated requirements.txt

* refactor: Refactored OpenFire for consistency

Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes pyronear#6

* test: Updated datasets.utils test

Sped up basename resolution

* test: Added unittests for datasets.OpenFire

* test: Fixed OpenFire unittests

As per pyronear#10, a few files can be unavailable for download temporarily.

* style: Improved warning for OpenFire download

Improved warning message that are thrown by OpenFire.download() to display the error that was caught.

* test: Refactored datasets unittests

* fix: Fixed datasets.OpenFire
frgfm added a commit that referenced this issue Oct 9, 2019
@frgfm
feat: Updated datasets.OpenFire (#15)
932ef78 
* test: Moved temporary path to context

* test: Added tests for download_url

* feat: Added multi-thread URL download

See #14

* docs: Updated docstring

* refactor: Switched silent mode to optional

* refactor: Fixed get_fname

More tolerant base name resolution

* test: Added unittests for download_urls

* feat: Updated URL to cleaned OpenFire

See #6 

* feat: Added multi-thread download to OpenFire

Closes #14

* chore: Updated requirements.txt

* style: Removed whitespaces

* refactor: Removed unused import

* style: Improved exception handling
@frgfm frgfm mentioned this issue Oct 12, 2019
Merged
frgfm added a commit that referenced this issue Oct 13, 2019
@frgfm
feat: Updated OpenFire URL (#18)
1f42aeb 
* feat: Updated OpenFire URL

See #6 

* docs: Updated readme

* fix: Fixed large image loading with pillow

* feat: Added option to create validation set

A valid_pct can now be passed when download=True to create the validation set by taking a percentage of the training set.

* test: Added unittests to OpenFire

Added test for the valid_pct argument of OpenFire

* feat: Improved random split of OpenFire sets

Used local seed setting of the random module to avoid disturbing global execution.
JoaoFdC pushed a commit to JoaoFdC/PyroNear that referenced this issue Oct 15, 2019
@frgfm @JoaoFdC
feat: Added datasets module (pyronear#11)
ecf2489 
* feat: Added datasets module

Added URL download function in pyronear.datasets.utils.

* test: Added datasets.utils unittests

* chore: Added __init__ to datasets

* docs: Added datasets docs

* docs: Updated doc theme

* docs: Updated color theme

* chore: Fixed datasets __init__

* refactor: Refactored download_url

Reduced dependencies and sped up execution

* feat: Added OpenFire dataset

Inherits from torchvision.datasets.vision.VisionDataset

* style: Updated docstrings

* docs: Added OpenFire to datasets

* chore: Updated requirements.txt

* refactor: Refactored OpenFire for consistency

Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes pyronear#6

* test: Updated datasets.utils test

Sped up basename resolution

* test: Added unittests for datasets.OpenFire

* test: Fixed OpenFire unittests

As per pyronear#10, a few files can be unavailable for download temporarily.

* style: Improved warning for OpenFire download

Improved warning message that are thrown by OpenFire.download() to display the error that was caught.

* test: Refactored datasets unittests

* fix: Fixed datasets.OpenFire
JoaoFdC pushed a commit to JoaoFdC/PyroNear that referenced this issue Oct 19, 2019
@frgfm @JoaoFdC
feat: Added datasets module (pyronear#11)
28f0354 
* feat: Added datasets module

Added URL download function in pyronear.datasets.utils.

* test: Added datasets.utils unittests

* chore: Added __init__ to datasets

* docs: Added datasets docs

* docs: Updated doc theme

* docs: Updated color theme

* chore: Fixed datasets __init__

* refactor: Refactored download_url

Reduced dependencies and sped up execution

* feat: Added OpenFire dataset

Inherits from torchvision.datasets.vision.VisionDataset

* style: Updated docstrings

* docs: Added OpenFire to datasets

* chore: Updated requirements.txt

* refactor: Refactored OpenFire for consistency

Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes pyronear#6

* test: Updated datasets.utils test

Sped up basename resolution

* test: Added unittests for datasets.OpenFire

* test: Fixed OpenFire unittests

As per pyronear#10, a few files can be unavailable for download temporarily.

* style: Improved warning for OpenFire download

Improved warning message that are thrown by OpenFire.download() to display the error that was caught.

* test: Refactored datasets unittests

* fix: Fixed datasets.OpenFire
JoaoFdC pushed a commit to JoaoFdC/PyroNear that referenced this issue Oct 19, 2019
@frgfm @JoaoFdC
feat: Added datasets module (pyronear#11)
17a5b6f 
* feat: Added datasets module

Added URL download function in pyronear.datasets.utils.

* test: Added datasets.utils unittests

* chore: Added __init__ to datasets

* docs: Added datasets docs

* docs: Updated doc theme

* docs: Updated color theme

* chore: Fixed datasets __init__

* refactor: Refactored download_url

Reduced dependencies and sped up execution

* feat: Added OpenFire dataset

Inherits from torchvision.datasets.vision.VisionDataset

* style: Updated docstrings

* docs: Added OpenFire to datasets

* chore: Updated requirements.txt

* refactor: Refactored OpenFire for consistency

Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes pyronear#6

* test: Updated datasets.utils test

Sped up basename resolution

* test: Added unittests for datasets.OpenFire

* test: Fixed OpenFire unittests

As per pyronear#10, a few files can be unavailable for download temporarily.

* style: Improved warning for OpenFire download

Improved warning message that are thrown by OpenFire.download() to display the error that was caught.

* test: Refactored datasets unittests

* fix: Fixed datasets.OpenFire
blenzi pushed a commit to blenzi/PyroNear that referenced this issue Oct 21, 2019
@frgfm
feat: Added datasets module (pyronear#11)
25480c5 
* feat: Added datasets module

Added URL download function in pyronear.datasets.utils.

* test: Added datasets.utils unittests

* chore: Added __init__ to datasets

* docs: Added datasets docs

* docs: Updated doc theme

* docs: Updated color theme

* chore: Fixed datasets __init__

* refactor: Refactored download_url

Reduced dependencies and sped up execution

* feat: Added OpenFire dataset

Inherits from torchvision.datasets.vision.VisionDataset

* style: Updated docstrings

* docs: Added OpenFire to datasets

* chore: Updated requirements.txt

* refactor: Refactored OpenFire for consistency

Removed torch.Tensor casting of target since the collate function of DataLoader will cast it anyway. Added a class_to_idx attribute. Closes pyronear#6

* test: Updated datasets.utils test

Sped up basename resolution

* test: Added unittests for datasets.OpenFire

* test: Fixed OpenFire unittests

As per pyronear#10, a few files can be unavailable for download temporarily.

* style: Improved warning for OpenFire download

Improved warning message that are thrown by OpenFire.download() to display the error that was caught.

* test: Refactored datasets unittests

* fix: Fixed datasets.OpenFire
blenzi pushed a commit to blenzi/PyroNear that referenced this issue Oct 21, 2019
@frgfm
feat: Updated datasets.OpenFire (pyronear#15)
8083d3f 
* test: Moved temporary path to context

* test: Added tests for download_url

* feat: Added multi-thread URL download

See pyronear#14

* docs: Updated docstring

* refactor: Switched silent mode to optional

* refactor: Fixed get_fname

More tolerant base name resolution

* test: Added unittests for download_urls

* feat: Updated URL to cleaned OpenFire

See pyronear#6 

* feat: Added multi-thread download to OpenFire

Closes pyronear#14

* chore: Updated requirements.txt

* style: Removed whitespaces

* refactor: Removed unused import

* style: Improved exception handling
blenzi pushed a commit to blenzi/PyroNear that referenced this issue Oct 21, 2019
@frgfm
feat: Updated OpenFire URL (pyronear#18)
df2133e 
* feat: Updated OpenFire URL

See pyronear#6 

* docs: Updated readme

* fix: Fixed large image loading with pillow

* feat: Added option to create validation set

A valid_pct can now be passed when download=True to create the validation set by taking a percentage of the training set.

* test: Added unittests to OpenFire

Added test for the valid_pct argument of OpenFire

* feat: Improved random split of OpenFire sets

Used local seed setting of the random module to avoid disturbing global execution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MateoLostanlen MateoLostanlen

@frgfm frgfm

@Akilditu Akilditu

Labels
help wanted Extra attention is needed module: datasets Related to datasets
Projects
None yet
Milestone
0.1.0
Development

No branches or pull requests
3 participants
@MateoLostanlen @frgfm @Akilditu