Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[datasets] Allow parallel processing of downloads #14

Closed
frgfm opened this issue Oct 5, 2019 · 0 comments
Closed

[datasets] Allow parallel processing of downloads #14

frgfm opened this issue Oct 5, 2019 · 0 comments
Assignees
@frgfm
Labels
module: datasets Related to datasets type: enhancement New feature or request
Milestone
0.1.0

Comments

@frgfm
Copy link
Member

frgfm commented Oct 5, 2019

🚀 Feature

Use parallel processing for dataset downloads when a list of URLs is involved.

Motivation

pyronear.datasets.OpenFire downloading process is taking several minutes while allowing multi-processing would shorten this by a lot.

Pitch

Add a multi-processing feature for URL downloads and let the users pick the number of workers. Potential libraries to check: Ray, python multiprocessing
A good implementation for URL downloading is already present in fastai (but it is a large dependency).
@frgfm frgfm added type: enhancement New feature or request module: datasets Related to datasets labels Oct 5, 2019
@frgfm frgfm added this to the 0.1.0 milestone Oct 5, 2019
@frgfm frgfm self-assigned this Oct 8, 2019
frgfm added a commit that referenced this issue Oct 8, 2019
@frgfm
feat: Added multi-thread URL download
ea02cf7 
See #14
frgfm added a commit that referenced this issue Oct 8, 2019
@frgfm
feat: Added multi-thread download to OpenFire
220d15a 
Closes #14
@frgfm frgfm mentioned this issue Oct 8, 2019
Merged
@frgfm frgfm closed this as completed in 932ef78 Oct 9, 2019
blenzi pushed a commit to blenzi/PyroNear that referenced this issue Oct 21, 2019
@frgfm
feat: Updated datasets.OpenFire (pyronear#15)
8083d3f 
* test: Moved temporary path to context

* test: Added tests for download_url

* feat: Added multi-thread URL download

See pyronear#14

* docs: Updated docstring

* refactor: Switched silent mode to optional

* refactor: Fixed get_fname

More tolerant base name resolution

* test: Added unittests for download_urls

* feat: Updated URL to cleaned OpenFire

See pyronear#6 

* feat: Added multi-thread download to OpenFire

Closes pyronear#14

* chore: Updated requirements.txt

* style: Removed whitespaces

* refactor: Removed unused import

* style: Improved exception handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frgfm frgfm

Labels
module: datasets Related to datasets type: enhancement New feature or request
Projects
None yet
Milestone
0.1.0
Development

No branches or pull requests
1 participant
@frgfm