Skip to content

Conversation

@bpoldrack
Copy link
Member

@bpoldrack bpoldrack commented Apr 1, 2022

Separating reading provider configs from downloading a key, since this
was a per-key operation unnecessarily slowing down HTTP downloads.

Magically, this also enables per-key progress bars (addressing datalad/datalad-ria#13 for the HTTP case). This seems to depend on the persistence of the Provider and/or Downloader instance across keys. Same call path does not work, when Providers are read anew from the configs on a per-key basis (as the implementation using download_url did).

Changelog

🐛 Bug Fixes

  • ORA special remote didn't show per-file progress bars when downloading over HTTP

💫 Enhancements and new features

  • Removed a per-file overhead cost for ORA when downloading over HTTP

@bpoldrack bpoldrack marked this pull request as ready for review April 1, 2022 13:13
Separating reading provider configs from downloading a key, since this
was a per-key operation unneccessarily slowing down HTTP downloads.

This also enables per-key progress bars when downloading over HTTP.
@qlty-cloud-legacy
Copy link

Code Climate has analyzed commit 2d748a7 and detected 2 issues on this pull request.

Here's the issue category breakdown:

Category Count
Security 2

View more on Code Climate.

@bpoldrack bpoldrack added semver-patch Increment the patch version when merged team-remotes (special) remote implementations (https://github.com/datalad/datalad/issues/6365) labels Apr 6, 2022
@codecov
Copy link

codecov bot commented Apr 6, 2022

Codecov Report

Merging #6609 (2d748a7) into master (97f648e) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    datalad/datalad#6609      +/-   ##
==========================================
+ Coverage   91.14%   91.15%   +0.01%     
==========================================
  Files         350      353       +3     
  Lines       44232    44552     +320     
==========================================
+ Hits        40316    40613     +297     
- Misses       3916     3939      +23     
Impacted Files Coverage Δ
datalad/distributed/ora_remote.py 80.72% <100.00%> (+0.66%) ⬆️
datalad/support/entrypoints.py 47.36% <0.00%> (-13.51%) ⬇️
datalad/interface/common_cfg.py 96.59% <0.00%> (-3.41%) ⬇️
datalad/cmd.py 92.97% <0.00%> (-1.12%) ⬇️
datalad/core/local/tests/test_status.py 97.79% <0.00%> (-0.68%) ⬇️
datalad/core/local/tests/test_save.py 97.95% <0.00%> (-0.17%) ⬇️
datalad/support/annexrepo.py 91.19% <0.00%> (-0.08%) ⬇️
datalad/support/tests/test_annexrepo.py 97.92% <0.00%> (-0.07%) ⬇️
datalad/coreapi.py 100.00% <0.00%> (ø)
datalad/support/tests/test_extensions.py 100.00% <0.00%> (ø)
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 97f648e...2d748a7. Read the comment docs.

@bpoldrack bpoldrack merged commit 789336f into datalad:master Apr 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver-patch Increment the patch version when merged team-remotes (special) remote implementations (https://github.com/datalad/datalad/issues/6365)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant