Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-33637: Use upload_file and add progress reporting for S3 #6

Merged
merged 2 commits into from Feb 11, 2022

Conversation

timj
Copy link
Member

@timj timj commented Feb 11, 2022

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

This has the advantage that behind the scenes it can be
parallelized and callbacks are supported.
Only enabled for DEBUG logging.
@codecov
Copy link

codecov bot commented Feb 11, 2022

Codecov Report

Merging #6 (c6b4272) into main (b6b22d6) will increase coverage by 0.11%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
+ Coverage   89.70%   89.81%   +0.11%     
==========================================
  Files          21       21              
  Lines        2302     2328      +26     
  Branches      329      330       +1     
==========================================
+ Hits         2065     2091      +26     
  Misses        162      162              
  Partials       75       75              
Impacted Files Coverage Δ
python/lsst/resources/s3.py 89.75% <100.00%> (+1.09%) ⬆️
tests/test_s3.py 81.81% <100.00%> (+4.04%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6b22d6...c6b4272. Read the comment docs.

Copy link
Member

@TallJimbo TallJimbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

Makes me wonder if we should the progress stuff here and in daf_butler up into utils so they could both be more broadly used. I don't see a lot of logic duplication, but it'd be nice to have a consistent interface for progress reporting with different displays (e.g. bars vs. logs). But I think that's another ticket, and only if you're enthusiastic; I don't think I can afford to take it on any time soon).

@timj
Copy link
Member Author

timj commented Feb 11, 2022

Yes, we never really had a reason to move your progress bar code out of butler but this does seem like it might well be a reason to do that.

@timj timj merged commit e25dd04 into main Feb 11, 2022
@timj timj deleted the tickets/DM-33637 branch February 11, 2022 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants