Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 downloader #124

Merged
merged 75 commits into from
Apr 25, 2023
Merged

Add S3 downloader #124

merged 75 commits into from
Apr 25, 2023

Conversation

TAlonglong
Copy link
Collaborator

@TAlonglong TAlonglong commented Aug 17, 2022

Add module/daemon for s3 download

@codecov
Copy link

codecov bot commented Aug 17, 2022

Codecov Report

Merging #124 (ff068d8) into main (4a4c8bc) will increase coverage by 1.32%.
The diff coverage is 99.33%.

@@            Coverage Diff             @@
##             main     #124      +/-   ##
==========================================
+ Coverage   88.61%   89.93%   +1.32%     
==========================================
  Files          22       24       +2     
  Lines        4425     5066     +641     
==========================================
+ Hits         3921     4556     +635     
- Misses        504      510       +6     
Flag Coverage Δ
unittests 89.93% <99.33%> (+1.32%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
trollmoves/s3downloader.py 98.17% <98.17%> (ø)
trollmoves/tests/conftest.py 100.00% <100.00%> (ø)
trollmoves/tests/test_s3downloader.py 100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@pnuu pnuu changed the title Issue123 s3downloader Add S3 downloader Aug 22, 2022
@TAlonglong
Copy link
Collaborator Author

Tests are failing, but this time something else.
I have a feeling I have seen this before, but I don't know how to handle this.

@pnuu
Copy link
Member

pnuu commented Apr 14, 2023

If I run the failing functional test locally it passes. If I run all the functional tests, they pass. If I run all the tests, I get failures in

  • trollmoves/tests/test_client.py::test_make_uris_local_destination_with_ftp
  • trollmoves/tests/test_client.py::test_make_uris_local_destination_with_ftp_and_login
  • trollmoves/tests/functional/test_move_it_server_client.py::test_simple_transfer

My first guess is some fixture with autouse=True wreaking havoc. I'll have a look.

@pnuu
Copy link
Member

pnuu commented Apr 14, 2023

Umm, those two failing tests in test_client.py fail locally also in main.

@pnuu
Copy link
Member

pnuu commented Apr 14, 2023

Ah, I guess the test_client.py stuff fail because of my VPN. So onward to investigate the failing functional test.

@pnuu
Copy link
Member

pnuu commented Apr 14, 2023

The failure happens here. I have zero idea what is going on and what these decorators do. @mraspaud ?

@TAlonglong
Copy link
Collaborator Author

yay

@TAlonglong
Copy link
Collaborator Author

I think I also need to review how credentials are used here. Maybe I get time later today to have a look

README.md Outdated Show resolved Hide resolved
bin/s3downloader.py Outdated Show resolved Hide resolved
bin/s3downloader.py Outdated Show resolved Hide resolved
Copy link
Contributor

@adybbroe adybbroe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lot of work, nice! I think this looks good, but have not gone in very deep detail. Added a few small comments

@TAlonglong
Copy link
Collaborator Author

The tests failed again, but I think its OK.

Do want to have another round of review?

@pnuu
Copy link
Member

pnuu commented Apr 25, 2023

While testing to see what's happening with the functional test failure, I see another one locally. Sorting this will need help from @mraspaud

Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pnuu pnuu merged commit 09a6140 into pytroll:main Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

s3downloader: download data from an s3 (object store) to localhost
5 participants