Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: urlretrieve is returning HTTPS 400 error #229

Closed
astrochun opened this issue Jun 30, 2021 · 0 comments · Fixed by #230
Closed

Bug: urlretrieve is returning HTTPS 400 error #229

astrochun opened this issue Jun 30, 2021 · 0 comments · Fixed by #230
Assignees
Labels
api Use of a third-party API bug Something isn't working curation Pertains to aspects of curation, including workflow management p1 Issues affecting production, multiple users
Milestone

Comments

@astrochun
Copy link
Contributor

astrochun commented Jun 30, 2021

Describe the bug

@jonathannoah identified a weird issue where the files could not be retrieved. I attempted to retrieve it via curl and was successful. Was able to reproduce the error locally with get_data. Ruled out it being a token issue. Today Figshare deployed changes to the API; however, we don't know what changed. This seems to be a urlretrieve issue completely. I think the urllib has not worked well with the Figshare API so I'm thinking we should switch back to requests with a chunking approach.
This is only a problem with under review deposits, not for public datasets/files.

Reproduction Steps

Retrieval via get_data of any file will suffice.

Outputted Messages

13:22:26 -     INFO: Writing: ****/1.ToDo/Chun_Ly_7083742/v2/METADATA/file_list_original_7083742.json
13:22:26 -     INFO: Writing: ****/1.ToDo/Chun_Ly_7083742/v2/METADATA/file_list_original_7083742.csv
13:22:26 -     INFO: Retrieving 1 of 1 : wccfl_logo.jpeg (99241)
13:22:26 -     INFO: URL: https://ndownloader.figsh.com/files/829622700
13:22:26 -     INFO: Retrieval attempt #1
13:22:28 -  WARNING: Caught an HTTPError: HTTP Error 400: Bad Request
13:22:28 -     INFO: URL might be public: https://ndownloader.figsh.com/files/829622700
13:22:28 -     INFO: Attempting retrieval without token
13:22:29 -  WARNING: Caught an HTTPError: HTTP Error 403: Forbidden
13:22:29 -  WARNING: Failed to retrieve: ****/1.ToDo/Chun_Ly_7083742/v2/ORIGINAL_DATA/wccfl_logo.jpeg
13:22:29 -     INFO: Retrieval attempt #2
13:22:30 -  WARNING: Caught an HTTPError: HTTP Error 400: Bad Request
13:22:30 -     INFO: URL might be public: https://ndownloader.figsh.com/files/829622700
13:22:30 -     INFO: Attempting retrieval without token
13:22:31 -  WARNING: Caught an HTTPError: HTTP Error 403: Forbidden
13:22:31 -  WARNING: Failed to retrieve: ****/1.ToDo/Chun_Ly_7083742/v2/ORIGINAL_DATA/wccfl_logo.jpeg
13:22:31 -     INFO: Retrieval attempt #3
13:22:32 -  WARNING: Caught an HTTPError: HTTP Error 400: Bad Request
13:22:32 -     INFO: URL might be public: https://ndownloader.figsh.com/files/829622700
13:22:32 -     INFO: Attempting retrieval without token
13:22:33 -  WARNING: Caught an HTTPError: HTTP Error 403: Forbidden
13:22:33 -  WARNING: Failed to retrieve: ****/1.ToDo/Chun_Ly_7083742/v2/ORIGINAL_DATA/wccfl_logo.jpeg
13:22:33 -  WARNING: File retrieval unsuccessful! Aborted after 3 tries

Version information

  • LD_Cool-P version: 1.1.3

Expected behavior

Screenshots

Additional context

Implemented in: TBD

@astrochun astrochun added api Use of a third-party API bug Something isn't working curation Pertains to aspects of curation, including workflow management p1 Issues affecting production, multiple users labels Jun 30, 2021
@astrochun astrochun added this to the v1.2.0 milestone Jun 30, 2021
@astrochun astrochun self-assigned this Jun 30, 2021
@astrochun astrochun added this to To do in ReDATA Data Curation Board via automation Jun 30, 2021
ReDATA Data Curation Board automation moved this from To do to Done Jul 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api Use of a third-party API bug Something isn't working curation Pertains to aspects of curation, including workflow management p1 Issues affecting production, multiple users
Projects
Development

Successfully merging a pull request may close this issue.

1 participant