Skip to content
This repository has been archived by the owner on Jan 29, 2024. It is now read-only.

Implement bbs_database download for biorxiv and medrxiv #519

Merged
merged 21 commits into from
Dec 7, 2021

Conversation

jankrepl
Copy link
Contributor

@jankrepl jankrepl commented Nov 30, 2021

Fixes #512

Description

Implements the download logic for biorxiv and medrxiv.

How to test?

Note that the entrypoints will ask you to enter the credentials!

To run the dry run

bbs_database download medrxiv -n 2021-10 output_folder

To run an actual download

bbs_database download biorxiv 2021-10 output_folder

Checklist

  • This PR refers to an issue from the issue tracker.
    (if it is not the case, please create an issue first).
  • Unit tests added.
    (if needed)
  • Documentation and whatsnew.rst updated.
    (if needed)
  • setup.py and requirements.txt updated with new dependencies.
    (if needed)
  • Type annotations added.
    (if a function is added or modified)
  • All CI tests pass.

@jankrepl jankrepl marked this pull request as ready for review December 1, 2021 14:31
@Stannislav
Copy link
Contributor

When I try the dry-run example it sill asks me for my AWS credentials and when I enter random characters I get the (expected) error:

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.

Is it possibly to do a dry-run without credentials or not at all?

@jankrepl
Copy link
Contributor Author

jankrepl commented Dec 6, 2021

When I try the dry-run example it sill asks me for my AWS credentials and when I enter random characters I get the (expected) error:

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.

Is it possibly to do a dry-run without credentials or not at all?

Unfortunately, you have to have the credentials even for the dry run. Why? Well, to do the dry run one needs to send a request to the AWS server and that requires authentification.

setup.py Show resolved Hide resolved
src/bluesearch/database/download.py Outdated Show resolved Hide resolved
src/bluesearch/database/download.py Show resolved Hide resolved
src/bluesearch/database/download.py Outdated Show resolved Hide resolved
src/bluesearch/database/download.py Outdated Show resolved Hide resolved
@jankrepl jankrepl merged commit abbd2ca into master Dec 7, 2021
@jankrepl jankrepl deleted the download_bio_med branch December 7, 2021 10:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement bbs_database download --source=biorxiv and --source=medrxiv
4 participants