Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload archive to S3 bucket #17

Closed
wants to merge 0 commits into from

Conversation

22PoojaGaur
Copy link
Member

@22PoojaGaur 22PoojaGaur commented Mar 26, 2020

The archives were right now stored in local system. The ci tests take a lot of time
to build intermine_boot. Storing the archives online can help the process get
faster.

Approach:
Created a function for uploading archives to amazon aws in an extendable way to include other targets in future.

@heralden
Copy link
Member

This is a great start! We still need to be able to fetch the archives from the S3 bucket however.
Right now, we only need to focus on the CI usecase, so we need a way to version the archives we upload. I haven't looked into AWS S3 buckets yet, but I believe they support including metadata with the file you upload. So you can include a version value with the archives: If a Git URL or branch was specified for InterMine, include the repo URL, branch and commit hash (you can concatenate these into one string, or have separate key-value pairs for each); otherwise include the version number.

When an existing archive has the properties the user asked for when invoking intermine_boot, it should fetch it and use it. Otherwise, it can build its own and upload that.

We should probably also delete older archives. When using --im-repo and/or --im-branch, the user probably wants the latest one, so in those instances we can delete older archives from the S3 bucket if they're no longer current (ie. the ones with the same repo and branch, but different commit hash).

@22PoojaGaur
Copy link
Member Author

Looking into it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants