Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 support for data storage #3319

Open
pinkmanbox opened this issue Feb 13, 2020 · 2 comments
Open

S3 support for data storage #3319

pinkmanbox opened this issue Feb 13, 2020 · 2 comments

Comments

@pinkmanbox
Copy link

@pinkmanbox pinkmanbox commented Feb 13, 2020

In a project I am working on, I want data in an S3 bucket to be available to pull locally, while maintaining the folder structure and the data as it is in the bucket. This is because I need to perform AWS operations on that data as well. But since I also need that data for local operations (and any changes on them), dvc control would be beneficial.

Essentially, S3 being the GitHub for the data, while dvc being the git for the data.

I apologize if this feature already exists, and if so, could someone please direct me on how to achieve this ?

Thanks.

@triage-new-issues triage-new-issues bot added the triage label Feb 13, 2020
@pared

This comment has been minimized.

Copy link
Member

@pared pared commented Feb 13, 2020

@efiop efiop added the question label Feb 13, 2020
@triage-new-issues triage-new-issues bot removed the triage label Feb 13, 2020
@efiop

This comment has been minimized.

Copy link
Member

@efiop efiop commented Feb 13, 2020

@pinkmanbox Sounds like you might be looking for dvc get/import commands. Please take a look at https://dvc.org/doc/use-cases/data-registries . TLDR; is that dvc remote is not a dvc repo, it requires git to store metadata, so you could address your files by name through your git repo and dvc will automatically download them from your s3 bucket. Let us know what you think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.