Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce dvc remote #596

Closed
dmpetrov opened this issue Mar 30, 2018 · 0 comments
Closed

Introduce dvc remote #596

dmpetrov opened this issue Mar 30, 2018 · 0 comments
Assignees
Labels
enhancement Enhances DVC
Milestone

Comments

@dmpetrov
Copy link
Member

dmpetrov commented Mar 30, 2018

Currently, DVC supports only a single cloud setting in DVC config file. We need to generalize the cloud settings by introducing a concept of DVC remote the same way that Git has.

A scenario of an ML project:

  1. I usually use my laptop to work on the project.
  2. The major experiments data have to be synced to a shared cloud folder in S3. However, I don't want to sync every experiment into the folder (i might have 10 new models per day).
  3. Sometimes I run models on my desktop because of 11GB GPU.
  4. In some cases, I use a temporary EC2 machine with 6 x 11Gb GPU.
  5. Also, I might decide to pull successful experiments (with data) from my colleagues' laptop which are not "successful" ehough to sync into our common cloud folder (see 2).

At the end of the day I'll have 4 remotes in additional to my local repository:

$ dvc remote -v
common     s3://dvc-share/segment_cnn/dvc_cache
mypc       rsync://john@johnpc/home/john/src/segment_cnn
robert     ssh://john@robert-laptop/home/robert/projects/code/segment_cnn
ec2_6gpu   ssh://ubuntu@ec2-34-205-127-69.compute-1.amazonaws.com/home/ubuntu/segment_cnn

Note, different protocols were in used: s3, rsync, ssh (see #594). Protocols gcp, ftp(s), http(s) should be supported eventually.

Open questions:

  • how to keep authentification information like passwords and ssh-keys?
  • should we keep "global" config settings? Probably we should for backward compatibility.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhances DVC
Projects
None yet
Development

No branches or pull requests

2 participants