You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, DVC supports only a single cloud setting in DVC config file. We need to generalize the cloud settings by introducing a concept of DVC remote the same way that Git has.
A scenario of an ML project:
I usually use my laptop to work on the project.
The major experiments data have to be synced to a shared cloud folder in S3. However, I don't want to sync every experiment into the folder (i might have 10 new models per day).
Sometimes I run models on my desktop because of 11GB GPU.
In some cases, I use a temporary EC2 machine with 6 x 11Gb GPU.
Also, I might decide to pull successful experiments (with data) from my colleagues' laptop which are not "successful" ehough to sync into our common cloud folder (see 2).
At the end of the day I'll have 4 remotes in additional to my local repository:
$ dvc remote -v
common s3://dvc-share/segment_cnn/dvc_cache
mypc rsync://john@johnpc/home/john/src/segment_cnn
robert ssh://john@robert-laptop/home/robert/projects/code/segment_cnn
ec2_6gpu ssh://ubuntu@ec2-34-205-127-69.compute-1.amazonaws.com/home/ubuntu/segment_cnn
Note, different protocols were in used: s3, rsync, ssh (see #594). Protocols gcp, ftp(s), http(s) should be supported eventually.
Open questions:
how to keep authentification information like passwords and ssh-keys?
should we keep "global" config settings? Probably we should for backward compatibility.
The text was updated successfully, but these errors were encountered:
Currently, DVC supports only a single cloud setting in DVC config file. We need to generalize the cloud settings by introducing a concept of DVC remote the same way that Git has.
A scenario of an ML project:
At the end of the day I'll have 4 remotes in additional to my local repository:
Note, different protocols were in used:
s3
,rsync
,ssh
(see #594). Protocolsgcp
,ftp(s)
,http(s)
should be supported eventually.Open questions:
The text was updated successfully, but these errors were encountered: