Specify what should be pushed to different remotes

[nirtiac]

I need to be able to push versioned trained models to a production server. Both data and models will be created on one machine and versioned using DVC. However, I have sensitive information in my training server data that must never be pushed off the server.

As it stands, we are currently figuring out how to push models to a remote that can be tied back to their versioned data/code within DVC while never also pushing the data. This requires a solution outside of DVC. It would be great if DVC could provide the capability to specify file types that can or can't be pushed to a remote once committed.

[Suor]

Sounds like this is not about file types, but a need to specify what to push in general. What if you have files of same type ones with sensitive data and others not? You have this nice property the latter being models and the former being something else, which won't always be the case.

[nirtiac]

Yep, very true @Suor - that would be even better!

[shcheklein]

so, add a flag per output, something like --out-local to avoid pushing it to remote at all? it should be pretty straightforward. My only concern that number of different options for different types of outputs double every time. We should think about some general mechanism to specify options per output?

[jorgeorpinel]

It seems we hear this feature request often on the chat, and there's even a very similar SO question: How to use different remotes for different folders?
: "I want my data and models stored in separate Google Cloud buckets. The idea is that I want to be able to share the data with others without sharing the models."

The current solution is to setup the remotes, and then use dvc push -r {REMOTE} {data/model} individually for each data/model file.

As suggested in this issue though, it would be useful to setup a map of certain outputs/directories/file types to specific default remotes. For example we could add a --out option to dvc remote default (somehow recorded in .dvc/config) so DVC knows that one output file should be pushed to a specific remote by default (unless overwritten with dvc push -r, of course. The option could be called something else like --map and accept not only outputs but any file/dir path or even file type filter (e.g. something like **/*.csv).

[Suor]

Mapping could be done two ways:

• specify outputs, possibly patterns, in a remote config
• specify remote in an output definition in a stage file

[dmpetrov]

@nirtiac for data from different storages or with different access permissions, you can use new DVC functionality - dvc import. This works only in multiple-repository environment (not monorepo) but it potentially gives you an elegant solution of separating data remotes.

1. The dataset can be presented in a separate git-repo with its own remote. Repo: https://github.com/mycorp/rawdata. Remote like: s3://rawdata-bucket
2. Modeling code with your ML model in another repo like https://github.com/mycorp/predict. Remote like: s3://predict-bucket.
3. You just import dataset from your model repo dvc import https://github.com/mycorp/rawdata images.
4. Now, the model repo has data from different data remotes. dvc push from the model repo won't push the imported dataset into the model remote.
5. You can share only your model repo predict for the ML model consumers (prod environment for example) and access key for the model bucket s3://predict-bucket. It prevents consumers from using source dataset - s3://rawdata-bucket.

Technically, this solution looks more or less the same as the proposed solution with output types (@shcheklein ) or a working solution with specified remotes (@jorgeorpinel ) but it provides a higher-level abstraction (dataset repo, ML model repo) and should be a better choice for users.

[nirtiac]

This is great, thank you @dmpetrov !