get/import/list/etc command should accept a remote as argument

[a0th]

The get uses the default remote defined in .dvc/config

If there isn't one defined, it will fail with
file error: No remote repository specified. Setup default repository with dvc config core.remote <name> or use: dvc pull -r <name>

But the get command doesn't allow the selection of a remote. Moreover, it might be the case that we want to choose another remote other than the default one.

[efiop]

Hi @a0th !

get/import assume that you've set some default remote already in the repo you are getting/importing from since it is a part of its state. Looks like you don't have that one setup. Was it an accident or you consciously don't want to setup default remote in that repository that you are downloading from? Could you elaborate on your scenario, please? 🙂

[efiop]

Ah, didn't notice the discord discussion. For the record: https://discordapp.com/channels/485586884165107732/485596304961962003/619026170369015818

TL;DR from the discord discussion: Implement the --remote option for dvc get to select other remote rather than the default.

This is in case that some files are pushd to remote alpha and other files pushd to remote bravo. I'm not sure if we should allow this but it seems like a good idea for flexibility.

[shcheklein]

Btw, there is a clear bug in this message: or use: dvc pull -r <name> - it has nothing to do with dvc get.

There is definitely a lot of value in being able to define a different remote per output, or even specify that we don't want to push certain outputs to the remote storage. The biggest use case here is related to security - I want to expose publicly only models but not raw data -> it's better to split them into different remotes, but I believe there are other case as well - different storages (due to price), etc.

@a0th could you please give a bit more context on your specific case? Why do you need multiple remotes in your case?

--remote is one of the solutions we can use to provide the interface for this, but long term we should think about something else (e.g. keep the info in the DVC-file per output). --remote is error-prone, cumbersome, etc, especially when you use it with dvc push/dvc pull.

[Suor]

I am against adding --remote for this. The use case of several remotes for different outputs should be handled in a more declarative way on the source repo part. It's either a stated remote in dvc-file for particular out or some declarative cache router config, which says what is pushed to where. This will simplify and make it less error prone for people using the source repo also - dvc push/pull will simply do the right thing possibly working with several remotes in parallel.

[Suor]

UPDATE: This issue applies to dvc.api.read()/open() as well as dvc get/import.

[shcheklein]

The last part of the #2711 (comment) by @alvaroabascar comment is related.

The same question to @alvaroabascar as I asked @a0th . It would be great to have a case/setup in mind when it's indeed necessary to provide a remote name like this. Mostly the question is why would we keep remote undefined in the external repo in the first place.

[efiop]

I can totally see a usecase for this in a form of a faster remote mirror :) For example, you want to dvc get from a public dvc repo, but your company has a local remote with that data already, so dvc get/import from it would be much faster than from a public repo.

[Suor]

@efiop what you are talking about is even more different. How do you configure your own remote on someone else's repo to refer to it?