no possibility to specify credentials/profile in dvc import

[pauqilaz]

Hi,

For many use cases, when importing from s3, the s3 bucket from which data is imported requires credentials or a profile. Right now, it does not seem possible to specify these (even though it is possible for a s3 remote with dvc remote modify myremote profile myprofile).

I know that is it possible to work around that by using dvc run instead, but providing a way to specify a profile or credential would be a welcome feature in dvc import. It could look like:

dvc import s3://mybucket/myfile destination --profile myprofile

Thanks!

[efiop]

Hi @marie-ml !

You could go about it this way:

dvc remote add mys3 s3://bucket/path
dvc remote modify mys3 profile myprofile
dvc import remote://mys3/myfile destination

Thanks,
Ruslan

[pauqilaz]

Hi @efiop ,

Thanks for the quick answer! And is there a way to specify where the destination.dvc file should be saved to ?

Thanks!

[efiop]

@marie-ml There is no such option for dvc import itself right now 🙁 By default, it will simply place destination.dvc right beside where destination is located. However, you could workaround that by using a more flexible dvc run command like so:

dvc run -d remote://mys3/myfile -o destination -f path/to/file.dvc

[efiop]

dvc import now supports -f #1895 . Closing this issue, please feel free to reopen 🙂

[nils-werner]

You could go about it this way:

dvc remote add mys3 s3://bucket/path
dvc remote modify mys3 profile myprofile
dvc import remote://mys3/myfile destination

This doesn't seem to work for Git remotes:

dvc remote add mirror ssh://git@server.com/repo.git
dvc import remote://mirror/ mnist/mnist.gz

fails with

cmdline: git clone -v --no-single-branch --progress remote://mirror/ /var/folders/gf/gn2yxq511p77gzs2gsn4gp2w0000gn/T/tmpu87oxfvgdvc-clone

In my usecase I want to add Git credentials for my imported data during a CI run.

[efiop]

@nils-werner remote notation is not supported for git arguments in dvc import/list/get. Though that is a pretty cool idea for the future (because remote:// is already confusing in url context, but will be even more confusing if we also use it for git).