`exp run`: copy (certain) git-ignored files to tmp folder on `--run-all`

[jdonzallaz]

UPDATE: Jump to #5800 (comment)

When running experiments with --run-all (with some experiments in the --queue), the repo is copied to the .dvc/tmp/ for each experiment to run. The problem is that the git-ignored files are not copied to this folder, and the commands that depend on those files fail.
The files that are git-ignored but are defined as stage dependencies should be copied.
Use-case is the following: Some config files containing sensible data (API keys, password) are not pushed to the git repo but are still needed by certain commands.

See discord discussion.
CC: @shcheklein @pmrowla

[jorgeorpinel]

Hey @jdonzallaz per #5029 it's possible to include untracked files in tmp exp dirs by staging them before queueing the experiment. So I think you could try something like

$ git add --force secret.txt
$ dvc exp run --queue
$ git remove secret.txt

⚠️ This would be a workaround! And it may have unwanted consequences like the secrets ending up in Git history.

Let's see what Engineering thinks of this one 🙂

[pmrowla]

I don't think that workaround is viable here given that the request here is specifically regarding:

Some config files containing sensible data (API keys, password)

And if an experiment is run this way, those API keys and password will end up in git history

[jorgeorpinel]

Do we need a separate issue to specifically prevent this workaround? B/c if I thought about it someone else may figure it out and it can lead to some hard-to-predict security risks.

[pmrowla]

There's not really anything we can do to prevent people from doing this, users can add whatever they want into git.

I think anyone using git add --force on a file containing secret credentials should already be aware that this will add that file into git, since that's what git add --force does by definition?

[jorgeorpinel]

that's what git add --force does by definition?

Not without git commit. In this scenario people would stage the secrets specifically because we copy staged (not committed) files into tmp exp dirs.

I keep coming back to the idea of not using Git as a proxy UI to DVC's behavior, but that's discussed is in #5801.

BTW, from #5801 (comment) @pmrowla:

in theory, stuff like (properly gitignored) venvs also needs to be copied over

Feels like too much. Is it common to have something like venv/ as a stage dependency? Or are we expanding the scope here (looking ahead)? Thanks

[pmrowla]

Not without git commit. In this scenario people would stage the secrets specifically because we copy staged (not committed) files into tmp exp dirs.

And anything that is staged will be included in the experiment commit, so that it can be reproduced again later.

Feels like too much. Is it common to have something like venv/ as a stage dependency? Or are we expanding the scope here (looking ahead)? Thanks

I don't think it would be listed as a dependency, but some user stages may still depend on that directory existing. For example, I could have a pipeline stage with the command set to .venv/bin/python my_script.py

[dberenbaum]

@jdonzallaz For your particular use case, how important is it to have the credentials files as relative paths within your repo? Are there workarounds for you like using environment variables, absolute paths, or making paths relative to your home dir?

This isn't meant to dismiss the issue but to gather some more context about when and why it's important to have files like this accessible as gitignored files within the repo itself.

[jorgeorpinel]

pipeline stage with the command set to .venv/bin/python my_script.py

Any script used in cmd run should be listed under deps right? In this case pretty tricky as .venv/ is probably Git or DVC ignored, so would DVC even cache it? Interesting Q for sure, but it feels like a separate issue/discussion (we could open it though).

[pmrowla]

Any script used in cmd run should be listed under deps right? In this case pretty tricky as .venv/ is probably Git or DVC ignored, so would DVC even cache it? Interesting Q for sure, but it feels like a separate issue/discussion (we could open it though).

Standard practice would only be to include my_script.py as a dep. If I have a stage that calls /usr/bin/bash my_script.sh, I don't add /usr/bin/bash as a pipeline dependency.

And yes, standard practice would also be for users to gitignore a venv directory, the same way that they should be gitignoring __pycache__. But in real world use cases, not everyone is gitignoring everything that they should be

[jdonzallaz]

Ah, I see this issue is tricky. I've also checked the other related issues.

To answer your question @dberenbaum, there is no technical reason the credentials files could not be in another, absolute path. However feels like a hack and does not feel right to not have project-related files in the project folder, and we should add another config entry to tell where is the credential file.

At the moment, the only possibility is to git add -f / git remove. It works as a workaround but is not really convenient.
I now understand why adding all files (even git-ignored) can be a problem because of folders like venv. I do not expect those to be copied though.

I feel that git-ignored files that are explicitly listed as stage dependencies (like my credentials files) should be copied to the temp folder (without relying on git). This either automatically or through a CLI option.