exp: include all expected files

[dberenbaum]

There are several issues where files/changes from the workspace end up not being reflected in the experiment. This is a meta-ticket to track those issues:

### Tasks
- [ ] #6297
- [ ] #8998
- [ ] https://github.com/iterative/dvc/issues/9058
- [ ] #8924 
- [ ] https://github.com/iterative/dvc/issues/9238
- [ ] https://github.com/iterative/dvclive/issues/510
- [ ] #7254
- [ ] #7842 
- [ ] #5800 

Edit: reordered to reflect priority

[dberenbaum]

Related: #9165 (comment)

[daavoo]

Related: #9165 (comment)

Added #9238 to the list

[daavoo]

Added treeverse/dvclive#510 to the list (kind of extracted from #9165 )

[daavoo]

@dberenbaum I think we are in good shape for getting this done in this sprint, except for #5800 .
I have tried to go through the discussion and its still unclear to me what is the request 😅 Do you recall anything in particular?

[dberenbaum]

Makes sense, thanks @daavoo! Now we are tracking everything that I can think of to reasonably track by default. If there are other files people expect to be included, we could:

1. Add an option to specify files to copy over.
2. Copy everything over from the workspace.

I'm torn on whether it's worth prioritizing anything else, but what do you think about including something like 1? Do you think it's easy to add?

[daavoo]

what do you think about including something like 1?

Sounds good to give the option although it feels like if you have to know about that option + provide the right values you might as well do git add, right?

We could anyhow do something like the exp save --include-untracked

Do you think it's easy to add?

I think so, yes. 1 day.

[dberenbaum]

Sounds good to give the option although it feels like if you have to know about that option + provide the right values you might as well do git add, right?

We could anyhow do something like the exp save --include-untracked

Yeah, I'm not sure that exp save --include-untracked has an obvious use case. The use cases I have in mind are:

1. Credentials or other sensitive info that shouldn't be tracked by Git (like the OP in term: replace 'DVC-file' and 'stage file' with correct terms (#3960) #5280)
2. Large files that you don't want to track (for example, it's static or easy to regenerate; see https://github.com/iterative/cse/issues/99)

So I think it needs to be more like --copy-untracked where the files get copied to the tmp dir but remain untracked.

[daavoo]

So I think it needs to be more like --copy-untracked where the files get copied to the tmp dir but remain untracked.

So, is it about the cases where --tmp / --queue fail because the experiment relies on something that is:

• Not git tracked
• Not defined as an absolute path

Correct?

---

For that, I am not sure about estimation without a clear scope.

[dberenbaum]

So, is it about the cases where --tmp / --queue fail because the experiment relies on something that is:

* Not git tracked

* Not defined as an absolute path

Correct?

Yes

For that, I am not sure about estimation without a clear scope.

I think you just defined the scope very well 😄. Do you still have questions, or you just would need to look closer to give a decent estimation?

[daavoo]

I think you just defined the scope very well 😄. Do you still have questions, or you just would need to look closer to give a decent estimation?

Forgot to estimate. 1 day was right 😅
I left some questions on the P.R. #9302