`track` field instead of `cache` in `dvc.yaml` outputs

[iesahin]

Currently dvc.yamls cache: false has multiple meanings:

• Don't track this file at all, I don't intend to track it, it will be discarded.
• Don't track this file at all, I'm tracking it in Git.
• Don't track this file, I'm using it in dvc exp as an intermediate file.

It seems having a more granular way to express the intent to keep the file away from the cache is needed.

Instead of

extract:
     ...
    outs:
    - models/mymodel.h5
       cache: false

we can have

extract:
     ...
    outs:
    - models/mymodel.h5
       track: {cache, git, exp, lock, ignore}

which defaults to cache, (= cache: true in the current version.)

For backward compatibility, cache: true can work as track: cache, and cache: false as cache: git.

• track: cache means use a .dvc file to track the changes and use the cache. This is the default.
• track: git means don't use the cache and don't update the .gitignore file. Do nothing about this file in dvc exp run
• track: exp means don't use the cache but dvc exp run will checkout this file to .dvc/tmp/exps in dvc exp run.
• track: lock means no need to create a .dvc file but the changes will be tracked in dvc.lock file. This is for the intermediate files produced in the pipeline.
• track: ignore means both DVC and Git will ignore this file. DVC will ensure that the file is in .gitignore and won't track it.

There may be more granular options in the future.

[dberenbaum]

* Don't track this file, I'm using it in `dvc exp` as an intermediate file.

Can you give some more context about why this is needed and how you would use it?

---

There are also questions of whether to cache the file for the run-cache. See #4428.

[pmrowla]

In general this seems way more complex than just having the user use cache: true/false and managing their own .gitignore for non-dvc tracked files as needed.

---

track: cache means use a .dvc file to track the changes and use the cache. This is the default.

If we are already listing the stage output in dvc.yaml why do we need to generate a separate .dvc file for it as well

track: lock means no need to create a .dvc file but the changes will be tracked in dvc.lock file. This is for the intermediate files produced in the pipeline.

I don't understand how this is any different than having a regular cached output (the end result in both cases goes into a dvc.yaml stage outputs section, and has an entry in dvc.lock)

[karajan1001]

Output in DVC stage is used mainly for two purposes,

1. Tracking the data to cache for vision control or share to someone else.
2. Bridges to connect different stages.

So if we use cache:false at some stage that usually means this output is not very important, it is only an intermedia result. We marked them as output is only to connect the following stages.

I don't know the use cases for all of the detailed descriptions, could you please elaborate on them?