Get rid of .dvc files?

[jorgeorpinel]

TLDR: Skip rant and go to the last bullet and code blocks.

If we're planning a non-backward compatible change in the near future, maybe it would be a good time to consider completely moving from .dvc files to dvc.yaml/dvc.lock. Or implement and leave the old .dvc file approach as optional for backward compatibility (for some time at least).

Some context in treeverse/dvc.org#1384 (review)

My motivation to suggest this is mostly conceptual right now but maybe it has some very practical implications too? Leaving that open to discussion (cc @iterative/engineering).

In DVC 1.x we created the pipelines file dvc.yaml which contains all the stages. From that point on .dvc files stopped being "stage files" and they only remain as placeholders for data files. They're no longer considered any kind of stage (we even removed the terms "stage file" and "orphan stage" from docs already). This has caused headaches in docs when explaining commands that use or affect both .dvc AND dvc.yaml or .lock files such as status, checkout, repro (pretty important ones), because we constantly need to mention both "stages and .dvc files" or "dvc.yaml and .dvc file", etc.

The options I see here are:

• Create some concept that encompasses both .dvc files and stages — already discussed with Ivan and we came up with "DVC file" but we don't love it because it's too similar to ".dvc file" so will probably cause confusion. This solution is purely docs-related and implies no action here (close this issue).

• Rethink some of these commands a little and make sure by default they only use/affect stages, and have a separate set of commands, or explicit option requirements for them to use .dvc files. To me this seems overkill

• Get rid of .dvc files! Why can't dvc.yaml (and lock) be used for this? It's a matter of introducing a new top section. Example below

dvc.yaml

data:
- corpus.csv
- dataset/

stages:
  cleanup:
    cmd: python clean.py corpus.csv df.h5
    deps:
    - corpus.csv
    outs:
    - df.h5
  ...

dvc.lock

outs:
- md5: 6137cde4893c59f76f005a8123d8e8e6
  path: df.h5
- md5: cde4876f0r5137c59f8e6a8423d8e936.dir
  path: dataset/

cleanup:
  cmd: python clean.py corpus.csv df.h5
  deps:
  - path: corpus.csv
    md5: 6137cde4893c59f76f005a8123d8e8e6
  outs:
  - path: df.h5
    md5: f40e3db3e1aa25562945045864a28deb
  ...

Something like that.

An additional advantage of this would be that it reduces the possible confusions between .dvc/ (internal dir) and .dvc (files).

[skshetry]

I have already suggested this on #3936 with a clear deprecation plan, with the same motivation as yours (simplifying ui/ux and docs).

[skshetry]

@jorgeorpinel, could also say dvc.yaml is a workflow, and .dvc is a DVC file?

[jorgeorpinel]

Ah yes, could be seen as a duplicate of that one Saugat indeed, thanks for pointing it out.

could also say dvc.yaml is a workflow, and .dvc is a DVC file?

We could but the point is I'm trying to avoid having to mention 2 things every time we're explaining the commands that affect both dvc.* and .dvc files... If there was a term that can encompass both (only idea so far is "DVC files") that would be ideal for now. Something like "internal files" or something...

[shcheklein]

My thoughts on this. No matter how do I feel about two types of files and the pain dealing with them, I would say that "I'm trying to avoid having to mention 2 things every time we're explaining the commands that affect both dvc.* and .dvc files" should not be a strong argument when we make DVC core decisions. Usability, use case, user workflow (including things like merge, working with data only) - should come first.

If for some reason it's better to have explicit .dvc when people deal with data - let's keep it. And find some elegant way to describe them in docs.

[jorgeorpinel]

I agree. I don't think that simplifying docs writing itself is a reason at all. But it tends to signal that something conceptually may need more work i.e. the workflow can be simplified for users.

If for some reason it's better to have explicit .dvc when people deal with data - let's keep it.

Great question. I guess it's a well known strategy e.g. used by Git-LFS and others (replace large files with placeholders) and it makes the data somewhat visible when you do ls even before dvc pull or similar.

On the other hand that's what we have dvc list . for too. And since we already went for the dvc.yaml file we could totally commit to that strategy and centralize everything in that almost single metafile to codify the entire DS project.

[jorgeorpinel]

Reopening for now as discussion seems active and more specific here.

[antonkulaga]

I am in favour of having everything in dvc yaml and dvc lock, otherwise you have a state where you have some files tracked by dvc.lock/yaml system and some by .dvc files. Also, .dvc files create a lot of headache for the newcomers to which I share my ML workflow as they often con

[bobertlo]

I personally like having the .dvc files. It makes sense in my head for the workflow. I wish there was like a -g flag in dvc add to automatically add them to the pending commit.

I track a lot of different datasets and I think git mechanics when reverting/merging should be considered.