consolidate `repro` and `exp run`

[casperdcl]

A possible reason why some features might be underused is naming inconsistency.

• dvc stage {add,list}
• dvc repro
• dvc run

surely should be unified as dvc stage {add,list,run} or dvc stage {add,list,repro}? Could sanitising these CLI subcommands be part of the next major release?

[dtrifiro]

I would argue that dvc run is not really useful, and I would just keep dvc stage add.

Using CLI to define stages gets very cumbersome for all but most simple stages: I'm assuming most people just just use dvc run (or stage add) to generate the boilerplate/dvc.yaml for the stage and then manually edit it according to their needs.

For example, comparing dvc run's and dvc stage add's help, the only different flags that differ are the following (in run):

  --no-exec             Only create dvc.yaml without actually running it.
  --no-commit           Don't put files/directories into cache.
  --no-run-cache        Execute the command even if this stage has already
                        been run with the same
                        command/dependencies/outputs/etc before.

My ideal workflow would be dvc stage add <name> <args> (manually tweak if required), then run dvc repro <stage> once satisfied.

To summarize:

• dvc stage {add,list} ✔️
• dvc repro ✔️
• dvc run ❌

[pmrowla]

I think this is mostly a duplicate of #5846

[dberenbaum]

In the docs, dvc run has already been removed almost everywhere in treeverse/dvc.org#3223.

I think this is mostly a duplicate of #5846

Mostly, although the suggested syntax is slightly different and means getting rid of repro.

We could also keep it open or move it to a discussion to keep track of ideas for renaming subcommands generally. We are already planning to split dvc status into dvc data status and dvc stage status. Many other commands already make sense as subcommands of dvc stage/data or might benefit by being split.

I don't think we are ready to promise this yet, though, and it's unclear whether we would actually deprecate the existing commands, which could be painful for many existing users.

Using CLI to define stages gets very cumbersome for all but most simple stages: I'm assuming most people just just use dvc run (or stage add) to generate the boilerplate/dvc.yaml for the stage and then manually edit it according to their needs.

The ideal workflow is probably to generate stages in VS Code with auto-completion (and maybe other helpers). I think it's already on the roadmap but not sure when it will happen. For now, dvc exp init is probably the easiest way to generate a boilerplate stage.

[karajan1001]

If we migrate all of the existing command to an experiment based workflow as well as dvc exp related commands, I think we can deprecate the subcommand exp then.

[karajan1001]

If we migrate all of the existing command to an experiment based workflow as well as dvc exp related commands, I think we can deprecate the syntax exp then.

[jorgeorpinel]

migrate all of the existing command to an experiment based workflow

Is there an issue or story about this?

[karajan1001]

Currently not.

[casperdcl]

@dtrifiro from this I understand you prefer repro over run and would like to (soft)deprecate the latter.

In that case, isn't dvc stage {add,list,repro} still less confusing than the current dvc {stage {add,list},repro}?

It would be consistent with dvc exp interface.

[skshetry]

dvc run only executes one single stage, so it's more like dvc stage add ... + dvc repro --single-stage <name>.
So it does not make sense to put that into dvc stage repro. stage is a lower level utility compared to a very high-level exp command.

[karajan1001]

Why can't we have them both, dvc stage repro to keep consistent with other stage commands and dvc repo as a shortcut?

[jorgeorpinel]

prefer repro over run and would like to (soft)deprecate the latter

run is alerady soft-deprecated.

Why can't we have them both

Why do we need them both? repro is a top level operation (not stage-specific) as @skshetry pointed out.
Otherwise, I think we should discuss the proposed stage run/repro in the existing #5846, as @pmrowla pointed out.

[casperdcl]

It's not clear to me that #5846 is a solution. I'm asking about use cases rather than implementation/adding-even-more-confusingly-named-options.

A) What are the underlying concepts? afaik it's:

1. Pipelines

• makefile-like DAG
• Git committed & pushed

2. Experiments

• ephemeral/throw-away, not Git pushed
• per-epoch checkpoints, DVCLive monitoring

B) does the commandline interface map 1:1 to the above concepts?
C) do the docs map 1:1?

Am I missing something? Need to sort out (A) before there's any hope of addressing (B) & (C).