Release 3.0

[efiop]

## Data Management
- [ ] https://github.com/iterative/dvc/issues/4658
- [x] #9531 (@efiop)
- [x] Rename `dvc add --jobs` to `--remote-jobs` (frequently gets confused with core.checksum_jobs)
- [ ] https://github.com/iterative/dvc/pull/9591

## Experiments and Pipelines
- [ ] #9221
- [x] add `dvc stage add --run`
- [x] [Remove Stage-level vars](https://dvc.org/doc/user-guide/project-structure/dvcyaml-files#:~:text=Stage%2Dspecific%20values%20are%20also%20supported) (@skshetry)
- [ ] ~`dvc exp push` changes? (see https://iterativeai.slack.com/archives/C01116VLT7D/p1686570450222809?thread_ts=1686321601.633079&cid=C01116VLT7D)~ decided not to make breaking changes now; will revisit dropping `--rev HEAD`

## Deprecations
- [x] https://github.com/iterative/dvc/discussions/5940. (@skshetry )
- [x] Remove [~`exp gc`~](https://github.com/iterative/dvc/issues/7991), [~`exp init`~](https://github.com/iterative/dvc/issues/9267), [~`exp show --html`~](https://github.com/iterative/dvc/issues/9268), ~`exp show --pcp`~.
- [x] Deprecate `dvc add --recursive` flag.
- [x] Remove `dvc run`.
- [x] Remove `--show-json`/`--show-csv`/`--show-md` in favor of existing `--json`/`--csv`/`--md` flags.
- [x] ‍‍‍‍‍[Drop top-level plots definition as a dictionary, can be defined as a list instead](https://github.com/iterative/dvc/pull/8412).
- [x] Drop 1.0 `dvc.lock` format support.
- [x] Soft deprecate `desc`/`type`/`labels`/`meta` in `dvc.yaml` and `.dvc` files, now that we have artifacts (drop CLI flags, `dvc data ls`, and drop them from docs).
- [x] https://github.com/iterative/dvc/issues/9457 (and replace with `--message` short flag)
- [x] Deprecate dvc add --file (obscure and more likely to confuse than help)

Other release blockers

• --allow-missing: skips stages even if deps have changed #9530
• data status --not-in-remote: not working #9541
• add/import-url: fix cloud versioning #9584

Studio Readiness for 3.0 release

• https://github.com/iterative/studio/issues/5944

[dmpetrov]

That's a great summary of the coming changes. I'd appreciate it if some of the items could be clarified.

• introduce data(name is subject to change) section in dvc.yaml for data (also need to look into imports)

Should we introduce similar for metrics, plots?

• default hash change.

It would be helpful if you could clarify this.

• deprecate --show-* flags
• remove --show-vega flag

Where does this requirement come from? How users will generate png files from our templates in CI/CD?

[efiop]

Should we introduce similar for metrics, plots?

Sure, seems like plots will be the first. Just trying to mark this new approach with introducing separate sections.

It would be helpful if you could clarify this.

CC @pared

[pared]

@dmpetrov

deprecate --show-* flags

I guess this is about migrating --show-json and friends to --json and similar, just to make it shorter.

remove --show-vega flag

We recently introduced '--json' for plots. It's not official yet, but since we support images, we will need to do it sooner or later (officially) in automation scenarios too. In that case --show-vega does not make sense, because its limited to only one plots type. Results obtained by --show-vega can be achieved by specifying singular target for --json so there is not too much argument for keeping it.

[dberenbaum]

@pared I thought --json was a hidden option for VS Code?

For the vega example in https://cml.dev/doc/cml-with-dvc using vl2png, how can this be handled with --json?

[pared]

@pared I thought --json was a hidden option for VS Code?

Well, it seems to me that we will have to start supporting it. It kind of does not make sense from user perspective, to support vega as json and not do the same for images - our current problems "where to store the results" should probably not affect the enduser.

For the vega example in https://cml.dev/doc/cml-with-dvc using vl2png, how can this be handled with --json?

Some json manipulation is needed for that. In this particular example:

dvc plots diff \
            --target classes.csv \
            --template confusion \
            -x actual \
            -y predicted \
            --json master | jq '."classes.csv" | .[0]' >> vega.json

[daavoo]

For the vega example in https://cml.dev/doc/cml-with-dvc using vl2png, how can this be handled with --json?

Some json manipulation is needed for that. In this particular example:

dvc plots diff \
            --target classes.csv \
            --template confusion \
            -x actual \
            -y predicted \
            --json master | jq '."classes.csv" | .[0]' >> vega.json

I think send-comment with vega plot is a widely used example. Opened issue iterative/cml.dev#172

[pared]

I think send-comment with vega plot is a widely used example. Opened issue

Good idea though let's not solve it until we agree on whether --json is what we want and that we remove --show-vega.

[dberenbaum]

Moved the plots discussion to #7183.

[karajan1001]

One question here, because there might be some subrepos deep inside directories. Even if we had deprecated .dvc files. We might still need to walk through the whole structure to find out all of the stages in subrepos. Or We will need to make a clear and better way to handle subrepos.

[skshetry]

@karajan1001, we do not traverse into subrepos except for dvc list. dvc.api/dvc get/dvc import uses particular logic to find subrepos, but does not require it as we avoid getting files from subrepos. There have been discussions on getting rid of the "subrepos traversal" completely as it's not worth it with the complexity it introduces.