Epic: exp init

[dberenbaum]

Parent issue for lightweight command for initializing experiments.

Linked issues:

• exp init: create command to append a default experiment stage #6441
• exp init: configure path values in experiment stage #6442
• exp init: configure experiment stage templates #6443
• exp init: add dvclive template #6444
• exp init: make configuration reusable across projects #6445

Small tasks that are left to do:

• Autostaging dvc.yaml/.gitignore/params.yaml (Epic: exp init #6446 (comment))
• init dvc repo if uninitialized (Epic: exp init #6446 (comment))
• Try to ask for commands before asking for paths (exp init: iterate on ui, tree and show to-be-generated yaml content #6750 (comment))
• Use train as a default name? (exp init: iterate on ui, tree and show to-be-generated yaml content #6750 (comment))
• Raise issue if dvc is initialized with --no-scm or if the project is not git init-ed.
• exp init: when to fail for non-existent params #6865
• Add data as a dvc-tracked directory (Epic: exp init #6446 (comment))
• Make confirmation prompt at the end look visually similar to other prompts.
• Update docs after doc: shorter links to user-guide dvc.org#2967 is merged.
• Add tests/QA and improve CLI help message (if any).
• Improve messaging/UI/guidance/prompts on the command.

[karajan1001]

Maybe we can have a label for these summaries?

[skshetry]

From the @dberenbaum's comment on #6681 (comment).

How do you anticipate this (dvclive scenario) working for non-interactive mode?

I'm still wondering whether the template option is needed (solely as a cli option, not as an actual template; it could be renamed) to separate the dvclive scenario and provide flexibility to add other scenarios in the future. What do you think?

One way may be to make --live mandatory for dvclive scenario unless live is set on the config (though there's a question about what to do if metrics and plots are also specified in the config). Maybe keeping live from config is also a way to go.

Another would be to create a --template or something similar flag (--with={live, etc.})?

[dberenbaum]

Another would be to create a --template or something similar flag (--with={live, etc.})?

Although it adds another option, I like this mainly because it's more explicit. Now that we don't have templates, it's less transparent what the command and various options are doing. Having a separate option for this suggests that it's not just another path to add but actually changes the stage configuration. We can also further explain it in the help text for that option.

On a similar note, at least interactive mode should probably have a confirmation so that users can see what will be added to dvc.yaml beforehand. There could be an option to do the same outside of interactive mode (--dry?). Not sure if this has been discussed before for dvc stage add.

[skshetry]

Tasks that remains:

• Improve UI/CLI, messaging, and prompts
• Add config support
• Add tests/QA
• Default metrics file #6168
• params: bulk params from file #6605

[daavoo]

Might be worth revisiting iterative/dvcyaml-schema#7 to better support the workflow:

dvc exp init for simple cases / basic bootstrapping -> manually edit dvc.yaml for customization?

[skshetry]

With dvcyaml-schema, my goal was to have same schema for JSONSchema and validation. We have been using pydantic for JSONSchema generation and voluptuous for validation.

There are few issues with using pydantic:

1. It does not have a good support for validating dictionaries with varying keys. stages in dvc.yaml is a dictionary with key being stage name and value being stage information. Pydantic is not very good at representing this. Similarly, deps/outs are also list of dictionaries.
2. It is almost twice as slow as voluptuous, because it uses type-hints.
3. It coerces types (though you can make it stricter). It may be a good thing sometimes, but it does make it quite tricky to make voluptuous-based schema to be compatible with pydantic-based schema.

In short term, I think it's better to move pydantic schema to dvc, and support both of those schema.

Voluptuous also has negatives, that it's not very actively maintained, and is a bit hard to extend to have custom validations
compared to pydantic (whose DX is great).

Also, looking for cattrs is another option, but I believe it'll have same issues as pydantic. Though migrating to any other libraries may take longer time, as it's quite hard to guarantee compatibilities (we rarely test unhappy paths for dvc.yaml) and the need for QA'ing.

[dberenbaum]

We should probably include docs work also.

[dberenbaum]

I noticed a couple of workflow issues when doing some QA of dvc exp init:

1. It's unclear when or what to git add. If I don't git add the dvc.yaml or the dependencies, dvc exp init works in the workspace but not in temp.
2. If I am adding a data dependency, I most likely want to dvc add it rather than git add.

cc @pmrowla

[pmrowla]

New/untracked files aren't pulled into the --temp/--queue temporary workspace by default.

from https://dvc.org/doc/user-guide/experiment-management/running-experiments#the-experiments-queue:

Note that Git-ignored files/dirs are explicitly excluded from queued/temp runs to avoid committing unwanted files into Git (e.g. once successful experiments are persisted).

💡 To include untracked files, stage them with git add first (before dvc exp run) and git reset them afterwards.

It sounds like we need:

• deps from exp init should get .dvc files (dvc added)
• exp run --temp should just automatically stage any untracked .dvc or dvc.yaml files (it seems like this is probably the desired default behavior in all cases, not just exp init specific ones?)
  • this doesn't address everything and there may be other untracked files that users actually need to manually stage for their experiment, but we can't really automate behavior for non-dvc-related files