Epic: exp queue CLI

[pmrowla]

Meta-issue for initial planned exp queue issues. (This does not cover "nice to haves" as outlined in the notion proposal)

dvc-task/exp backend tasks:

• exp: implement internal queue.kill support #7587

exp queue CLI tasks:

• exp queue: start/stop queue processing #7589
• exp queue: monitor queue status #7590
• exp queue: remove queued experiments #7591

[daavoo]

Should we close #5615 in favor of this one?

[pmrowla]

I think we can leave it open for now, this is more just tasking for the current planned work, but the original discussion contains some more features that are currently unplanned/out of scope

[pmrowla]

@dberenbaum queue start/stop/remove/kill are all available in the dev branch for testing now. The status commands and stop --kill will be added in follow-up PRs.

[dberenbaum]

This is looking good now! I wanted to summarize some of the questions/issues I still have:

1. What to do with existing commands and flags? Specifically, dvc exp run --run-all and dvc exp remove --queue?
2. How can we check if the queue is active/running/processing?
3. Should we show both queued and failed experiments in dvc exp show and have flags to show/hide them?
4. How should queued experiments be named and should they match the final experiment name?

[dberenbaum]

I would say the first two are critical to resolve before making it public.

Also, there's one more general bug I have seen: sometimes the queue does not reflect the order in which experiments were added. Let me know if you need more info.

[pmrowla]

1. What to do with existing commands and flags? Specifically, dvc exp run --run-all and dvc exp remove --queue?

I think the existing commands should essentially be aliases to the new behavior, and formally deprecated, so when the user runs them we would output something like

dvc exp run --run-all is deprecated and will be removed in a future DVC release. Please use dvc queue start instead.

For the aliased behavior:

• exp run --temp should just be exp run --queue + queue start + queue logs <exp>
• exp run --run-all should just be queue start. I don't think we should try to duplicate the existing log behavior for --run-all since it wasn't particularly useful anyways (instead we can just output a message saying use queue logs to view logs for a specific experiment)
• exp remove --queue should be something like queue remove --all
  • We still need to add an --all flag for queue remove, but since we also show finished experiments in queue status now we need distinct flags for removing "all queued experiments" vs "all failed/successful experiments" vs "all queued+completed experiments" (see further comments below)

2. How can we check if the queue is active/running/processing?

For now I think we can just add this to queue status (start the output with line(s) listing the number of running workers). Eventually it will probably be worth it to have a queue ps or queue proc command for getting details about each worker, but for now status should at least indicate whether or not the queue worker is actively running.