Queued experiments fail, can't see logs to understand what's up

[shcheklein]

Screen.Recording.2023-07-23.at.10.29.18.AM.mov

Single experiment runs fine even within a queue. Might be related to some resource allocation when it tries to do 2 of them, but I can't say what's going on.

It's related to me researching user complaining:

But if i run the same queue again..No metrics are getting displayed in Studio
Also these logged experiments are not getting deleted

[mattseddon]

Depends on iterative/dvc#9425

[shcheklein]

cc @dberenbaum

[dberenbaum]

@shcheklein Does dvc queue logs motor-abac work?

Edit: asking to see if it's only about iterative/dvc#9425 or if it is also related to iterative/dvc#9616.

[mjunker]

I run into the same issue. My impression is that this happens mainly when running lots of experiments (100 or more). I create experiments using the cli like this:

dvc exp run -S <several params with ranges> --queue --temp -f -n <name> dvc queue start -j 8

Also I do not get any output from dvc queue logs . When I apply the failed experiment to workspace it runs without any problem.

[shcheklein]

Making it p1 (even though it's blocked) since it comes my very often (cc @dberenbaum - let me know what you think). For the record we had also a new ticket #4524 , also I think I saw it today on Discord.

[dberenbaum]

The priority makes sense, but there are two underlying issues, and I'm not sure if we are trying to cover both here:

1. exp show: provide executor information for finished experiments dvc#9425: VS Code needs to detect which experiments have logs.
2. queue: log dvc errors dvc#9616: Queued experiments fail to generate logs if error happened before stage runs.

[shcheklein]

@dberenbaum I think eventually we try to cover both. If I understand correctly the Queued experiments fail to generate logs if error happened before stage runs.. Those errors could happen if there is no data, or someone forgot to commit params.yaml, etc, etc. If there is no easy way to see those errors and then restart those experiments it becomes a problem during onboarding.

The first item will be fixed automatically if the first one is fixed, right?

What is your take? What is the complexity and scope of both on the DVC side?

[dberenbaum]

The first item will be fixed automatically if the first one is fixed, right?

What did you mean here? I don't know that either one will fix the other. We are discussing in the tickets above what the options are to solve each and what level of effort it takes.