exp --queue and --run-all can hang and cause future exp show runs to hang

[rogermparent]

The original Issue has since been solved, but there's still a related problem which this Issue now tracks. See this comment for info on the still open issue.

---

Original issue:

I had mentioned this when working on plots, but it seems this is more of a general problem that occurs on master as well.

After running one experiment, live updating experiments stops working delays for at least 5 and up to 15 checkpoints before dumping all of them at once, sometimes crashing VSCode in the process. You can see the output changes as well, showing nothing until the experiments finishes and pushing all the checkpoints at once. It almost looks like it could be a dvc issue, but I don't know how it would be considering it's related to the vscode session.

live-update-bug-demo.mp4

This issue also happens when running dvc exp show on the CLI outisde of VSCode, with slightly different symptoms on the table itself where updates don't happen until a bit after the command is finished

cli-live-update-issue-demo.mp4

I've also check combinations of running from both CLI and VSCode, in both CLI -> VSCode and VSCode -> CLI. Table symptoms reflect the command run second.

[shcheklein]

So, this is DVC issue?

[mattseddon]

To me it looks like an issue with restarting from existing checkpoints, what happens if you increase the number of epochs to 15 or some number like that? I think that the 3 epochs happen so quickly that it seems like "nothing is sent until the experiment finishes". What happens if you change the params so that you are forcing another experiment to start?

Here is a demo from another repo:

Screen.Recording.2021-09-21.at.9.00.16.am.mov

We still have issues where commands are running into each other so things can be slow to update but live updates are not broken.

[rogermparent]

Let me try to replicate with higher epochs and non-continuation, I remember the same thing happening when I did so when trying plots, but it's always good to confirm.

[rogermparent]

With a few tries and a bit of luck, I finally got a demo of a 15 epoch run on two experiments that didn't crash.

cli-live-update-issue-many-epochs-demo.mp4

On a few attempts, I saw what @mattseddon hypothesized with a delayed update before live update resumed, but even when that happened the delay was between 5 and 10 epochs, and occasionally VSCode crashed at the time where other successful attempts dumped those epochs and resumed. They all share the same commonality of the first run working totally fine, but subsequent runs having issues.

[rogermparent]

@shcheklein It's really hard to pin down whether or not this actually is a DVC issue. At least for now, I'd lean towards no since symptoms can only be seen within VSCode.

[mattseddon]

@rogermparent I cannot reproduce:

Screen.Recording.2021-09-21.at.12.07.25.pm.mov

Screen.Recording.2021-09-21.at.12.08.13.pm.mov

When was the last time that you upgraded your .env (source .env/bin/activate && pip install -r requirements.txt -U)?

[mattseddon]

@rogermparent wait a minute, HOW is exp show taking 121s to run?

When was the last time you garbage collected experiments... even then WAT! How long does that action take in the CLI? The problem is whatever is causing exp show to run for an eternity. Takes ~1.2s on my machine (and my machine is garbage).

[rogermparent]

When was the last time that you upgraded your .env (source .env/bin/activate && pip install -r requirements.txt -U)?

I ran a git clean -fxd and reinstalled all dependencies before attempting that second recording, and just verified with those commands that my .env is up-to-date. It's worth noting that extension dev host + dvc reaches 13/16gb of my RAM, but doesn't seem to cross the threshold.

wait a minute, HOW is exp show taking 121s to run?

When was the last time you garbage collected experiments... even then WAT! How long does that action take in the CLI?

Great catch! It looks like exp show is starting alongside exp run, and idling until some point in the middle of exp run without ever failing like it does on your examples. Maybe I spoke too soon, and this could be a Linux-specific DVC issue related to commands running into each other- specifically run and show?

I ran GC quite a few times in the process, running it after that example (and a couple experiments later) only cleaned 4 experiments.

[mattseddon]

relates to treeverse/dvc#5739