exp show: Allow running while DVC is locked

[rogermparent]

While working on a way to consume dvc exp show --show-json's output, we found that DVC being locked while running an experiment can pose an issue when trying to show the table, especially if a long-running experiment were to be started outside the repo and a user wanted to view it in VS Code.

Would it be possible to have DVC CLI not check for a lock on read-only commands like exp show? Doing so could be huge for applications that want to consume DVC's output like the VSCode extension.

#3783 may also be related in that is helps solve this sort of usage.

[efiop]

@rogermparent Could you please elaborate? Some example would be great.

[rogermparent]

@rogermparent Could you please elaborate? Some example would be great.

Sure, any DVC project capable of exp show can serve as an example, but to be specific the project I encounter this issue on is an embedded https://github.com/iterative/dvc-checkpoints-mnist that we use to test the VSCode Extension against.

The extension uses dvc exp show --show-json to update the table, but this command can fail if it is run while another command, usually dvc exp run, is locking the repo. There are many ways around this on the VSCode Extension end, of course, but the question of "Why does a lock prevent a read-only command" has come up enough in discussion to warrant a ticket on the DVC CLI project, for clarification if nothing else.

There's also related discussion going on in #5750, some of it may actually be more appropriate for this Issue.

[dberenbaum]

@rogermparent Is this an intermittent error, or is there some reproducible example you can provide of when this happens? I'm trying to reproduce it now but keep getting the output in #5750 instead of any kind of error. Also, do you have the error output?

[pmrowla]

@dberenbaum this is an actual issue, but it will be difficult to reproduce manually due to the nature of the problem. The error itself will be this message:

dvc/dvc/lock.py

Lines 19 to 22 in 02d9684

	FAILED_TO_LOCK_MESSAGE = (
	"Unable to acquire lock. Most likely another DVC process is running or "
	"was terminated abruptly. Check the page {} for other possible reasons "
	"and to learn how to resolve this."

During dvc exp run, the repo will be unlocked while we run the user's pipeline commands. The repo will be completely locked while DVC is doing anything else (like writing git commits, managing exp refs, etc).

As far as actual CPU execution time goes, 99+% of the time will be spent executing the user's pipeline (so the repo will be unlocked, giving the output in #5750). If you are trying to test it manually it's unlikely you will run dvc exp show at precisely the right time to trigger the problem raised in this issue. But if you wrote a script to constantly run dvc exp show in a loop, you would see the "repo is locked" error eventually.

[rogermparent]

Thanks for the explanation @pmrowla!

As far as actual CPU execution time goes, 99+% of the time will be spent executing the user's pipeline (so the repo will be unlocked, giving the output in #5750). If you are trying to test it manually it's unlikely you will run dvc exp show at precisely the right time to trigger the problem raised in this issue.

No wonder I was having such a difficult time trying to make an explicit replication!

But if you wrote a script to constantly run dvc exp show in a loop, you would see the "repo is locked" error eventually.

With how much the vscode extension calls dvc, I'm not surprised at this replication. Is there another dvc command that reliably locks the repo for a while?

[pmrowla]

The repo is locked during pretty much any DVC command - it only gets unlocked for commands which execute user stage code (dvc run, dvc repro, dvc exp run).

You should be able to reproduce it if you set up some long running dvc add (add a directory with a very large # of files) or dvc push/pull (push/pull a large # of files) operation

[dberenbaum]

@rogermparent Is this still an issue, or is it sufficient to retry if the repo is locked?

[rogermparent]

@rogermparent Is this still an issue, or is it sufficient to retry if the repo is locked?

The integration in question (and similar ones) could certainly get by with just retrying when locked, but there seems to be interest in this issue outside of that initial issue. I'm not partial one way or the other in this regard, even if it's closed it can be reopened if necessary.

[dberenbaum]

Thanks! We can leave it open but good to know it's not a blocker for you.