New command: dvc verify - check that the pipeline is up to date without having to pull or run it

[sjawhar]

Bringing this over from Discord.

What I'd like is a command to use in a CI/CD process to check that the DVC pipeline is in a valid state. By "valid", I mean that:

1. All deps of all stages are either in the remote cache or match what's in the workspace, using the hashes in dvc.lock
2. All outs of all stages are either in the remote cache or match what's in the workspace, using the hashes in dvc.lock
3. If any stage has an out that is the dep of any other stage, they have the same hash in dvc.lock
4. All params match what's in dvc.lock

Essentially, this is the same as running dvc pull then dvc repro --dry and checking that "Data and pipelines are up to date", except I don't want to have to run dvc pull. This is especially important if you're working with large datasets, as pulling them every time on a CI machine could be quite costly in time and/or actual dollar bills 💰

I played around a bit to see if there's a workaround with only the current functionality. Here's what I found.

dvc status by itself will tell you if any non-cached deps (e.g. source code) don't match dvc.lock. That'll look like this:

train:
        changed deps:
                deleted:            data/processed/data_train.npz
                deleted:            data/processed/data_val.npz
                modified:           src/train <-- This one

dvc status -c will tell you if any outputs of any stages listed in dvc.lock aren't in remote storage. That'll look like this:

        missing:            data/processed/data_test.npz   <-- This one 
        deleted:            data/processed/data_train.npz
        deleted:            data/processed/data_val.npz

dvc params diff will obviously catch param changes. I don't think anything covers point 3 above, and even if I stitched these all together, it would be very brittle as it relies on the outputs of all these commands not changing.

As always, I'm happy to contribute the change if you all think it would be valuable. I know I would use it right away in several projects. Or let me know if I'm simply overlooking some existing functionality that would serve the same purpose.

[efiop]

Hi @sjawhar !

As always, great suggestion! I think this feature has been requested before in different shapes and forms (can't pinpoint a particular ticket right now, sorry) and is generally related to dvc being able to operate in a "virtual environment" with remote and run-cache in mind, so that missing local cache is not a problem as long as stuff is available on remote. I think the functionality you've requested still makes sense for dvc status (maybe as a flag) instead of a separate verify command, unless that new separate command is more tailored to the pipelines (we've had this idea of smth like dvc pipeline/stage/e.g. status for a long time).

Maybe this might fit into dvc exp better these days? 🤔 (CC @dberenbaum @pmrowla) E.g. there will be remote executors and stuff, so maybe this global state will somehow fit better into that paradigm. This is just a random thought though, no more than a rant, so nevermind.

[dberenbaum]

@efiop Although I haven't used status extensively, I agree that it intuitively makes sense as a flag for that command. I also agree with your final analysis that it probably doesn't make sense as part of dvc exp 😄

[antonpetkoff]

+1 for dvc verify! It is very useful for CI tests that prevent you from going forward with a stale pipeline.

[davesteps]

+1 I think this would be a very nice feature for when you are doing CI/CD tests to confirm your branch is ready to merge. I opened this iterative/cml#681 issue because I thought it might be a cml feature but actually I think this would fit the bill.

[yummydum]

Hi, I consider this feature very useful in my workflow as well.
For the time being, if there is a workaround script available, would anyone share it?

[mraxilus]

Any updates on this feature?
I would find it quite useful for my CI pipeline process.

[dberenbaum]

tl;dr It's on our roadmap but not close enough to give an estimate for it yet.

A lot of the functionality today is part of dvc status. Our plan is to separate the data management aspects of dvc status from this pipeline-focused dvc verify functionality (like dvc data status and dvc stage status; actual command names TBD) and make each more robust and useful on their own. We are currently focused on addressing gaps in data management generally, including updating dvc status, and those improvements should be coming soon. Once that's complete, we should have more capacity to focus on pipeline improvements, and this feature request will be a high priority.