exp run: missing dependencies of frozen stages raise an error

[sjawhar]

Bug Report

Description

When calling dvc exp run with a frozen stage, an error is raise if dependencies of that frozen stage or outputs of stages before that stage are missing in the project directory.

Reproduce

stages:
  dep:
    cmd: mkdir dep && echo 'dep' > dep/dep.txt
    deps:
      - start.txt
    outs:
      - dep

  frozen:
    cmd: echo 'frozen' > frozen.txt
    deps:
      - dep
    outs:
      - frozen.txt

  next:
    cmd: echo 'next' > next.txt
    deps:
      - frozen.txt
    outs:
      - next.txt

1. Save the dvc.yaml above
2. dvc config --local cache.type reflink,hardlink,symlink
3. echo start > start.txt
4. dvc repro
5. rm $(readlink dep/dep.txt) start.txt frozen.txt
6. rm -rf dep
7. echo new > frozen.txt
8. dvc freeze frozen
9. dvc exp run dvc.yaml
   • The same error is raised by --temp and --queue then --run-all

$ dvc exp run --temp -v dvc.yaml
2021-11-06 04:23:39,265 DEBUG: Detaching HEAD at 'HEAD'               
2021-11-06 04:23:39,283 DEBUG: Stashing workspace
2021-11-06 04:23:39,394 DEBUG: Stashing changes in 'refs/stash'
2021-11-06 04:23:39,417 DEBUG: Applying stash commit 'cd53502fcbe907f71d45df87420d7466bfaa2ecf'
2021-11-06 04:23:40,119 DEBUG: Stashing changes in 'refs/exps/stash'
2021-11-06 04:23:40,163 DEBUG: Stashed experiment 'f03880d' with baseline '6ffd8ba' for future execution.
2021-11-06 04:23:40,171 DEBUG: Restoring stashed workspace
2021-11-06 04:23:40,171 DEBUG: Popping from stash 'refs/stash'
2021-11-06 04:23:40,172 DEBUG: Applying stash commit 'cd53502fcbe907f71d45df87420d7466bfaa2ecf'
2021-11-06 04:23:40,180 DEBUG: Dropping 'refs/stash@{0}'
2021-11-06 04:23:40,185 DEBUG: Restore HEAD to 'master'
2021-11-06 04:23:40,186 DEBUG: Reproducing experiment revs 'f03880d'
2021-11-06 04:23:40,512 DEBUG: Writing experiments local config '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/.dvc/config.local'                                                                                                                         
2021-11-06 04:23:40,512 DEBUG: Init temp dir executor in dir '<ExpTemporaryDirectory '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'>'
2021-11-06 04:23:40,602 DEBUG: Running repro in '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'
2021-11-06 04:23:40,603 DEBUG: Removing '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/.dvc/tmp/repro.dat'
2021-11-06 04:23:41,127 DEBUG: Cache type 'reflink' is not supported: reflink is not supported                                                                                                                                                         
2021-11-06 04:23:41,127 DEBUG: Cache type 'hardlink' is not supported: failed to link                                                                                                                                                                  
2021-11-06 04:23:41,128 DEBUG: Created 'symlink': ../../../../../cache/4d/70e93c98a3a5ff1e7e5160d909a9b9 -> next.txt                                                                                                                                   
2021-11-06 04:23:41,129 DEBUG: state save (58985507, 1636170526467139584, 5) 4d70e93c98a3a5ff1e7e5160d909a9b9                                                                                                                                          
2021-11-06 04:23:41,144 DEBUG: state save (58985507, 1636170526467139584, 5) 4d70e93c98a3a5ff1e7e5160d909a9b9                                                                                                                                          
2021-11-06 04:23:41,145 DEBUG: Created 'symlink': ../../../../../cache/9c/d599a3523898e6a12e13ec787da50a -> frozen.txt                                                                                                                                 
2021-11-06 04:23:41,146 DEBUG: state save (58985508, 1636171699415656448, 4) 9cd599a3523898e6a12e13ec787da50a                                                                                                                                          
2021-11-06 04:23:41,154 DEBUG: state save (58985508, 1636171699415656448, 4) 9cd599a3523898e6a12e13ec787da50a                                                                                                                                          
2021-11-06 04:23:41,156 DEBUG: Created 'symlink': ../../../../../cache/2a/4732b1de5db823e94d662d207b8fb2 -> dep/dep.txt                                                                                                                                
2021-11-06 04:23:41,157 ERROR: unexpected error - [Errno 2] No such file or directory: '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/dep/dep.txt'                                                                                         
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/experiments/executor/base.py", line 446, in _repro_dvc
    yield dvc
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/experiments/executor/base.py", line 343, in reproduce
    dvc_checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/checkout.py", line 98, in checkout
    result = stage.checkout(
  File "/usr/local/lib/python3.8/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/usr/local/lib/python3.8/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/stage/__init__.py", line 572, in checkout
    key, outs = self._checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/stage/__init__.py", line 584, in _checkout
    result = out.checkout(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/output.py", line 746, in checkout
    modified = checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/objects/checkout.py", line 275, in checkout
    _checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/objects/checkout.py", line 221, in _checkout
    _checkout_file(
  File "/usr/local/lib/python3.8/site-packages/dvc/objects/checkout.py", line 151, in _checkout_file
    state.save(path_info, fs, change.new.obj.hash_info)
  File "/usr/local/lib/python3.8/site-packages/dvc/state.py", line 82, in save
    mtime, size = get_mtime_and_size(path_info, fs, self.dvcignore)
  File "/usr/local/lib/python3.8/site-packages/dvc/utils/fs.py", line 63, in get_mtime_and_size
    base_stat = fs.info(path)
  File "/usr/local/lib/python3.8/site-packages/dvc/fs/local.py", line 160, in info
    return self.fs.info(path_info)
  File "/usr/local/lib/python3.8/site-packages/fsspec/implementations/local.py", line 83, in info
    out = os.stat(path, follow_symlinks=True)
FileNotFoundError: [Errno 2] No such file or directory: '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/dep/dep.txt'
------------------------------------------------------------
2021-11-06 04:23:41,159 DEBUG: Removing '/home/sjawhar/example/.dvc/tmp/exps/run/f03880d31cbf3d857c2aba7c6057a50fa276ed84.run'
2021-11-06 04:23:41,164 ERROR: Failed to reproduce experiment 'f03880d'
2021-11-06 04:23:41,164 DEBUG: Removing tmpdir '<ExpTemporaryDirectory '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'>'
2021-11-06 04:23:41,165 DEBUG: Removing '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'
2021-11-06 04:23:41,178 DEBUG: Dropping 'refs/exps/stash@{0}'         
2021-11-06 04:23:41,181 DEBUG: Analytics is enabled.
2021-11-06 04:23:41,226 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp4qhfkjh0']'
2021-11-06 04:23:41,227 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp4qhfkjh0']'

Expected

An experiment to be run successfully, starting from stage next

Environment information

I'm using the symlink cache type

Output of dvc doctor:

DVC version: 2.8.1 (pip)
---------------------------------
Platform: Python 3.8.8 on Linux-5.13.0-7614-generic-x86_64-with-glibc2.2.5
Supports:
        hdfs (fsspec = 2021.10.0, pyarrow = 4.0.1),
        webhdfs (fsspec = 2021.10.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2021.10.0, boto3 = 1.17.106),
        ssh (sshfs = 2021.11.0)
Cache types: symlink
Cache directory: ext4 on /dev/mapper/data-root
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/data-root
Repo: dvc, git

Additional Information (if any):
From some basic debugging, I think I've tracked this down to how experiments collect stages. Specifically, they call StageLoad.collect(with_deps=True), whereas dvc repro doesn't. So the experiment seems to be pulling extra stages.

dvc/dvc/repo/experiments/__init__.py

Lines 288 to 305 in be61284

	def _stash_commit_deps(self, *args, **kwargs):
	if len(args):
	targets = args[0]
	else:
	targets = kwargs.get("targets")
	if isinstance(targets, str):
	targets = [targets]
	elif not targets:
	targets = [None]
	for target in targets:
	self.repo.commit(
	target,
	with_deps=True,
	recursive=kwargs.get("recursive", False),
	force=True,
	allow_missing=True,
	data_only=True,
	)

[karajan1001]

I'm sorry, I did not quite understand here.

dvc exp run dvc.yaml

This will run all of the stages in dvc.yaml and I didn't see any frozen flag in your dvc.yaml. I tried to use dvc freeze frozen to freeze the frozen stage and run dvc exp run next it shows something below:

dvc exp run next
WARNING: stage: 'frozen' is frozen. Its dependencies are not going to be reproduced.
Stage 'frozen' didn't change, skipping
Stage 'next' didn't change, skipping

[sjawhar]

Sorry, I just forgot to add the dvc freeze frozen step before step 8. I'll amend the instructions above.

[karajan1001]

Still can not reproduce here.

 $ dvc exp run dvc.yaml

ERROR: failed to reproduce 'dvc.yaml': [Errno 2] No such file or directory: '/Users/gao/Code/test/fix6943/start.txt'
$ dvc exp run frozen

WARNING: stage: 'frozen' is frozen. Its dependencies are not going to be reproduced.
Stage 'frozen' didn't change, skipping
$ dvc exp run dep

ERROR: failed to reproduce 'dvc.yaml': [Errno 2] No such file or directory: '/Users/gao/Code/test/fix6943/start.txt'

$ dvc exp run --temp -v dvc.yaml
2021-11-08 15:31:26,714 DEBUG: Detaching HEAD at 'HEAD'
2021-11-08 15:31:26,745 DEBUG: Stashing workspace
2021-11-08 15:31:26,753 DEBUG: No changes to stash
2021-11-08 15:31:27,353 DEBUG: Stashing changes in 'refs/exps/stash'
2021-11-08 15:31:27,370 DEBUG: Stashed experiment 'ab4b052' with baseline 'c849c37' for future execution.
2021-11-08 15:31:27,373 DEBUG: Restore HEAD to 'master'
2021-11-08 15:31:27,374 DEBUG: Reproducing experiment revs 'ab4b052'
2021-11-08 15:31:27,393 DEBUG: Init temp dir executor in dir '<ExpTemporaryDirectory '/Users/gao/Code/test/fix6943/.dvc/tmp/exps/tmpmmhys94q'>'
2021-11-08 15:31:27,393 DEBUG: Writing experiments local config '/Users/gao/Code/test/fix6943/.dvc/tmp/exps/tmpmmhys94q/.dvc/config.local'
2021-11-08 15:31:27,778 DEBUG: Running repro in '/Users/gao/Code/test/fix6943/.dvc/tmp/exps/tmpmmhys94q'
2021-11-08 15:31:27,778 DEBUG: Removing '/Users/gao/Code/test/fix6943/.dvc/tmp/exps/tmpmmhys94q/.dvc/tmp/repro.dat'
2021-11-08 15:31:28,058 DEBUG: Created 'reflink': ../../../cache/.cache_type_test_file -> dep/.kydxDbdtvfc3iH8dmGNn4A
2021-11-08 15:31:28,059 DEBUG: Removing 'dep/.kydxDbdtvfc3iH8dmGNn4A'
2021-11-08 15:31:28,059 DEBUG: Removing '../../../cache/.cache_type_test_file'
2021-11-08 15:31:28,060 DEBUG: Created 'reflink': ../../../cache/2a/4732b1de5db823e94d662d207b8fb2 -> dep/dep.txt
2021-11-08 15:31:28,061 DEBUG: state save (129183809, 1636356469717646848, 4) 2a4732b1de5db823e94d662d207b8fb2
2021-11-08 15:31:28,064 DEBUG: state save (129183806, 475e51fc867a85818e85a2a127f57a9e, 4) 9d3358ff12f2eb972c3fd7fe3d9327fc.dir
2021-11-08 15:31:28,066 DEBUG: Created 'reflink': ../../../cache/4d/70e93c98a3a5ff1e7e5160d909a9b9 -> next.txt
2021-11-08 15:31:28,066 DEBUG: state save (129183814, 1636356469808347648, 5) 4d70e93c98a3a5ff1e7e5160d909a9b9
2021-11-08 15:31:28,067 DEBUG: state save (129183814, 1636356469808347648, 5) 4d70e93c98a3a5ff1e7e5160d909a9b9
2021-11-08 15:31:28,069 DEBUG: Created 'reflink': ../../../cache/27/e01d3d367250e9e0b455a721fd7ea8 -> frozen.txt
2021-11-08 15:31:28,069 DEBUG: state save (129183817, 1636356469772402688, 7) 27e01d3d367250e9e0b455a721fd7ea8
2021-11-08 15:31:28,070 DEBUG: state save (129183817, 1636356469772402688, 7) 27e01d3d367250e9e0b455a721fd7ea8
2021-11-08 15:31:28,081 WARNING: stage: 'frozen' is frozen. Its dependencies are not going to be reproduced.
Stage 'frozen' didn't change, skipping
Stage 'next' didn't change, skipping
2021-11-08 15:31:28,087 DEBUG: Dependency 'start.txt' of stage: 'dep' changed because it is 'deleted'.
2021-11-08 15:31:28,087 DEBUG: stage: 'dep' changed.
2021-11-08 15:31:28,088 DEBUG: Removing output 'dep' of stage: 'dep'.
2021-11-08 15:31:28,088 DEBUG: Removing 'dep'
2021-11-08 15:31:28,089 ERROR: failed to reproduce 'dvc.yaml': [Errno 2] No such file or directory: '/Users/gao/Code/test/fix6943/.dvc/tmp/exps/tmpmmhys94q/start.txt'

errors only come from stage dep which lacks dependency start.txt. Looks like we are in different link type, and my configuration did not work.

$ dvc config -l
cache.type=symlink

[sjawhar]

This is me following the steps outlined above and producing the error

[karajan1001]

I switched to a Linux computer but still can't reproduce it, can anybody else reproduce it? @iterative/dvc

[pared]

Seems like I am able to reproduce on mac:

#!/bin/bash

set -ex

pushd $TMPDIR
rm -rf test_wspace
mkdir test_wspace
pushd test_wspace

mkdir test_repo
pushd test_repo

git init 
dvc init

dvc config --local cache.type symlink

git add -A
git commit -am "init"

echo start >> start.txt

echo -e "stages:
  dep:
    cmd: mkdir dep && echo 'dep' > dep/dep.txt
    deps:
      - start.txt
    outs:
      - dep

  frozen:
    cmd: echo 'frozen' > frozen.txt
    deps:
      - dep
    outs:
      - frozen.txt

  next:
    cmd: echo 'next' > next.txt
    deps:
      - frozen.txt
    outs:
      - next.txt" >> dvc.yaml

dvc repro next
git add -A
git commit -am "ble"

rm -f $(readlink dep/dep.txt) start.txt frozen.txt
rm -rf dep

echo new > frozen.txt
dvc freeze frozen
dvc exp run dvc.yaml

version:

DVC version: 2.8.4.dev3+g87fadd32 (osxpkg)
---------------------------------
Platform: Python 3.8.12 on macOS-11.6-x86_64-i386-64bit
Supports:
        azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.7.0),
        gdrive (pydrive2 = 1.10.0),
        gs (gcsfs = 2021.11.0),
        hdfs (fsspec = 2021.11.0, pyarrow = 5.0.0),
        webhdfs (fsspec = 2021.11.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2021.11.0, boto3 = 1.17.106),
        ssh (sshfs = 2021.9.0),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.3),
        webdavs (webdav4 = 0.9.3)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git

[sjawhar]

I'm running on Linux (Pop!OS 21.04), so this isn't a Mac thing. Also, just for clarity, the rm $(readlink) step is just to simulate someone trying to reproduce a step without pulling cache for upstream, frozen stages (e.g. because they are very large)