exp run: cannot clean up temp directory runs on Linux + NFS

[karajan1001]

Hello,

I am having this issue still. When I check the pack directories, one is empty and one has files in it:

(almds_dl) [starrgw1@login01 dvctest]$ ls .dvc/tmp/exps/tmpm5ix9f4j/.git/objects/pack/
pack-1186ca2730fcc6628a18da2631e206dc5d09791b.idx  pack-1186ca2730fcc6628a18da2631e206dc5d09791b.pack
(almds_dl) [starrgw1@login01 dvctest]$ ls .dvc/tmp/exps/tmpb02b0lgm/.git/objects/pack/
(almds_dl) [starrgw1@login01 dvctest]$

Below is the debug output:

(almds_dl) [starrgw1@login01 dvctest]$ dvc exp run --run-all -j 2 -v
2022-03-10 19:21:22,130 DEBUG: Reproducing experiment revs '96d579e, 03eef69'
2022-03-10 19:21:22,195 DEBUG: Writing experiments local config '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.dvc/config.local'
2022-03-10 19:21:22,195 DEBUG: Init temp dir executor in '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm'
2022-03-10 19:21:22,227 DEBUG: Writing experiments local config '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/.dvc/config.local'
2022-03-10 19:21:22,227 DEBUG: Init temp dir executor in '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j'
2022-03-10 19:21:22,337 DEBUG: Running repro in '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j'
2022-03-10 19:21:22,338 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/.dvc/tmp/repro.dat'
2022-03-10 19:21:22,338 DEBUG: Running repro in '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm'
2022-03-10 19:21:22,338 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.dvc/tmp/repro.dat'
2022-03-10 19:21:22,502 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,502 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,503 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,503 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,506 DEBUG: Dependency 'params.yaml' of stage: 'stage1' changed because it is '{'input_text': 'modified'}'.
2022-03-10 19:21:22,506 DEBUG: Dependency 'params.yaml' of stage: 'stage1' changed because it is '{'input_text': 'modified'}'.
2022-03-10 19:21:22,506 DEBUG: stage: 'stage1' changed.
2022-03-10 19:21:22,506 DEBUG: stage: 'stage1' changed.
2022-03-10 19:21:22,508 DEBUG: Removing output 'metrics.json' of stage: 'stage1'.
2022-03-10 19:21:22,508 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/metrics.json'
2022-03-10 19:21:22,508 DEBUG: Removing output 'metrics.json' of stage: 'stage1'.
2022-03-10 19:21:22,509 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/metrics.json'
2022-03-10 19:21:22,511 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,511 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,514 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,514 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,514 DEBUG: {}
2022-03-10 19:21:22,515 DEBUG: {}
2022-03-10 19:21:22,516 DEBUG: defaultdict(<class 'dict'>, {'params.yaml': {'input_text': 'modified'}})
2022-03-10 19:21:22,516 DEBUG: defaultdict(<class 'dict'>, {'params.yaml': {'input_text': 'modified'}})
2022-03-10 19:21:22,517 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,518 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,520 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,520 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,523 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:22,523 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
Running stage 'stage1':
Running stage 'stage1':
> python submit_job.py stage1.py
> python submit_job.py stage1.py
/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/stage1.py
/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/stage1.py
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='Q', queue='short')
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
JobStatus(job_id='120799.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
JobStatus(job_id='120800.vectivus.cm.cluster', name='qsub_script.sh', user='starrgw1', time_use='0', status='R', queue='short')
2022-03-10 19:21:58,158 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:58,162 DEBUG: state save (2286788293, 1646958103056720896, 8) 1166a8fbe4acb9cbfd182cfbb5fd9fdf
2022-03-10 19:21:58,162 DEBUG: state save (2286788293, 1646958103056720896, 8) 1166a8fbe4acb9cbfd182cfbb5fd9fdf
2022-03-10 19:21:58,163 DEBUG: Output 'metrics.json' doesn't use cache. Skipping saving.
2022-03-10 19:21:58,164 DEBUG: Computed stage: 'stage1' md5: '8aa9486314f0f8befb9277b8eb4e8def'
2022-03-10 19:21:58,165 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:58,168 DEBUG: state save (2286788291, 1646958082216025344, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:58,176 DEBUG: state save (2286788293, 1646958103056720896, 8) 1166a8fbe4acb9cbfd182cfbb5fd9fdf
2022-03-10 19:21:58,178 DEBUG: Preparing to transfer data from 'memory://dvc-staging/d701057ee2ce3fdcd5408c3336f74eedf76e8d2655257ef739b3ea22a3904799' to '/home/starrgw1/code/dvctest/.dvc/cache'
2022-03-10 19:21:58,178 DEBUG: Preparing to collect status from '/home/starrgw1/code/dvctest/.dvc/cache'
2022-03-10 19:21:58,178 DEBUG: Collecting status from '/home/starrgw1/code/dvctest/.dvc/cache'
2022-03-10 19:21:58,179 DEBUG: Preparing to collect status from 'memory://dvc-staging/d701057ee2ce3fdcd5408c3336f74eedf76e8d2655257ef739b3ea22a3904799'
2022-03-10 19:21:58,182 DEBUG: state save (2286788293, 1646958103056720896, 8) 1166a8fbe4acb9cbfd182cfbb5fd9fdf
2022-03-10 19:21:58,187 DEBUG: Uploading '/home/starrgw1/code/dvctest/.dvc/cache/.J4vv5NDtBQtjSh8ycSkWhC.tmp' to '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/.SnbjvSEZ76cQkVrwvWcLN6.tmp'
2022-03-10 19:21:58,189 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/.SnbjvSEZ76cQkVrwvWcLN6.tmp'
2022-03-10 19:21:58,189 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/cache/.J4vv5NDtBQtjSh8ycSkWhC.tmp'
2022-03-10 19:21:58,197 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/metrics.json'
2022-03-10 19:21:58,198 DEBUG: Uploading '/home/starrgw1/code/dvctest/.dvc/cache/11/66a8fbe4acb9cbfd182cfbb5fd9fdf' to '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpm5ix9f4j/metrics.json'
2022-03-10 19:21:58,200 DEBUG: state save (2286788296, 1646958118198226432, 8) 1166a8fbe4acb9cbfd182cfbb5fd9fdf
2022-03-10 19:21:58,205 DEBUG: state save (2286788296, 1646958118198226432, 8) 1166a8fbe4acb9cbfd182cfbb5fd9fdf
2022-03-10 19:21:58,210 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:58,211 DEBUG: stage: 'stage1' was reproduced
2022-03-10 19:21:58,213 DEBUG: state save (1211851840, 1646958108677908736, 8) 9b7916dcfbccc49c18581fd80884fa56
2022-03-10 19:21:58,214 DEBUG: state save (1211851840, 1646958108677908736, 8) 9b7916dcfbccc49c18581fd80884fa56
2022-03-10 19:21:58,215 DEBUG: Output 'metrics.json' doesn't use cache. Skipping saving.
2022-03-10 19:21:58,216 DEBUG: Computed stage: 'stage1' md5: '7eb88065797b13721ef3ffa7cf7b13ed'
Updating lock file 'dvc.lock'
2022-03-10 19:21:58,217 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:58,220 DEBUG: state save (1211851838, 1646958082182024192, 350) 1079e31771794bac9a75210e2ac3ffda
2022-03-10 19:21:58,224 DEBUG: Staging files: {'stage1.py', 'dvc.yaml', 'params.yaml', 'dvc.lock', 'metrics.json'}
2022-03-10 19:21:58,227 DEBUG: state save (1211851840, 1646958108677908736, 8) 9b7916dcfbccc49c18581fd80884fa56
2022-03-10 19:21:58,228 DEBUG: Preparing to transfer data from 'memory://dvc-staging/d701057ee2ce3fdcd5408c3336f74eedf76e8d2655257ef739b3ea22a3904799' to '/home/starrgw1/code/dvctest/.dvc/cache'
2022-03-10 19:21:58,228 DEBUG: Preparing to collect status from '/home/starrgw1/code/dvctest/.dvc/cache'
2022-03-10 19:21:58,228 DEBUG: Collecting status from '/home/starrgw1/code/dvctest/.dvc/cache'
2022-03-10 19:21:58,229 DEBUG: Preparing to collect status from 'memory://dvc-staging/d701057ee2ce3fdcd5408c3336f74eedf76e8d2655257ef739b3ea22a3904799'
2022-03-10 19:21:58,231 DEBUG: state save (1211851840, 1646958108677908736, 8) 9b7916dcfbccc49c18581fd80884fa56
2022-03-10 19:21:58,236 DEBUG: Uploading '/home/starrgw1/code/dvctest/.dvc/cache/.MBmG3RvyuNGA6zEyzrYayc.tmp' to '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.nDJoTijpRqfnoaZLmjd2Bk.tmp'
2022-03-10 19:21:58,238 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.nDJoTijpRqfnoaZLmjd2Bk.tmp'
2022-03-10 19:21:58,238 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/cache/.MBmG3RvyuNGA6zEyzrYayc.tmp'
2022-03-10 19:21:58,238 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/metrics.json'
2022-03-10 19:21:58,239 DEBUG: Uploading '/home/starrgw1/code/dvctest/.dvc/cache/9b/7916dcfbccc49c18581fd80884fa56' to '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/metrics.json'
2022-03-10 19:21:58,240 DEBUG: Commit to new experiment branch 'refs/exps/96/bf9d4272ad61392a913bf8f1e7f77faf69defb/exp-bb312'
2022-03-10 19:21:58,240 DEBUG: state save (1211851845, 1646958118239227648, 8) 9b7916dcfbccc49c18581fd80884fa56
2022-03-10 19:21:58,245 DEBUG: state save (1211851845, 1646958118239227648, 8) 9b7916dcfbccc49c18581fd80884fa56
2022-03-10 19:21:58,252 DEBUG: stage: 'stage1' was reproduced
Updating lock file 'dvc.lock'
2022-03-10 19:21:58,265 DEBUG: Staging files: {'stage1.py', 'dvc.yaml', 'params.yaml', 'dvc.lock', 'metrics.json'}
2022-03-10 19:21:58,268 WARNING: The following untracked files were present in the experiment directory after reproduction but will not be included in experiment commits:
        qsub_script.sh, qsub_script.sh.o120799, qsub_script.sh.e120799
2022-03-10 19:21:58,284 DEBUG: Commit to new experiment branch 'refs/exps/96/bf9d4272ad61392a913bf8f1e7f77faf69defb/exp-b936d'
2022-03-10 19:21:58,308 WARNING: The following untracked files were present in the experiment directory after reproduction but will not be included in experiment commits:
        qsub_script.sh, qsub_script.sh.o120800, qsub_script.sh.e120800
2022-03-10 19:21:58,325 DEBUG: Collected experiment '2ba86e6'.
2022-03-10 19:21:58,326 DEBUG: Removing tmpdir '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm'
2022-03-10 19:21:58,326 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm'
2022-03-10 19:21:58,337 ERROR: unexpected error - [Errno 39] Directory not empty: '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.git/objects/pack'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'pack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/cli/__init__.py", line 78, in main
    ret = cmd.do_run()
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/commands/experiments/run.py", line 32, in run
    results = self.repo.experiments.run(
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 825, in run
    return run(self.repo, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/run.py", line 28, in run
    return repo.experiments.reproduce_queued(jobs=jobs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 457, in reproduce_queued
    results = self._reproduce_revs(**kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 53, in wrapper
    return f(exp, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 644, in _reproduce_revs
    exec_results.update(self._executors_repro(manager, **kwargs))
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 64, in wrapper
    ret = f(exp, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 675, in _executors_repro
    return manager.exec_queue(self.repo, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager/base.py", line 159, in exec_queue
    return self._exec_attached(repo, jobs=jobs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager/base.py", line 232, in _exec_attached
    self.cleanup_executor(rev, executor)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager/base.py", line 270, in cleanup_executor
    executor.cleanup()
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/local.py", line 110, in cleanup
    remove(self.root_dir)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/utils/fs.py", line 135, in remove
    shutil.rmtree(path, onerror=_chmod)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/utils/fs.py", line 120, in _chmod
    func(p)
OSError: [Errno 39] Directory not empty: '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.git/objects/pack'
------------------------------------------------------------
2022-03-10 19:21:58,503 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 95] Operation not supported
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'pack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/cli/__init__.py", line 78, in main
    ret = cmd.do_run()
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/commands/experiments/run.py", line 32, in run
    results = self.repo.experiments.run(
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 825, in run
    return run(self.repo, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/run.py", line 28, in run
    return repo.experiments.reproduce_queued(jobs=jobs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 457, in reproduce_queued
    results = self._reproduce_revs(**kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 53, in wrapper
    return f(exp, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 644, in _reproduce_revs
    exec_results.update(self._executors_repro(manager, **kwargs))
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 64, in wrapper
    ret = f(exp, *args, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 675, in _executors_repro
    return manager.exec_queue(self.repo, **kwargs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager/base.py", line 159, in exec_queue
    return self._exec_attached(repo, jobs=jobs)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager/base.py", line 232, in _exec_attached
    self.cleanup_executor(rev, executor)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager/base.py", line 270, in cleanup_executor
    executor.cleanup()
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/repo/experiments/executor/local.py", line 110, in cleanup
    remove(self.root_dir)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/utils/fs.py", line 135, in remove
    shutil.rmtree(path, onerror=_chmod)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/utils/fs.py", line 120, in _chmod
    func(p)
OSError: [Errno 39] Directory not empty: '/home/starrgw1/code/dvctest/.dvc/tmp/exps/tmpb02b0lgm/.git/objects/pack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/fs/local.py", line 144, in reflink
    System.reflink(from_info, to_info)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/system.py", line 112, in reflink
    System._reflink_linux(source, link_name)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/system.py", line 96, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 95] Operation not supported

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/fs/utils.py", line 32, in _link
    raise OSError(
OSError: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/starrgw1/.conda/envs/almds_dl/lib/python3.8/site-packages/dvc/fs/utils.py", line 77, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-03-10 19:21:58,504 DEBUG: Removing '/home/starrgw1/code/.PHQHFmRzwbSbJrd69xLnsE.tmp'
2022-03-10 19:21:58,504 DEBUG: Removing '/home/starrgw1/code/.PHQHFmRzwbSbJrd69xLnsE.tmp'
2022-03-10 19:21:58,504 DEBUG: Removing '/home/starrgw1/code/.PHQHFmRzwbSbJrd69xLnsE.tmp'
2022-03-10 19:21:58,505 DEBUG: Removing '/home/starrgw1/code/dvctest/.dvc/cache/.C7nbLHG6hDepE3gwJDP8Jb.tmp'
2022-03-10 19:21:58,585 DEBUG: Version info for developers:
DVC version: 2.9.4 (conda)
---------------------------------
Platform: Python 3.8.12 on Linux-3.10.0-693.el7.x86_64-x86_64-with-glibc2.10
Supports:
        webhdfs (fsspec = 2022.2.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: hardlink, symlink
Cache directory: nfs on master:/home
Caches: local
Remotes: None
Workspace directory: nfs on master:/home
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-03-10 19:21:58,586 DEBUG: Analytics is enabled.
2022-03-10 19:21:58,618 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpq2t5205u']'
2022-03-10 19:21:58,619 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpq2t5205u']'

Originally posted by @gregstarr in #5641 (comment)

[pmrowla]

This is the same underlying problem as #5641. I looked into this some more and the existing workaround probably won't work for us now that we use a lot more pygit functionality. scmrepo will need some updates to make the behavior safer when we are using mixed git backends.

[gregstarr]

Thanks for the response. Can you think of any temporary workaround for me to use?

[karajan1001]

Returning back to a previous version might work, but #5641 was solved long time ago, not recommend to roll back to such an old version

[gregstarr]

Is this bug caused by a change in a dependency? Or by DVC development? Do you have an idea of which change caused the bug?

This is the same underlying problem as #5641. I looked into this some more and the existing workaround probably won't work for us now that we use a lot more pygit functionality. scmrepo will need some updates to make the behavior safer when we are using mixed git backends.

Would not using mixed git backends help?

[pmrowla]

There's not really a quick workaround for this issue right now, it's due to a lot of DVC changes that have been added in the last year since #5641 was addressed.

Would not using mixed git backends help?

The short answer is "yes", but the long answer is "it's more complicated than that", as we need different functionality from each of the backends we use (since each backend does not support the same featureset).

Since this is a regression, this issue is something that we will prioritize and hopefully will have a fix for it within the next couple of weeks.

[gregstarr]

Ok gotcha. Is there anything I can do to help?

[gregstarr]

I haven't tried any others, but it looks like at least 2.5.4 does not have this issue

[gregstarr]

2.9.3 is the first version which has this issue, it also looks like this version updates scmrepo from 0.0.4 to 0.0.7