Skip to content

'dvc add dir' fails when 'dir' contains a file and a relative symlink on this file  #4160

@CRodVig

Description

@CRodVig

Bug Report

$ dvc version -v
DVC version: 1.1.3
Python version: 3.6.9
Platform: Linux-5.3.0-1028-aws-x86_64-with-Ubuntu-18.04-bionic
Binary: False
Package: pip
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache: reflink - not supported, hardlink - supported, symlink - supported
Repo: dvc, git

I wanted to add a directory containing binaries on Linux. For dynamic libraries, we often have the "real" file as libXXX.A.B.C.so and some symlinks of the form libXXX.A.B.so, libXXX.A.so and libXXX.so. But adding the directory failed. After investigations, I found a simple way to reproduce this bug. Hope this will help:

mkdir bug
cd bug
git init
dvc init
mkdir test
echo "content" > test/file.txt.1
ln -s ./file.txt.1 test/file.txt
dvc add test # It fails
Adding...
ERROR: unexpected error - [Errno 2] No such file or directory: './file.txt.1'

As you can see, the file that is not found has exactly the same path than the one stored in the symlink. So I tested to put an absolute path in the symlink:

[...]
ln -s /absolute/path/to/test/file.txt.1 test/file.txt
[...]

And in this case it works! Hope this will help you.

Output of dvc add --verbose test:

ERROR: unexpected error - [Errno 2] No such file or directory: 'file.txt.1'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/main.py", line 53, in main
    ret = cmd.run()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/command/add.py", line 22, in run
    external=self.args.external,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/repo/__init__.py", line 36, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/repo/add.py", line 96, in add
    stage.commit()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/stage/decorators.py", line 35, in rwlocked
    return call()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 416, in commit
    out.commit()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/output/base.py", line 285, in commit
    self.cache.save(self.path_info, self.cache.tree, self.info)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/remote/base.py", line 1161, in save
    return self._save(path_info, tree, hash_, save_link, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/remote/base.py", line 1168, in _save
    return self._save_dir(path_info, tree, hash_, save_link, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/remote/base.py", line 1140, in _save_dir
    entry_info, tree, entry_hash, save_link=False, **kwargs
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/remote/base.py", line 1077, in _save_file
    self.tree.move(path_info, cache_info, mode=self.CACHE_MODE)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/remote/local.py", line 138, in move
    move(from_info, to_info, mode=mode)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dvc/utils/fs.py", line 100, in move
    shutil.copy(os.readlink(os.fspath(src)), tmp)
  File "/usr/lib/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'file.txt.1'
------------------------------------------------------------

Metadata

Metadata

Assignees

Labels

bugDid we break something?p2-mediumMedium priority, should be done, but less importantresearch

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions