SSH: ValueError: Can't create any SFTP connections!

[gcoter]

Bug Report

I open this issue as a follow up to iterative/dvc#6138

Description

dvc push raises an error when trying to push to an SFTP remote. It used to work with older versions. The SFTP remote I use is a personal Raspberry Pi server. I did not change anything on the server.

Reproduce

Unfortunately, since I use a private server, I don't know whether it would be easy to reproduce.

After updating dvc, I tried to run dvc push and I got these logs:

$ dvc push -v -a
2021-10-03 11:59:51,299 DEBUG: Preparing to transfer data from '.dvc/cache' to 'ssh://<REMOTE URL>/gcoter/music-generation-v2.dvc'
2021-10-03 11:59:51,300 DEBUG: Preparing to collect status from 'ssh://<REMOTE URL>/gcoter/music-generation-v2.dvc'
2021-10-03 11:59:51,318 DEBUG: Collecting status from 'ssh://<REMOTE URL>/gcoter/music-generation-v2.dvc'
2021-10-03 11:59:51,918 DEBUG: Querying 38 hashes via object_exists
2021-10-03 12:00:30,881 ERROR: unexpected error - Can't create any SFTP connections!                                                                                         
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/command/data_sync.py", line 57, in run
    processed_files_count = self.repo.push(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/repo/push.py", line 48, in push
    pushed += self.cloud.push(obj_ids, jobs, remote=remote, odb=odb)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/data_cloud.py", line 85, in push
    return transfer(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/transfer.py", line 221, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 160, in compare_status
    dest_exists, dest_missing = status(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 36, in _indexed_dir_hashes
    indexed_dir_exists.update(odb.list_hashes_exists(indexed_dirs))
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 420, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 411, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 96, in exists
    return self.fs.exists(self._with_bucket(path_info))
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 480, in _exists
    await self._info(path)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/sshfs/spec.py", line 135, in _info
    async with self._pool.get() as channel:
  File "/usr/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/sshfs/pools/soft.py", line 38, in get
    raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!
------------------------------------------------------------
2021-10-03 12:00:31,971 DEBUG: Version info for developers:
DVC version: 2.7.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-88-generic-x86_64-with-glibc2.29
Supports:
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	ssh (sshfs = 2021.9.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda5
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda5
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-10-03 12:00:31,973 DEBUG: Analytics is enabled.
2021-10-03 12:00:32,040 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpkmytybwl']'
2021-10-03 12:00:32,043 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpkmytybwl']'

Expected

Since I did not change the configuration of my server and it used to work, I would expect dvc push to work.

Environment information

$ dvc doctor
DVC version: 2.7.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-89-generic-x86_64-with-glibc2.29
Supports:
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        ssh (sshfs = 2021.9.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda5
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda5
Repo: dvc, git

[efiop]

@gcoter Maybe you could try to reproduce it with a docker image with the same config options? So far we are not able to reproduce ourselves.

[gcoter]

@efiop Yes it is a good idea, actually I deployed the SFTP server as a docker container on my Raspberry Pi. However, the image I used has been built for ARM. I will try to reproduce the error locally on my computer with the original docker image.

[sjawhar]

I created a quick SSH server in a Docker container, mounted my DVC cache into it, and set it as the remote for my local project. Running dvc pull didn't result in an error, but it did open over 1000 connections (according to netstat -tn). Is that expected?

[sjawhar]

Also potentially relevant: when I get this error, it is always during the step of querying the remote cache. If I retry enough times and get past that step, then the actual up/downloading always succeeds.

[karajan1001]

I created a quick SSH server in a Docker container, mounted my DVC cache into it, and set it as the remote for my local project. Running dvc pull didn't result in an error, but it did open over 1000 connections (according to netstat -tn). Is that expected?

Do you set any --job related config and how many cores are there in your computer.

[sjawhar]

Do you set any --job related config and how many cores are there in your computer.

No, I didn't use the --job flag. 8 cores, 16 threads.

[karajan1001]

Looks like we need to add some limitations on the status query.

[gcoter]

Hi all, sorry for my late response! As I was about to try to reproduce this issue locally (as proposed by @efiop), I upgraded dvc to the last version (2.9.2) and now it works 🙂

@sjawhar Maybe you can try it and confirm whether it solves the issue for you as well?

[sjawhar]

Unfortunately still an issue on 2.9.3

$ dvc pull --verbose --recursive pipelines/finger_tapping/
2021-12-29 21:15:54,657 DEBUG: Adding '/home/user/app/.dvc/config.local' to gitignore file.
2021-12-29 21:15:54,679 DEBUG: Adding '/home/user/app/.dvc/tmp' to gitignore file.
2021-12-29 21:15:54,679 DEBUG: Adding '/home/user/app/.dvc/cache' to gitignore file.
2021-12-29 21:15:54,687 DEBUG: Checking if stage 'pipelines/finger_tapping/' is in 'dvc.yaml'
2021-12-29 21:15:55,608 DEBUG: Preparing to transfer data from '/usr/data/project/dvc' to '/home/user/app/.dvc/cache'
2021-12-29 21:15:55,608 DEBUG: Preparing to collect status from '/home/user/app/.dvc/cache'
2021-12-29 21:15:55,619 DEBUG: Collecting status from '/home/user/app/.dvc/cache'
2021-12-29 21:15:55,735 DEBUG: Preparing to collect status from '/usr/data/project/dvc'                                                                                                                                                                                                                                                                                                                                   
2021-12-29 21:15:55,740 DEBUG: Collecting status from '/usr/data/project/dvc'
2021-12-29 21:15:55,856 DEBUG: Querying 126 hashes via object_exists
2021-12-29 21:16:14,510 ERROR: unexpected error - Can't create any SFTP connections!                                                                                                                                                                                                                                                                                                                                                              
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/command/data_sync.py", line 30, in run
    stats = self.repo.pull(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/pull.py", line 29, in pull
    processed_files_count = self.fetch(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 67, in fetch
    d, f = _fetch(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 87, in _fetch
    downloaded += repo.cloud.pull(obj_ids, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/data_cloud.py", line 114, in pull
    return transfer(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 164, in compare_status
    src_exists, src_missing = status(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 36, in _indexed_dir_hashes
    indexed_dir_exists.update(odb.list_hashes_exists(indexed_dirs))
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 421, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 412, in exists_with_progress
    ret = self.fs.exists(fs_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 91, in exists
    return self.fs.exists(path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 549, in _exists
    await self._info(path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/spec.py", line 135, in _info
    async with self._pool.get() as channel:
  File "/usr/local/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/pools/soft.py", line 38, in get
    raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!
------------------------------------------------------------
2021-12-29 21:16:14,984 DEBUG: Adding '/home/user/app/.dvc/config.local' to gitignore file.
2021-12-29 21:16:14,990 DEBUG: Adding '/home/user/app/.dvc/tmp' to gitignore file.
2021-12-29 21:16:14,990 DEBUG: Adding '/home/user/app/.dvc/cache' to gitignore file.
2021-12-29 21:16:14,991 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 18] Invalid cross-device link
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/command/data_sync.py", line 30, in run
    stats = self.repo.pull(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/pull.py", line 29, in pull
    processed_files_count = self.fetch(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 67, in fetch
    d, f = _fetch(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 87, in _fetch
    downloaded += repo.cloud.pull(obj_ids, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/data_cloud.py", line 114, in pull
    return transfer(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 164, in compare_status
    src_exists, src_missing = status(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 36, in _indexed_dir_hashes
    indexed_dir_exists.update(odb.list_hashes_exists(indexed_dirs))
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 421, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 412, in exists_with_progress
    ret = self.fs.exists(fs_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 91, in exists
    return self.fs.exists(path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 549, in _exists
    await self._info(path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/spec.py", line 135, in _info
    async with self._pool.get() as channel:
  File "/usr/local/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/pools/soft.py", line 38, in get
    raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/local.py", line 148, in reflink
    System.reflink(from_info, to_info)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/system.py", line 112, in reflink
    System._reflink_linux(source, link_name)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/system.py", line 96, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 18] Invalid cross-device link

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 32, in _link
    raise OSError(
OSError: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 77, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2021-12-29 21:16:14,992 DEBUG: Removing '/home/user/.VxePWiE728u2v5gnSQT3vY.tmp'
2021-12-29 21:16:14,992 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'hardlink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 18] Invalid cross-device link: '/home/user/app/.dvc/cache/.RhJXijpmS46m4MKMHUFkXk.tmp' -> '/home/user/.VxePWiE728u2v5gnSQT3vY.tmp'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/command/data_sync.py", line 30, in run
    stats = self.repo.pull(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/pull.py", line 29, in pull
    processed_files_count = self.fetch(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 67, in fetch
    d, f = _fetch(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 87, in _fetch
    downloaded += repo.cloud.pull(obj_ids, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/data_cloud.py", line 114, in pull
    return transfer(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 164, in compare_status
    src_exists, src_missing = status(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 36, in _indexed_dir_hashes
    indexed_dir_exists.update(odb.list_hashes_exists(indexed_dirs))
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 421, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 412, in exists_with_progress
    ret = self.fs.exists(fs_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 91, in exists
    return self.fs.exists(path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 549, in _exists
    await self._info(path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/spec.py", line 135, in _info
    async with self._pool.get() as channel:
  File "/usr/local/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/sshfs/pools/soft.py", line 38, in get
    raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/local.py", line 141, in hardlink
    System.hardlink(from_info, to_info)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/system.py", line 39, in hardlink
    os.link(src, link_name)
OSError: [Errno 18] Invalid cross-device link: '/home/user/app/.dvc/cache/.RhJXijpmS46m4MKMHUFkXk.tmp' -> '/home/user/.VxePWiE728u2v5gnSQT3vY.tmp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 32, in _link
    raise OSError(
OSError: [Errno 95] 'hardlink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/user/.cache/pypoetry/virtualenvs/app-gcEbEm5J-py3.8/lib/python3.8/site-packages/dvc/fs/utils.py", line 77, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2021-12-29 21:16:14,993 DEBUG: Removing '/home/user/.VxePWiE728u2v5gnSQT3vY.tmp'
2021-12-29 21:16:14,993 DEBUG: Removing '/home/user/.VxePWiE728u2v5gnSQT3vY.tmp'
2021-12-29 21:16:14,993 DEBUG: Removing '/home/user/app/.dvc/cache/.RhJXijpmS46m4MKMHUFkXk.tmp'
2021-12-29 21:16:15,000 DEBUG: Version info for developers:
DVC version: 2.9.3 (pip)
---------------------------------
Platform: Python 3.8.8 on Linux-5.15.8-76051508-generic-x86_64-with-glibc2.2.5
Supports:
        hdfs (fsspec = 2021.11.1, pyarrow = 4.0.1),
        webhdfs (fsspec = 2021.11.1),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        ssh (sshfs = 2021.11.2)
Cache types: symlink
Cache directory: ext4 on /dev/mapper/data-root
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/data-root
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-12-29 21:16:15,002 DEBUG: Analytics is enabled.
2021-12-29 21:16:15,085 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpiyk0nprz']'
2021-12-29 21:16:15,088 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpiyk0nprz']'

[efiop]

@sjawhar Could you try --jobs 1?

Also, how many dvc files do you have in pipelines/finger_tapping/? Could you run

find pipelines/finger_tapping -type f -name '*.dvc' | wc -l

[mistermult]

I had the same error. Unfortunately, it is hard to replicate:

• host1with no (or almost no) files to push: dvc push in repository root - no error
• host2: dvc push in repository root on - Error
• host2: Called dvc push --recursive . in each subdirectory (500 files/500GB total; 30 files/100GB total; 1-5 files/50GB). Files hat sizes up to 90GB - No Error
• host2: dvc push in repository root on after doing it in each subdirectory (see above) - no error anymore

I also have directories with tens of thousands of files in the repository. But they where pushed in the past and there was no error.

@gcoter @sjawhar Maybe you can try it with an much older version, e.g. 1.2.x. In the past, I did exactly the same steps that resulted in this error with the newest version. With 1.2.x it worked in the past.