ssh tunnel Windows

[SlavaDm]

I use ssh jump and it works correctly on Linux. But when I tried to use it on Windows I got this error.

PS C:\Users\alex\OneDrive\Desktop\d> dvc push -v
2023-01-25 00:11:01,824 DEBUG: Preparing to transfer data from 'C:\Users\alex\OneDrive\Desktop\d.dvc\cache' to '/home/alex/test1'
2023-01-25 00:11:01,827 DEBUG: Preparing to collect status from '/home/alex/test1'
2023-01-25 00:11:01,828 DEBUG: Collecting status from '/home/alex/test1'
0% Querying remote cache| |0/1 [00:00<?, ?files/s]2023-01-25 00:11:01,837 DEBUG: Querying 1 oids via object_exists
2023-01-25 00:11:02,924 ERROR: unexpected error

Traceback (most recent call last):
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\cli_init_.py", line 184, in main
ret = cmd.do_run()
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\cli\command.py", line 26, in do_run
return self.run()
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\commands\data_sync.py", line 59, in run
processed_files_count = self.repo.push(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\repo_init_.py", line 66, in wrapper
return f(repo, *args, **kwargs)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\repo\push.py", line 92, in push
result = self.cloud.push(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\data_cloud.py", line 142, in push
return self.transfer(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc\data_cloud.py", line 123, in transfer
return transfer(src_odb, dest_odb, objs, **kwargs)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_data\hashfile\transfer.py", line 203, in transfer
status = compare_status(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_data\hashfile\status.py", line 178, in compare_status
dest_exists, dest_missing = status(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_data\hashfile\status.py", line 134, in status
exists = hashes.intersection(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_data\hashfile\status.py", line 55, in indexed_dir_hashes
dir_exists.update(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\tqdm\std.py", line 1195, in iter
for obj in iterable:
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_objects\db.py", line 355, in list_oids_exists
in_remote = self.fs.exists(paths, batch_size=jobs)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_objects\fs\base.py", line 338, in exists
if self.fs.async_impl:
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\funcy\objects.py", line 50, in get
return prop.get(instance, type)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\funcy\objects.py", line 28, in get
res = instance.dict[self.fget.name] = self.fget(instance)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\dvc_ssh_init.py", line 116, in fs
return _SSHFileSystem(**self.fs_args)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\fsspec\spec.py", line 76, in call
obj = super().call(*args, **kwargs)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\sshfs\spec.py", line 65, in init
self._client, self._pool = self.connect(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\fsspec\asyn.py", line 111, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\fsspec\asyn.py", line 96, in sync
raise return_result
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\fsspec\asyn.py", line 53, in _runner
result[0] = await coro
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\asyncio\tasks.py", line 445, in wait_for
return fut.result()
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\sshfs\utils.py", line 27, in wrapper
return await func(*args, **kwargs)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\sshfs\spec.py", line 82, in _connect
client = await self._stack.enter_async_context(_raw_client)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 619, in enter_async_context
result = await _cm_type.aenter(cm)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\asyncssh\misc.py", line 274, in aenter
self._coro_result = await self._coro
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\asyncssh\connection.py", line 7834, in connect
return await asyncio.wait_for(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\asyncio\tasks.py", line 408, in wait_for
return await fut
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\asyncssh\connection.py", line 433, in _connect
conn = await _open_proxy(loop, proxy_command, conn_factory)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\site-packages\asyncssh\connection.py", line 352, in _open_proxy
_, tunnel = await loop.subprocess_exec(_ProxyCommandTunnel, *command)
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 1670, in subprocess_exec
transport = await self._make_subprocess_transport(
File "C:\Users\alex\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError

2023-01-25 00:11:04,507 DEBUG: link type reflink is not available ([Errno 129] no more link types left to try out)
2023-01-25 00:11:04,512 DEBUG: Removing 'C:\Users\alex\OneDrive\Desktop.JPa2bpLTWUJ7v62VC8BupR.tmp'
2023-01-25 00:11:04,519 DEBUG: Removing 'C:\Users\alex\OneDrive\Desktop.JPa2bpLTWUJ7v62VC8BupR.tmp'
2023-01-25 00:11:04,523 DEBUG: link type symlink is not available ([WinError 1314] The client does not have the required rights: 'C:/Users/alex/OneDrive/Desktop/d/.dvc/cache/.gptZVWp6nogaCn6mo8Rh85.tmp' -> 'C:/Users/alex/OneDrive/Desktop/.JPa2bpLTWUJ7v62VC8BupR.tmp')
2023-01-25 00:11:04,532 DEBUG: Removing 'C:\Users\alex\OneDrive\Desktop.JPa2bpLTWUJ7v62VC8BupR.tmp'
2023-01-25 00:11:04,535 DEBUG: Removing 'C:\Users\alex\OneDrive\Desktop\d.dvc\cache.gptZVWp6nogaCn6mo8Rh85.tmp'
2023-01-25 00:11:04,552 DEBUG: Version info for developers:
DVC version: 2.43.1 (pip)

Platform: Python 3.10.8 on Windows-10-10.0.19045-SP0
Subprojects:
dvc_data = 0.35.1
dvc_objects = 0.19.0
dvc_render = 0.0.17
dvc_task = 0.1.11
dvclive = 1.3.3
scmrepo = 0.1.6
Supports:
gdrive (pydrive2 = 1.14.0),
http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
ssh (sshfs = 2023.1.0)
Cache types: hardlink
Cache directory: NTFS on C:
Caches: local
Remotes: ssh
Workspace directory: NTFS on C:
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-01-25 00:11:04,586 DEBUG: Analytics is enabled.
2023-01-25 00:11:05,595 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\Users\alex\AppData\Local\Temp\tmpfl4lgwuh']'
2023-01-25 00:11:05,604 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\Users\alex\AppData\Local\Temp\tmpfl4lgwuh']'

dvc doctor
PS C:\Users\alex\OneDrive\Desktop\d> dvc doctor
DVC version: 2.43.1 (pip)

Platform: Python 3.10.8 on Windows-10-10.0.19045-SP0
Subprojects:
dvc_data = 0.35.1
dvc_objects = 0.19.0
dvc_render = 0.0.17
dvc_task = 0.1.11
dvclive = 1.3.3
scmrepo = 0.1.6
Supports:
gdrive (pydrive2 = 1.14.0),
http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
ssh (sshfs = 2023.1.0)
Cache types: hardlink
Cache directory: NTFS on C:
Caches: local
Remotes: ssh
Workspace directory: NTFS on C:
Repo: dvc, git

If I use ssh without jump it works on Windows.

ssh jump:

Host remoteserver
HostName ip_main_server
User user

ProxyJump user@ip_jumpserver
ProxyCommand ssh -W %h:%p ip_jumpserver

I can write in command line ssh remoteserver and It will connect to my server. But It does not work on Windows+dvc (Linux+dvc works correctly)

At first I wrote to dvc developers in discord and they said:
" Looks like you've hit something interesting on windows https://stackoverflow.com/questions/44633458/why-am-i-getting-notimplementederror-with-async-and-await-on-windows " and " you might want to open an issue in https://github.com/ronf/asyncssh/issues (this is a library that we are using) ".

[ronf]

It looks like you are trying to use the ProxyCommand feature here. Unfortunately, that relies on the asyncio method subprocess_exec, which has some limitations on Windows.

Do you know what kind of event loop you are using? According to https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess, this is implemented only for the ProactorEventLoop. This is supposedly the default, but I'm wondering if maybe your application (or the dvc library) is using a SelectorEventLoop.

Do you need to use ProxyCommand here, or would ProxyJump be sufficient? That would avoid running into this issue.

[efiop]

Hey @ronf , thanks for a quick response as usual 🙏

We don't seem to set anything explicitly in dvc or sshfs so I was quick to jump to a conclusion that this was happening somewhere in asyncssh, but I was wrong and I think I found a culprit https://github.com/fsspec/filesystem_spec/blob/012816bb142de1e507423b81dc3b2c925d4424c3/fsspec/asyn.py#L128 and from git blame fsspec/filesystem_spec#656 Looks like we'll need to do a little bit more digging there.

We can safely close this issue and move discussion to fsspec/sshfs#29

Sorry for the noise and thank you.

[ronf]

No problem. That call to request the Selector event loop would explain this. Thanks for following up with the details!