ProxyJump not working on 17.0.0

[tjim]

Checklist

• Using latest version on the App Store
• Read the docs
• Searched for existing GitHub issues

Configuration

Just updated to blink 17.0.0, iPad OS 16.1.1

Describe the bug

I have configured host XXXXX to use host YYYYY as ProxyJump. In the previous version of blink I was able to

ssh XXXXX

In the new version 17.0.0, this hangs. When I ssh -vvvvv XXXXX, the debug message just before the hang is

ssh_socket_connect_proxycommand: Executing proxycommand 'ssh -W [XXXXX]:22 YYYYY'

I am able to log in to YYYYY fine with ‘ssh YYYYY’.

I am able to log in to XXXXX using YYYYY as a jump host in iSH on the same iPad (ssh -J YYYYY XXXXX).

Possibly related: ssh-agent is not working.

blink> ssh-agent
Failed loading ssh_agent_main from MAIN, cause = dlsym(RTLD_MAIN_ONLY, ssh_agent_main): symbol not found
ssh-agent: command not found

[carloscabanero]

These are usually configuration issues. Could you send the .blink/ssh_config file? And also, the full output from -vvvvv? I will try to replicate from my side.

Previously discussed here #1840

[carloscabanero]

Oh, forgot to mention. The issue with ssh-agent is a rare issue with ios_system "executing" binaries in Blink. Usually restarting the app should do the trick. That may be what is causing the issue itself as there is another process in the background for the ProxyCommand, so worth trying.

[tjim]

I restarted the app, did not fix ssh-agent issue. Rebooted the iPad, this also did not fix ssh-agent (results in the same behavior as above).

[carloscabanero]

Sorry for the confusion. I thought you meant ssh-add. ssh-agent runs by default and is disabled, you don’t need to execute it yourself. Are you adding keys to the agent for log in through the proxy? I would be more able able to help you with the information from ssh -vvvvv and the blink/ssh_config file. If sending here is a problem, please do it to support at blink.sh

[tjim]

OK, here is config, I have had to rename some hosts to protect the innocent:

# ATTENTION! THIS IS GENERATED FILE. DO NOT CHANGE IT DIRECTLY.
# GENERATED 2023-09-20T17:29:43Z
#
# Use config command do configure your hosts
# Or put your configuration to ~/.ssh/config

Host YYYYY
  User tjim
  HostName BBBBB
  IdentityFile id_rsa
  ForwardAgent yes

Host XXXXX
  User tjim
  HostName AAAAA
  IdentityFile id_rsa
  ProxyJump YYYYY

Full output from -vvvvv:

blink> ssh -vvvvv XXXXX
Setting up connection callbacks.
Starting connection to AAAAA
ssh_connect: libssh 0.9.3 (c) 2003-2019 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
ssh_socket_connect_proxycommand: Executing proxycommand 'ssh -W [AAAAA]:22 YYYYY'

At that point it hangs.

The ssh-agent problem only came up because I was trying to debug, I don’t usually invoke it. (I was trying to log in manually to the jump host YYYYY, then ssh from there to XXXXX.) In other words, I get the behavior above if ssh XXXXX is the first command I issue to blink on startup.

[carloscabanero]

Ok, I will try and reproduce it. One more question from my side.

If you first connect to YYYYY and then in another tab connect to XXXXX, does that go through?

Weird that this broke on iOS17 for you, I think there must be something very strange going on.

[tjim]

If I first connect to YYYYY and then in another tab connect to XXXXX, it DOES NOT go through.

Sounds like ssh-agent is a red herring. I suggest that you remove it from completions (that is, when I type ‘ssh-a’ it shows as one of the possible completions), otherwise it may be confusing for someone used to Linux.

If I ssh-add, then ssh YYYYY, then I can ssh XXXXX.

I don’t know if there is something strange on my system; I will say that ProxyJump did not work for me until 16.0, but it seemed to work for a lot of other people before that. Previously I had to ssh-add, ssh YYYYY, then ssh XXXXX, which is what is working for me now.

[carloscabanero]

Just released 17.0.1. Let us know

[tjim]

Behavior is different but still unable to connect.

v17.0.1.840

blink> ssh -vvvvv XXXXX
Setting up connection callbacks.
Starting connection to AAAAA
ssh_connect: libssh 0.9.3 (c) 2003-2019 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
ssh_socket_connect_proxycommand: Executing proxycommand 'ssh -W [AAAAA]:22 YYYYY'
ssh_socket_connect_proxycommand: ProxyCommand connection pipe: [58,59]
socket_callback_connected: Socket connection callback: 1 (0)
ssh_connect: Socket connecting, now waiting for the callbacks to work
ssh_connect: current state : 2
Starting connection to AAAAA
ssh_connect: current state : 2
Starting connection to AAAAA
ssh_connect: current state : 2
ssh_socket_exception_callback: Socket exception callback: 1 (0)
ssh_socket_exception_callback: Socket error: disconnected
connError(msg: "Session Exception - Socket error: disconnected")
Starting connection to AAAAA
ssh_connect: current state : 9
Error connecting to XXXXX. connError(msg: "Socket error: disconnected")
Session deinit

As before, I am able to ssh-add, ssh YYYYY, and from there ssh XXXXX, it is just ProxyJump that is not working.

Also, it still does not work if I am logged in to YYYYY in another tab.

I have seen this kind of behavior on blink versions < 16, when it seemed like ProxyJump was working for others but not me.

[carloscabanero]

Yeah, but we can see more now. The other side is disconnecting when we do the stdio (-W) tunnel. We haven’t touched this from 16 to 17 so it is weird that this worked for you before. We can figure it out though, just need you to run a couple quick tests for me, if you still have the patience:

• In a different shell, run 'ssh -W [AAAAA]:22 YYYYY'. It should stay connected and blank. If it disconnects, then add -vvvvv so we can see what is going on.
• Does the Files.app integration work? We use a different method to do the jump and that would point me on a direction for the fix.

Thank you!

[tjim]

This is working now! I did a reboot, that may have done something. Files.app also works.
Thanks!

[carloscabanero]

Ok, I will keep an eye on this one in case it repeats. I understand you did a reboot of the device and not the server, right?

[yury]

Tracked down original issue and made PR