SSH ProxyCommand timing to Cisco devices

[ktbyers]

I am trying to use an SSH Proxy with Cisco IOS networking devices (and other networking vendors). I have verified that my SSH config parsing is working correctly. I have also verified that I can manually use the ProxyCommand from bash.

I am able to get this to work with Paramiko going from linux > to linux > to linux.
I am also able to get this to work going from linux > to linux > to Juniper

But if I try to do from linux > to linux > to Cisco, then I am receiving the following:

Traceback (most recent call last):
  File "./test_proxy.py", line 46, in <module>
    net_connect = ConnectHandler(**device)
  File "build/bdist.linux-i686/egg/netmiko/ssh_dispatcher.py", line 82, in ConnectHandler
  File "build/bdist.linux-i686/egg/netmiko/base_connection.py", line 62, in __init__
  File "build/bdist.linux-i686/egg/netmiko/base_connection.py", line 159, in establish_connection
  File "build/bdist.linux-i686/egg/paramiko/client.py", line 325, in connect
  File "build/bdist.linux-i686/egg/paramiko/transport.py", line 492, in start_client
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

And in my log file I see:

DEBUG:paramiko.transport:starting thread (client mode): 0xb69fc3ecL
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_1.16.0
ERROR:paramiko.transport:Exception: Error reading SSH protocol banner
ERROR:paramiko.transport:Traceback (most recent call last):
ERROR:paramiko.transport:  File "build/bdist.linux-i686/egg/paramiko/transport.py", line 1710, in run
ERROR:paramiko.transport:    self._check_banner()
ERROR:paramiko.transport:  File "build/bdist.linux-i686/egg/paramiko/transport.py", line 1858, in _check_banner
ERROR:paramiko.transport:    raise SSHException('Error reading SSH protocol banner' + str(e))
ERROR:paramiko.transport:SSHException: Error reading SSH protocol banner
ERROR:paramiko.transport:

The Cisco side indicates it has sent the banner:

Feb 2 13:15:17.572 PST: SSH0: starting SSH control process
Feb 2 13:15:17.572 PST: SSH0: sent protocol version id SSH-2.0-Cisco-1.25

If I add a short delay before the thread starts in transport.py, then the problem is fixed.

$ diff transport.py.new transport.py
486,487d485
<         # delay starting thread for SSH proxies
<         event.wait(0.2)

I was also able to fix it by adding a short delay right before the banner is output (line 1710 in transport.py). Right before the following line:

self.packetizer.write_all(b(self.local_version + '\r\n'))

[doddsie]

This totally fixed my same issue

[bitprophet]

That is pretty bizarre, thanks for the details. My gut instinct is to make this optional/controllable somehow as a mandatory sleep/wait (even a small one) feels kinda bleh, especially since we don't (?) know why this makes things work in your situation. Also because it's a bit of a corner case - though I do wonder how many other oddball hang reports in this tracker might be due to this or similar things...

Speaking of which, what exactly is "your same issue", @doddsie? I love details! Please give me details.

[ktbyers]

Yeah, I never really liked my fix which is why I never submitted a PR on it.

[doddsie]

I am running my commands through a ssh tunnel for bastion servers and get
the same error:

ERROR:paramiko.transport:SSHException: Error reading SSH protocol banner

My config file in ~/.ssh looks like the following:
Host prodtunnel
HostName bastion1.elvs.fubar.com
DynamicForward localhost:22222
ServerAliveInterval 60
ServerAliveCountMax 3
Compression yes
KeepAlive yes

To start the tunnel I use the following:
ssh -fN prodtunnel

If there is a way for me to provide more information to you or any specific
information you need, let me know.

Certainly adding a delay to make this work is not ideal, but that is
working so I will take it!

Cheers,
Dave

On Sun, Apr 24, 2016 at 8:02 PM, Kirk Byers notifications@github.com
wrote:

Yeah, I never really liked my fix which is why I never submitted a PR on
it.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#675 (comment)

[lundybernard]

Also dealing with this issue, as it affects NetMiko.

I attempted to apply @ktbyers fix, but must be missing something. simply inserting event.wait(0.2) does not work:

try:
    try:
        event.wait(0.2)  # MONKEY_PATCH: Added delay
        self.packetizer.write_all(b(self.local_version + '\r\n'))
        self._log(DEBUG, 'Local version/idstring: %s' % self.local_version)

I added some additional details to the netmiko issue here, but there's nothing particularly new or interesting, and my problem looks exactly like what ktbyers describeds
#ktbyers/netmiko#172