Paramiko fails to connect due to greenlet error 'This operation would block forever'

[coryfklein]

Trying to create an ssh connection like so:

    host = "192.168.33.10"
    user = "vagrant"
    key = paramiko.RSAKey.from_private_key_file(os.path.expanduser(private_key))
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(host, username = user, pkey = key)

And I get the following exception:

Traceback (most recent call last):
  File "/Users/coryklein/domo/domovm/vmshell/svctop.py", line 299, in run_ssh_command
    client.connect(host, username = user, pkey = key)
  File "/Users/coryklein/src/paramiko/paramiko/client.py", line 325, in connect
    t.start_client()
  File "/Users/coryklein/src/paramiko/paramiko/transport.py", line 493, in start_client
    raise e
SSHException: Error reading SSH protocol banner('This operation would block forever', <Hub at 0x1059f44b0 select pending=0 ref=0>)

Installed paramiko from source locally and modified transport.py like so:

    def _check_banner(self):
        ... // SNIP
            except Exception as e:
                self._log(DEBUG, "Banner error " +  traceback.format_exc())  // I added this line
                raise SSHException('Error reading SSH protocol banner' + str(e))

And got this stack as the cause of the error:

  File "/Users/coryklein/src/paramiko/paramiko/transport.py", line 1855, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "/Users/coryklein/src/paramiko/paramiko/packet.py", line 327, in readline
    buf += self._read_timeout(timeout)
  File "/Users/coryklein/src/paramiko/paramiko/packet.py", line 481, in _read_timeout
    x = self.__socket.recv(128)
  File "/usr/local/lib/python2.7/site-packages/gevent/_socket2.py", line 272, in recv
    self._wait(self._read_event)
  File "/usr/local/lib/python2.7/site-packages/gevent/_socket2.py", line 171, in _wait
    self.hub.wait(watcher)
  File "/usr/local/lib/python2.7/site-packages/gevent/hub.py", line 597, in wait
    result = waiter.get()
  File "/usr/local/lib/python2.7/site-packages/gevent/hub.py", line 845, in get
    return self.hub.switch()
  File "/usr/local/lib/python2.7/site-packages/gevent/hub.py", line 576, in switch
    return greenlet.switch(self)
LoopExit: ('This operation would block forever', <Hub at 0x1059f44b0 select pending=0 ref=0>)

User jamadden commented at gevent:

A LoopExit is raised when the there's nothing else for the Hub to switch to. That is, there's only one greenlet running in the hub and it wants to switch out---but there's nothing to switch to. All other greenlets have exited (or aren't running in the hub). Is your entire process monkey-patched at an early stage so that the listening socket (at least) is running in a greenlet?

At this point it's so far down the stack I'm a bit out of depth as to how to diagnose this. I am using grequests in my app to make several HTTP requests as well, so I imagine that could potentially have side effects related to this issue.

Any ideas anyone?

[kaaveland]

Hey,

To me, this looks like the socket module was monkey-patched to use gevent. I'm not very familiar to gevent, but I imagine this could happen if the the socket module was monkey-patched to use gevent but other modules that paramiko use (such as threading) were not. Maybe you could try to add the following to the top of your file:

from gevent import monkey
monkey.patch_all()

And see if you still have the same problem?

[coryfklein]

That resolved it. Thanks.

[catchthemonster]

Beautiful answer, helped me a lot ...
Huh, two days four different ssh implementations in python all using paramiko ...

[akaars]

OMG, I wasted so much time seeking for a solution... Thank you very-very much!!!

[migumun]

Thanks a lot for this solution!!! It helped a lot!!