Execute Command doesnt work with dropbear

[legraps]

Environment:
SSHLibrary version: 2.1.1
Paramiko version: 1.15.1
Used interpreter: Python 2.7.6
Operating system: Ubuntu 14.04.1 LTS

Since upgrading dropbear in my embedded target (to 2014.66) Executing Command or rather waiting for the Command Result leads to hanging in an endless loop in read_outputs in paramikos recv_exit_status() function. It seems that upon receiving dropbears EOF we never send a EOF which leads to the connection still being open. SSHLibraries _shell_open() function however makes us think the connection was teared down correctly.

I have tracked down the issue in dropbear to their changeset:

936:d93a6bcf616f
Improve handling lots of concurrent forwarded connections. Increase connection backlog, avoid check_close() for channels that haven't had IO

but I cant actually change that code in our target. (all other ssh connections do work without problems).

So below is a patch that fixes my problem by checking if exit_status_ready() will be successful before actually exexuting recv_exit_status(). In case it's not, the write direction is closed forcefully (which sends out the missing? EOF).

It works for me but I'm not sure if this is ok for everybody...? Any experts on this?

Index: SSHLibrary/src/SSHLibrary/pythonclient.py

--- SSHLibrary.orig/src/SSHLibrary/pythonclient.py 2014-12-15 11:38:59.830379909 +0100
+++ SSHLibrary/src/SSHLibrary/pythonclient.py 2014-12-15 11:40:13.132928618 +0100
@@ -169,6 +169,8 @@

 def read_outputs(self):
     stderr, stdout = self._receive_stdout_and_stderr()

•    if not self._shell.exit_status_ready():
  

•        self._shell.shutdown_write()
   rc = self._shell.recv_exit_status()
   self._shell.close()
   return stdout, stderr, rc
  

[pekkaklarck]

1. The included patch is broken because GitHub parses is as Markdown. You should put it into a fenced code block.

2. I'm not an expert regarding to this subject but the fix looks good.

3. Have you tested does the same problem occur also if you run tests using Jython? If it does, could you investigate is there similar fix for it?

[legraps]

Hi,
3) Havnt tested Jython. I see if I get around it today.
2) As to maturity: I've seen seemingly similar hangup during a large sftp transfer. Needs investigating..

1. below is the patch in clean form again

Index: SSHLibrary/src/SSHLibrary/pythonclient.py
===================================================================
--- SSHLibrary.orig/src/SSHLibrary/pythonclient.py  2014-12-15 11:38:59.830379909 +0100
+++ SSHLibrary/src/SSHLibrary/pythonclient.py   2014-12-15 11:40:13.132928618 +0100
@@ -169,6 +169,8 @@

     def read_outputs(self):
         stderr, stdout = self._receive_stdout_and_stderr()
+        if not self._shell.exit_status_ready():
+            self._shell.shutdown_write()
         rc = self._shell.recv_exit_status()
         self._shell.close()
         return stdout, stderr, rc

[legraps]

Unfortunately at the moment I dont find time to make my tests run with jython. The code looks different though so this might not be a problem.
The sftp issue might be rooted in the sftp-server I use, so it's not part of this issue.

[causten]

#1 thanks for the fix. It works for my embedded project using dropbear too
#2 any chance this is going in soon?

[jomofer]

Hi

Once again, thanks for this fix. Updating dropbear to v2015.68 on an embedded project seems to require this hotfix in SSHLibrary.

Tested with pybot

[legraps]

Actually I reverted the patch in my setup because unfortunately it doesnt fix the issue 100%. Finally I was able to change dropbear in our target, so I reverted the commit I indicated in my initial post.

[causten]

so you fixed it in dropbear? Any chance you upstreamed it or have a patch?

[jomofer]

Yes, I have seen some problems regarding SSH after more tries, too.
At the end, I have changed dropbear to v2014.63 and SSHLibrary to 2.1.1 without any patches

[mkj]

I'll look at this from the Dropbear side, is there a simple testcase I can run with SSHLibrary? Thanks

[legraps]

Hi mkj, thanks for looking into it. Here's a quick setup:
pip install robotframework
pip install robotframework-SSHLibrary

Create a file dropbear.txt:

*** Settings ***
Library           SSHLibrary

*** Test Cases ***
Repeatedly issue commands via dropbear ssh
    Enable Ssh Logging    ssh.log
    Open Connection    ${IP}
    Login    root    dropbear
    : FOR    ${i}    IN RANGE    100
    \    SSHLibrary.Execute Command    echo dropbear ${i}

Run it with (replace localhost with whatever your host is)
pybot -L TRACE -v IP:localhost dropbear.txt

It'll hang after a few repetitions. If you end the test by killing the active dropbear thread on the other side you'll get a nice log.html, but it wont give you too much information. Maybe the ssh.log helps.

Am curious what you find...

[mkj]

I think this should be fixed in mkj/dropbear@ee35384 Fix ses.channel_signal_pending race

There was a signal handler race so Dropbear would sometimes miss noticing the child process exiting.

[jussimalinen]

Alright, thanks @mkj! If I understood correctly, this means that we can close this issue on our side. We can reopen this if some action is required on our side.

[legraps]

Tested the latest dropbear issuing 5000 commands via SSHLibrary and dont see the problem any longer (on x86). Thanks mkj!
For now I dont think there's anything else required on SSHLibrary side.