Leaked ssh connections

[qais-yousef]

After running several tests on my target using LISA I suddenly started encountering ssh related problems.

I had 2 mode of failres: failed to scp and failed to connect altogether.

ssh'ing directly into the target from the terminal (ie: outside LISA test environment) worked.

After a bit of digging it seemed that there were many ssh connection looming around that cause the target to reach a limit after which it drops further ssh connection requests.

I am not entirely sure how everything works - but I believe devlib handles the connection steps with the target? If not please let me know and I'll file a bug against the correct module.

NOTE: I might have ctrl+c to interrupt some tests. Not sure if this contributed to the issue or just continuously running tests on a target without rebooting will cause this to happen eventually.

[marcbonnici]

Hi,
Thanks for reporting this, I have submitted a PR (#310) to ensure that the connections are closed automatically upon exit.
Would you be be able to let me know if this fixes the issue you've been having?

[qais-yousef]

Thanks for the fast response Marc. I'll test this and let you know.

[qais-yousef]

Hi Marc

With the proposed fix I can no longer see ssh sessions leaking. I tested it with a simple ps aux | grep ssh where you could see a left over ssh session. And by running overnight test which didn't encounter the dropped connection problem.

I don't know what's the protocol - I can close this issue or feel free to close it once your PR is merged.

Cheers.

[marcbonnici]

Thank you very much for testing that out for me.

That's fine I can close the issue when we merge the PR.

[qais-yousef]

Actually just after I wrote the last comment - the test got stuck and had to ctrl+c and got this backtrace - something went wrong at atexit and the connection is still open.

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/qais/src/lisa/libs/devlib/devlib/utils/ssh.py", line 237, in close
    self.conn.logout()
  File "/usr/local/lib/python2.7/dist-packages/pexpect/pxssh.py", line 443, in logout
    index = self.expect([EOF, "(?i)there are stopped jobs"])
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 111, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/pty_spawn.py", line 471, in read_nonblocking
    [self.child_fd], [], [], timeout
  File "/usr/local/lib/python2.7/dist-packages/pexpect/utils.py", line 143, in select_ignore_interrupts
    return select.select(iwtd, owtd, ewtd, timeout)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/qais/src/lisa/libs/devlib/devlib/utils/ssh.py", line 237, in close
    self.conn.logout()
  File "/usr/local/lib/python2.7/dist-packages/pexpect/pxssh.py", line 443, in logout
    index = self.expect([EOF, "(?i)there are stopped jobs"])
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 111, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/pty_spawn.py", line 471, in read_nonblocking
    [self.child_fd], [], [], timeout
  File "/usr/local/lib/python2.7/dist-packages/pexpect/utils.py", line 143, in select_ignore_interrupts
    return select.select(iwtd, owtd, ewtd, timeout)
KeyboardInterrupt

[qais-yousef]

My hikey960 went dead as well. It might be a one time off problem. If the backtrace doesn't shout anything obvious to you (I didn't really try to dig into it) then we can leave it and if it happens again I can raise a new issue if this one was already closed.

[marcbonnici]

Ah ok, I'm not currently sure what could have caused that and have not managed to reproduce it on my end yet.
When you said the connection was left open was that from the perspective of the host or hikey960?

[qais-yousef]

I meant ps aux | grep ssh in the host returned a process that was run to communicate with hikey960 (it had its IP address in the cmdline)

Hikey960 was dead in a sense that the serial console was no longer responsive.

[marcbonnici]

Oh right ok, I guess that would make sense if the board is no longer responding correctly to close the connection successfully although I'm not currently sure what else we can do at this stage.

[qais-yousef]

Not really. At least not to my (little) knowledge.
I think we're hitting an issue similar to this

pexpect/pexpect#441

Looking at pxssh documentation the logout method just tries an 'exit' and since the system has died there's no exit executed so it just 'hangs' waiting for something to happen but nothing happens.

Can we do

try:
    self.conn.logout()
except:
    # Force kill the connection

?

Again I am not particularly knowledgeable about this - so maybe my suggestion doesn't make sense.
I am assuming ctrl+c when self.conn.logout() hangs will trigger the 'except' path.