See #774 (comment) ; also commutative to #520.
Interestingly, ProxyCommand doesn't appear to expose any 'closed' attributes whatsoever - no closed, no _closed, etc.
It only implements ClosingContextManager which calls self.close, but no state is ever tracked for whether it's closed or not.
Since it wraps a subprocess.Popen I think the "most correct" analogue to "is the socket closed" would be if process.returncode is not None?
if process.returncode is not None
Untested fix re #789
@nvgoldin If possible, please try cherry-picking or just applying 228ed87 to your local Paramiko. It's pretty basic so I think it'll work, but I haven't actually tested it myself yet. Hope to later.
The exception is gone, thanks!!
But, there is another problem which I am not sure is directly related to this(tell me if to open another issue). Seems like the proxy command process isn't killed properly and it leaves zombie processes, running the following for several loops:
proxy_cmd='ssh -o StrictHostKeyChecking=no -W localhost:22 localhost'
logging.debug('running PID %s', os.getpid())
ssh = paramiko.SSHClient()
ssh_proxy = paramiko.ProxyCommand(proxy_cmd)
sftp = ssh.open_sftp()
logging.debug('going to sleep')
Results in: (20125 is the program pid):
>ps xao pid,ppid,pgid,sid,comm | grep 20125
20126 20125 20125 15530 ssh <defunct>
20169 20125 20125 15530 ssh <defunct>
20228 20125 20125 15530 ssh <defunct>
20271 20125 20125 15530 ssh <defunct>
20330 20125 20125 15530 ssh <defunct>
20373 20125 20125 15530 ssh <defunct>
20416 20125 20125 15530 ssh <defunct>
20459 20125 20125 15530 ssh <defunct>
20502 20125 20125 15530 ssh <defunct>
20545 20125 20125 15530 ssh <defunct>
20588 20125 20125 15530 ssh <defunct>
The logs after authentication look like this:
DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
DEBUG:paramiko.transport:[chan 0] Max packet out: 32768 bytes
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
INFO:paramiko.transport.sftp:[chan 0] Opened sftp connection (server version 3)
INFO:paramiko.transport.sftp:[chan 0] sftp session closed.
DEBUG:paramiko.transport:[chan 0] EOF sent (0)
DEBUG:root:going to sleep
DEBUG:paramiko.transport:EOF in transport thread
I tried changing the 'close' method of ProxyCommand from: os.kill to Popen's kill, with no success(from the docs that looks like recommended method, though not related to this.)
Maybe I should use a different order of closing(i.e. sftp/ssh/proxy)?
Update: changing proxy.py close method to:
Resolves the issue(no zombie process leftovers). Though I'm not sure if this has any side-affects.
Ping. Wonder if we can get this going. I've been testing it for the past weeks and the above fix seems to be working(no zombie processes).
Want me to create a new PR?(based on 228ed87 and adding the process.poll() to update the exit status)
Thanks for #811, I'll try verifying it on my end when I get to the next bugfix release. (May need to bump it to a feature since we're manipulating the public API, but either way it'll get looked at.)
Starting to wonder if I should investigate using Invoke's Runner for this stuff, sigh (as it, too, has to handle all sorts of subprocess shutdowns and suchlike). Not worth it in the short term though, it's not explicitly designed for wholly-noninteractive byte-forwarding (even though that SHOULD work fine and I very much want it to if it does not).
Poking #811 now...
I also doublechecked and the impl of ProxyCommand.close has been this way since 2012 - so it's unlikely to be super incorrect or I'd have expected tickets about zombies before now. Makes me wonder if some extra factor is at work in your case?
Either way I don't think it should block the basic attribute-error-fixing commit from merging so I'm gonna do that and we can spin this discussion into a new ticket if it's still affecting you. Let me know. Thanks!
Changelog re #789
@bitprophet - thanks for looking into this. I'll open a new issue if this happens again.