[BUG] - Connect failed and timeout exection when I run exec_command in threads and loop

[HippoSoX]

Are you using paramiko as a client or server?

Client

What feature(s) aren't working right?

SSH

What version(s) of paramiko are you using?

3.3.1

What version(s) of Python are you using?

3.7.9

What operating system and version are you using?

windows10 as local client, and linux at remote

If you're connecting as a client, which SSH server are you connecting to?

OpenSSH 8.6

If you're using paramiko as part of another tool, which tool/version?

No response

Expected/desired behavior

I am trying to use multiple ssh to run about 15 commands by exec_command in parallel by threads in loop. It could call about 15 channels at the same time and finally reach over 1000 channels that is ever used.

Actual behavior

It seems like 2 kinds of error.
First one is about most parallel connect, second one is about most history run.
The first one occur at the beginning, when I try to run exec_command in about 15 threads, which seems like create 15 channel and run at the same time.

Successfully connected to  192.168.3.136
Secsh channel 12 open FAILED: open failed: Connect failed
Secsh channel 13 open FAILED: open failed: Connect failed
Secsh channel 14 open FAILED: open failed: Connect failed
Secsh channel 15 open FAILED: open failed: Connect failed
Traceback (most recent call last):
  File "\path\to\my\project\devices\myssh.py", line 69, in wrapper
    return func(*args, **kwargs), True
  File "\path\to\my\project\devices\myssh.py", line 282, in exec
    stdin, stdout, stderr = self.ssh_client.exec_command(command, bufsize=65535, timeout=waittime) 
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 947, in open_session
    timeout=timeout,
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 1085, in open_channel
    raise e
  File "\path\to\my\project\devices\myssh.py", line 69, in wrapper
    return func(*args, **kwargs), True
Traceback (most recent call last):
  File "\path\to\my\project\devices\myssh.py", line 69, in wrapper
    return func(*args, **kwargs), True
  File "\path\to\my\project\devices\myssh.py", line 282, in exec
    stdin, stdout, stderr = self.ssh_client.exec_command(command, bufsize=65535, timeout=waittime)
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')
  File "\path\to\my\project\devices\myssh.py", line 69, in wrapper
    return func(*args, **kwargs), True
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "\path\to\my\project\devices\myssh.py", line 282, in exec
    stdin, stdout, stderr = self.ssh_client.exec_command(command, bufsize=65535, timeout=waittime)
  File "\path\to\my\project\devices\myssh.py", line 282, in exec
    stdin, stdout, stderr = self.ssh_client.exec_command(command, bufsize=65535, timeout=waittime)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 947, in open_session
    timeout=timeout,
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 947, in open_session
    timeout=timeout,
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 1085, in open_channel
    raise e
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 1085, in open_channel
    raise e
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 947, in open_session
    timeout=timeout,
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 1085, in open_channel
    raise e
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')

Then it goes so many channel failed while running, almost 40%. Log shows below.

I think maybe it reach the limit of ssh config that 10/100/1000 which means most parallel connect cannot over 10 and most history connect cannot over 1000, so I try some way to limit max parallel workers under 5. It seems like running good without any error until reach 1000 channel. Then make the second error.

When I read the source code I found all channels even if create by calling exec_command, put into a weakref map, which means it still taking away memory resources. Does it release ssh connect after running?

Secsh channel 45 open FAILED: open failed: Connect failed
Traceback (most recent call last):
  File "D:\workspace\VSCodeProject\production_test_tool\zmc900\devices\myssh.py", line 69, in wrapper
    return func(*args, **kwargs), True
  File "D:\workspace\VSCodeProject\production_test_tool\zmc900\devices\myssh.py", line 282, in exec
    stdin, stdout, stderr = self.ssh_client.exec_command(command, bufsize=65535, timeout=waittime)
Secsh channel 46 open FAILED: open failed: Connect failed
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 947, in open_session
    timeout=timeout,
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 1085, in open_channel
    raise e
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')
Traceback (most recent call last):
  File "D:\workspace\VSCodeProject\production_test_tool\zmc900\devices\myssh.py", line 69, in wrapper
    return func(*args, **kwargs), True
  File "D:\workspace\VSCodeProject\production_test_tool\zmc900\devices\myssh.py", line 282, in exec
    stdin, stdout, stderr = self.ssh_client.exec_command(command, bufsize=65535, timeout=waittime)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 947, in open_session
    timeout=timeout,
  File "C:\Users\hemanjun\AppData\Local\Programs\Python\Python37\lib\site-packages\paramiko\transport.py", line 1085, in open_channel
    raise e
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')

How to reproduce

No response

Anything else?

No response

[jun66j5]

That is not a paramiko issue. The MaxSessions in sshd_config is 10 by default. The parameter should be configured if you need more sessions.

[HippoSoX]

Thanks for answers. I try to limit the local maxsessions but when I use exec_command sometimes it still reach the maxsessions. for example, when remote config setup max sessions as 10, I cannot normally access until I reduce parallel workers to 5. I found comments that channel will auto closed after exec_command, but also I found that all channel will put into weakref map to avoid auto garbage collection. I want to make sure that the channels are really release the ssh sock after called by exec_command. Can it find some way to promise to do that or how can I modify it if necessary?

[jun66j5]

I found comments that channel will auto closed after exec_command, ...

That is incorrect. If you're using SSHClient.exec_command, the channel is not closed until std{in,out,err} are garbage collected. The channel is set on attribute of std{in,out,err} and you could close the channel. Or, you could use directly Transport.open_session() and Channel.exec_command().

cli = paramiko.SSHClient()
# connect to host...
stdin, stdout, stderr = cli.exec_command(...)
try:
  # write data to stdin, read data from stdout/stderr...
finally:
  stdout.channel.close()