throw Disconnected instead ChannelOpenError SSH connection closed?

[NewUserHa]

I'm making a thing to detect the online status of machines and retire other status using run.

already know asyncssh will throw ChannelOpenError: SSH connection closed either if disabling target network service or kill the ssh process.
shouldn't throw disconnect for that situation? otherwise when will throw that exactly?

And is there a good way for retry ssh connect with asyncssh, please?
And I did some tests but couldn't figure out what is keepalive_interval for. would you like to share those info in document page?
thanks

[ronf]

The ChannelOpenError exception is raised when a failure occurs while a channel is in the process of being opened, meaning before responses are received to the MSG_CHANNEL_OPEN and related messages which set up the session. A DisconnectError would occur on channels which were fully established before the error occurred.

AsyncSSH doesn't provide any built-in way to retry a failed connect, but it would be straightforward to add your own loop to do that however many times you like when you get back an exception.

Regarding keepalive_interval, there's documentation about this in the create_connection() method. See https://asyncssh.readthedocs.io/en/latest/api.html#create-connection for more details. Here's the relevant text:

• keepalive_interval (int or float) – (optional) The time in seconds to wait before sending a keepalive message if no data has been received from the server. This defaults to 0, which disables sending these messages.
• keepalive_count_max (int) – (optional) The maximum number of keepalive messages which will be sent without getting a response before disconnecting from the server. This defaults to 3, but only applies when keepalive_interval is non-zero.

Basically, when the keepalive is enabled, messages are sent automatically by AsyncSSH on a connection when no other data has been sent for a while to make sure it is still alive. If keepalive_count_max is set and no response is received after that many keepalive messages, AsyncSSH will close the connection and report a connection failure. This is modeled after the similar feature in OpenSSH.

[NewUserHa]

thanks. your code is awesome and your answer is very detailed.

for the ChannelOpenError:

async def run_client():
    conn, _ = await asyncssh.create_connection(0, '192.168.0.234', username='root', password='123456-234', known_hosts=None, keepalive_interval=0)
    await asyncio.sleep(20)
    result = await conn.run(""" """, check=True)
    print(result.stdout, end='')
asyncio.ensure_future(run_client())
asyncio.get_event_loop().run_forever()

if kill the ssh process before conn.run, it still throw ChannelOpenError

for retry:
is there a good example that could be documented?

[ronf]

if kill the ssh process before conn.run, it still throw ChannelOpenError

I think the reason for this is that the fact that the connection has closed is not detected until Python actually tries to read/write some data on the connection. This only happens after the conn.run() starts to run, and more specifically only happens after AsyncSSH begins to open the channel. So, that channel open operation called as part of conn.run() will get back the ChannelOpenError.

for retry: is there a good example that could be documented?

It really depends on what kind of retry you want to do, and what criteria you want to use to decide when to retry and how many attempts to make. At a high level, though, I'm envisioning something like:

    attempts = 0
    conn = None

    while attempts < 5:
        try:
            conn = await asyncssh.connect(...some arguments...)
            break
        except (OSError, asyncssh.Error):
            ...possibly do something to log/record the exception here...
            attempts += 1

    if conn:
        ...connection succeeded...
    else:
        ...connection failed...

Instead of checking for conn being set here, you could have the "except" clause do the check against the max number of attempts, and "raise" the exception when the max attempts is reached. Then, you know when you break out of the loop that the connect must have succeeded. However, you'd only be seeing the exception associated with the final attempt in cases where all attempts failed.

[NewUserHa]

asyncssh mistreats disconnect Exception as ChannelOpenError Exception. can treat it as a bug to fix?

retrying if first connect fails should be handled by userspace. I meant how to retry in the ChannelOpenError situation, like implement in sshclient?

[ronf]

asyncssh mistreats disconnect Exception as ChannelOpenError Exception. can treat it as a bug to fix?

Sorry - this is not a bug. Even though the reason for the open failure is that the underlying SSH connection was closed, the fact is the the channel open was the operation that failed, and that's why the exception raised is ChannelOpenError. There's no reason you can't have a single "except" clause which looks for both ChannelOpenError and DisconnectError, if you want to treat both of them the same.

retrying if first connect fails should be handled by userspace. I meant how to retry in the ChannelOpenError situation, like implement in sshclient?

I'm not sure what you mean here. If the reason for the ChannelOpenError is that the SSH connection was closed, there's no point in retrying the conn.run(). It will consistently fail no matter how many times you retry it. At that point, you need to make a new call to asyncssh.create_connection() (or asyncssh.connect()) to re-open the SSH connection and then do your conn.run() call on that new connection.

[NewUserHa]

is there difference between ChannelOpenError and DisconnectError for connection lost?
it might be wrong but shouldn't ChannelOpenError raises only if the connection is fine but just can't open a session?

then the only way retry is to call asyncssh.create_connection() (or asyncssh.connect()) if the connection process on remote is killed?
is there an easy way to retry in sshclient overriding def connection_lost() to self-retry(without user to check which connection failed and find out its user/password from somewhere then pass to again)?

[NewUserHa]

like raise in connection_lost() of sshclient. but I can't find where to handle/capture it

[ronf]

One thing to realize here is that DisconnectError is normally associated with an actual MSG_DISCONNECT message being sent by the remote server, and there are specific disconnect codes associated with that which get returned in a "reason" attribute on the exception (see https://asyncssh.readthedocs.io/en/latest/api.html#disconnect-reasons). Similarly ChannelOpenError has a different set of reason codes (see https://asyncssh.readthedocs.io/en/latest/api.html#channel-open-failure-reasons). In the case where some lower-level error occurs such as a TCP connection failure, AsyncSSH has to fill in its own reason code from the list of available values, but the intent was to raise a consistent exception type regardless of whether the failure was an explicit message from the server with a reason code or a lower-level failure. Either way, the application code could check the "reason" value in the exception and know it would always come from a specific set of values depending on which call raised the exception. If the channel open call could return either a DisconnectError or a ChannelOpenError, there'd be an overlap in the "reason" values:

# SSH disconnect reason codes
DISC_HOST_NOT_ALLOWED_TO_CONNECT    = 1
DISC_PROTOCOL_ERROR                 = 2
DISC_KEY_EXCHANGE_FAILED            = 3
DISC_RESERVED                       = 4
DISC_MAC_ERROR                      = 5
DISC_COMPRESSION_ERROR              = 6
DISC_SERVICE_NOT_AVAILABLE          = 7
DISC_PROTOCOL_VERSION_NOT_SUPPORTED = 8
DISC_HOST_KEY_NOT_VERIFIABLE        = 9
DISC_CONNECTION_LOST                = 10
DISC_BY_APPLICATION                 = 11
DISC_TOO_MANY_CONNECTIONS           = 12
DISC_AUTH_CANCELLED_BY_USER         = 13
DISC_NO_MORE_AUTH_METHODS_AVAILABLE = 14
DISC_ILLEGAL_USER_NAME              = 15

# SSH channel open failure reason codes
OPEN_ADMINISTRATIVELY_PROHIBITED    = 1
OPEN_CONNECT_FAILED                 = 2
OPEN_UNKNOWN_CHANNEL_TYPE           = 3
OPEN_RESOURCE_SHORTAGE              = 4

I don't control these numeric values - they are specified by the SSH protocol.

Back to your question, though, it would be possible to have the connection_lost() callback in SSHClient trigger a call to asyncio.ensure_future() to open a new connection when the existing connection is closed, and you could potentially store the arguments you need to make that new connection in the existing object. However, I don't think that's necessarily the best approach. I think you'd be better off with your run_client() async task having a loop in it that made the connection and then waited for it to close by calling "await conn.wait_closed()". When that unblocks, you can decide whether or not to loop back around and connect again. You'll get a different SSHClient/SSHClientConnection each time, but the run_client() task could keep around all the values needed to call create_connection() multiple times.

By doing it this way, you'd also be able to decide whether to try to connect again when the create_connection() call fails. If you did something based on connection_lost(), there would be no way to handle this initial failure, as you would not get back an SSHClient object and connection_lost() would never be called.

[NewUserHa]

thank you very much.
but in some situation like my situation that there's only one entry point since my script needs to combine with aiohttp which only provide a way to add a background task, which seems not dynamic. but I need to add connection on the fly.

async def wait(conn):
    await conn.wait_closed()

async def main(app):
    for item in hosts.items():
        for _ in range(2):
            conn = await connect(item[0], username=item[1][1], password=item[1][2])
            item[1][0] = conn
            await wait(conn)
            logging.error('retrying.. connect lost ' + item[0])

not gonna work.

am I wrong? will asyncssh consider adding auto-reconnecting function and that might make things like this easy?

I think the only way is to catch the ChannelOpenError error to modify the host list and retry it somewhere. this way feels complicated

[ronf]

I agree that what you've written above wouldn't work here, but it seems like it wouldn't work even if you weren't trying to do a reconnect. As written, it appears to wait until the first connection is fully closed before it begins to open the next connection in your "for _ in range(2)" loop. I would think you'd want to perform these connect operations in parallel, and have separate tasks for processing data on each of the connections that ran independently of one another.

As long as you have a separate task for each connection, you should be able to support adding a loop in that which would reconnect whenever the connection closed. If necessary, you can create a task specifically for this, but I'd need to understand your existing main loop better in order to suggest a specific strategy.

It would be difficult to add this as a built-in feature in AsyncSSH, as there are a lot of different ways I can imagine it working depending on what kind of failure occurs. I can also imagine wanting control over how quickly retries are performed, and how many total retries are performed before giving up. There are also considerations like wanting to disable the reconnect and wait for the exiting connections to close when the process is being shut down. While I could try to build up a set of parameters that covers all of this, it still wouldn't be as flexible as the application coding this itself, and it's not difficult to do that in a way that's hidden from the application code making the connection.

Basically, if you have an async coroutine now for connect(), it should be possible to turn that into a coroutine that does a reconnect in a loop in a way that the caller doesn't know the difference, and then you can code whatever rules you want for when to retry, how quickly to retry, when to stop retrying, etc.

[NewUserHa]

the main() is the somekind main loop. it'll be added to aiohttp background task as by https://docs.aiohttp.org/en/latest/web_advanced.html#background-tasks

connect() is just a wrapper returning a connection

async def connect(host, username, password):
    conn, _ = await asyncssh.create_connection(0, host=host, username=username, password=password, known_hosts=None, keepalive_interval=2)
    return conn

conn.wait_close() will block right?
I can't figure out how it's possible to turn it into a coroutine supporting reconnect.

[ronf]

Yes - conn.wait_closed() will block until the connection closes, which means in the code you provided above you won't get to start the other items in your "hosts" list until the prior host's connection is established and closed twice. At any given time, you'll only be connected to one host, and unless that host chooses to close the connection or there's a network problem of some kind, you will block forever waiting for the connection to close and never get to the later hosts in the iist.

It looks like the aiohttp mechanism you pointed at here can be used to add as many background tasks as you like, using app.on_startup.append() to append each of them. However, this probably wouldn't be the right mechanism to use if you are dynamically opening connections. Instead, you could create a list of tasks (one per connection you want to make) in a startup function that you dynamically append to, and have a function registered with on_cleanup that would cancel each of the reconnect tasks in a similar fashion to how the example is canceling and waiting for the Redis listener to exit.

Basically, at startup, you'd call asyncio.create_task() to create tasks that called a "reconnect" function for each server you wanted to connect to, keeping a list of these tasks. The reconnect function would sit in a "while True" loop calling connect() and then conn.wait_closed(), and you'd wrap that loop in a try/except/finally looking for the CanceledError, doing a conn.close() in the "finally". You'd also register the on_cleanup handler like you see in their example to iterate over the list of tasks, calling cancel() and then await on each of them.