ssh connection troubleshooting

[alexandrejbr]

Is your feature request related to a problem? Please describe.
For troubleshooting purposes, we would like to be able to have more detailed information of setup of the an SSH connection. For instance the proposed algorithms by each role and the authentication methods. This would be useful for both connections that are established successfully and for connections that fail to be established.

What we have in mind is to be able to debug a connection like one would do in openssh with the verbose modes, but since that is perhaps quite ambitious, starting with what each role is proposes for the connection and information about the authentications attempts would already be great.
Describe the solution you'd like
It's hard to imagine what would be the best way to obtain this information, but I imagine that a callback could work for both roles, even though for the client role the error would come from the result of ssh:connect and then a callback function would be called as well.

Logging also works, but perhaps a bit less flexible.

Do you think it would be interesting to have such troubleshooting capabilities in the ssh application?

[u3s]

have you seen ssh_dbg module? it is pretty powerful.

for seeing result of algorithm negotiation I would use:
ssh:start(), ssh_dbg:on([alg]).

and for also getting SSH messages leading to negotiation result:
ssh:start(), ssh_dbg:on([alg, ssh_messages]).

to get all debug:
ssh:start(), ssh_dbg:on().

apply above before establishing connection. should work for both roles.
it is based on tracing feature.

[alexandrejbr]

I'll have a look and see if it will work for our use case. What we want is to every time an SSH connection is established we want to have this information stored/logged so we can afterwards investigate why a connection could not be established

If we use the traces I imagine we would need to activate this, capture the traces we are interested and later on disable the tracing. Could work.

[u3s]

you disable it with ssh_dbg:off.

it might be slightly challenging to predict which connection will fail ... traces and production systems are tricky to combine I'm afraid.

ssh_dbg is not documented feature so you would need to read source to understand how to it works.

[u3s]

ping

[alexandrejbr]

I think I agree with you, while the ssh_dbg module is extremely powerful it seems to me that it can be a bit of an hack to create features on top of it.

If one could get similar information in callbacks or some other stream of events it would be great.

Being pragmatic, for the client role we have some information already with the connection info, but that's for successful connections, if for failed connections one could get more information and a way to correlate with the ssh:connect call would be great.

For the server role it would be interesting to have the same capabilities, so to know that a connections was established and in which terms and if it ends why it ended. The same for a connection attempts that existed, why have those failed to establish the connection.

[u3s]

If one could get similar information in callbacks or some other stream of events it would be great.

There was a discussion about some generic solution related to tracing of OTP apps (@rickard-green). Maybe even an OTP behavior.
Unfortunately they're having lower backlog priority and no details were agreed at this stage.

I guess your thoughts lean towards a possibility for subscribing to troubleshooting events, which you could then process in whatever way you like or store it for later.

[alexandrejbr]

I guess your thoughts lean towards a possibility for subscribing to troubleshooting events, which you could then process in whatever way you like or store it for later.

Yes, I think you phrased it quite well. I think this could replace the existing logs and if one prefers to have the logs instead we could have a built in subscriber of these events that would translate them into info level logs.

[u3s]

What about this callbacks already present?

https://www.erlang.org/doc/apps/ssh/ssh.html#t:callbacks_daemon_options/0