bind: Address already in use - trying to restart sslh-ev with fork: true

[onehalf3570]

It would be easier to describe the steps with the netstat output, I guess:

1. systemctl start sslh
   netstat -apn4 (unrelated connections/ports omitted for clarity

tcp        0      0 external_ip:443       0.0.0.0:*               LISTEN      65773/sslh-ev       

So, 65773 is the pid of the main sslh process

2. establish an ssh session via sslh:

tcp        0      0 external_ip:443       0.0.0.0:*               LISTEN      65773/sslh-ev       
tcp        0      0 external_ip:443       client_ip:54244      ESTABLISHED 65776/sslh-ev       
..
tcp        0      0 client_ip:54244      192.168.31.254:48025    ESTABLISHED 65776/sslh-ev       
tcp        0      0 192.168.31.254:48025    client_ip:54244      ESTABLISHED 65777/sshd-session: 

pid 65776 is the fork that serves the ssh session (65777)

3. systemctl restart sslh

Feb 13 21:34:49 external_fqdn systemd[1]: Stopping sslh.service - SSL/SSH multiplexer...
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Main process exited, code=exited, status=15/n/a
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Failed with result 'exit-code'.
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Unit process 65776 (sslh-ev) remains running after unit stopped.
Feb 13 21:34:49 external_fqdn systemd[1]: Stopped sslh.service - SSL/SSH multiplexer.
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Found left-over process 65776 (sslh-ev) in control group while starting unit. Ignoring.
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 13 21:34:49 external_fqdn systemd[1]: Started sslh.service - SSL/SSH multiplexer.
Feb 13 21:34:49 external_fqdn sslh-ev[65800]: external_fqdn:https:bind: Address already in use
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Failed with result 'exit-code'.
Feb 13 21:34:49 external_fqdn systemd[1]: sslh.service: Unit process 65776 (sslh-ev) remains running after unit stopped.

65776 is still running - this is expected, as it should serve the existing ssh session - however the new 'main' process (65800) fails to bind

tcp        4      0 external_ip:443       0.0.0.0:*               LISTEN      65776/sslh-ev       
tcp        0      0 external_ip:443       client_ip:54244      ESTABLISHED 65776/sslh-ev       
tcp        0      0 client_ip:54244      192.168.31.254:48025    ESTABLISHED 65776/sslh-ev       
tcp        0      0 192.168.31.254:48025    client_ip:54244      ESTABLISHED 65777/sshd-session: 

Turns out the the process 65776 has bound itself to :443 meanwhile - this isn't expected (apparently neither for me nor for systemctl ;))

Did I misconfigure something? The config is pretty straightforward (basically a copy-paste from an example), I guess:

user: "sslh";
timeout: 5;
transparent: true;
numeric: false;

foreground: true;

udp_max_connections: 16;

...

listen:
(
    { host: "external_ip"; port: "443"; }
);

protocols: (
    {   name: "tls";
        host: "192.168.31.254";
        port: "443";
        sni_hostnames: [ "external_fqdn" ];
    },
    {   name: "tls";
        host: "192.168.31.254";
        port: "80";
        sni_hostnames: [ "www.external_fqdn" ];
        tfo_ok: true;
    },
    {   name: "ssh";
        service: "ssh";
        host: "192.168.31.254";
        port: "48025";
        keepalive: true;
        fork: true;
        tfo_ok: true;
    }
);

Thanks.

[yrutschle]

Indeed, I can reproduce this. This is strange, I think bind() is only ever called at startup.

[yrutschle]

Ok, this is caused by the child process not closing its file descriptors, so it still has a file descriptor to the listening socket, which is somehow kept around by the kernel, and the listening socket being reattributed to the child process.

[onehalf3570]

Thanks for a really quick investigation! Merci!

[yrutschle]

this should be fixed in e2d01a7