SSL server can assume a state where it no longer answers to client requests

[triska]

Every few days (sometimes: weeks) of continuous uptime, the HTTPS server I am running assumes a state where it still accepts connections, but does not complete an SSL handshake.

In that state, when I try to connect to the server with openssl, I get:

$ openssl s_client -host $HOST -port 443
CONNECTED(00000003)

followed by nothing more, i.e., nothing more appears and the handshake seems to hang. When the server is still working correctly, the output continues with:

depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2008 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA - G3
verify return:1
depth=1 C = US, O = "thawte, Inc.", OU = Domain Validated SSL, CN = thawte DV SSL SHA256 CA
verify return:1
...

Please let me know what I can do to trace this down. I can use gdb to trace locked threads with:

gdb swipl 
(gdb) info threads
 ...
(gdb) thread 
(gdb) bt

I will try this next time this hangs. So far, I could not replicate this issue with any tests. I have run several benchmarks on the server, using the Apache bench tool ab, and it seems to behave as expected in all scenarios I have tried. When the hang occurs, there are still enough available resources on the server to easily make further connections. I can also use all other ports, and can SSH into the server without any problems. Only the HTTPS connections no longer work.

To collect experience with such scenarios, please consider moving the SWI website or other frequently used sites to HTTPS, using native SWI-Prolog to make the connection without intermediary tools. I would also greatly appreciate feedback from other users who use the SSL package, and advice from networking/openssl experts who are interested in helping with this. @chigley, I would greatly appreciate your input in particular, at your convenience.

[JanWielemaker]

Unfortunately the services we run here at the VU are hard to run that way because our infrastructure requires a proxy in between. This rules out eu.swi-prolog.org and swish.swi-prolog.org. I could use SSL on us.swi-prolog.org. Not sure how the CDN will respond if one of its backends uses HTTPS and the other HTTP. It is also doubtful whether that works as 99% of the connections will come from fastly. Your issue might be caused by a misbehaving client (which of course should not lead to a DoS).

Anyway, make sure the system is compiled with this line uncommented in build:

export COFLAGS="-O2 -gdwarf-2 -g3"

If it hangs, you can find the blocking thread using gdb and you can (often) find the mutex and print its details to see whether it is an attempt at a recursive lock or which thread is holding the lock. You can also use

(gdb) call PL_backtrace(25,1)

to dump a Prolog stack trace to stderr for the current thread. If it is a deadlock, these things usually suffice to find out what is happening.

[triska]

I have constructed a self-contained test case that shows the problem, or at least an issue that is likely very closely related.

First, please create server.key and server.crt (alternatively, you can use existing files instead):

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

The Prolog server that shows the issue is server.pl:

:- use_module(library(ssl)).
run :-
        ssl_context(server, SSL, [key_file('server.key'),
                                  certificate_file('server.crt'),
                                  cipher_list('DEFAULT:!DES:!RSA')
                                  ]),
        tcp_socket(Socket),
        Port = 8371,
        tcp_bind(Socket, localhost:Port),
        format("listening on port: ~w\n", [Port]),
        tcp_listen(Socket, 5),
        tcp_open_socket(Socket, AcceptFd, _),
        repeat,
            tcp_accept(AcceptFd, Client, _PeerName),
            tcp_open_socket(Client, StreamPair),
            stream_pair(StreamPair, PlainRead, PlainWrite),
            catch(ssl_negotiate(SSL, PlainRead, PlainWrite, _Read, _Write),
                  _,
                  true),
            catch(close(StreamPair), _, true),
            false.
:- initialization(run).

Run it with:

$ swipl server.pl

It will listen on port 8371, you can change it easily in the source.

To hammer the server with repeated connection attempts, we use the sslscan tool. You can install it easily via apt-get. A single run is not enough, so we use the script thrash.sh to thrash the server:

#!/bin/bash
i=0
port=$1
while true
do
    echo thrashing $i
    sslscan localhost:$port | grep -i acc
    i=$((i+1))
done

You run the script via:

$ ./thrash.sh 8371

(EDIT: corrected port)

Depending on the ciphers that are enabled, it will hang after "thrashing 21", "thrashing 45" or a similar (rather small) number of runs.

I have also produced a backtrace as suggested, but it does not tell me a lot:

  [8] socket:tcp_accept/3 [PC=1 in supervisor]
  [7] run/0 [PC=79 in clause 1]
  [6] $toplevel:residue_vars/2 [PC=5 in clause 2]
  [5] $toplevel:$execute_goal2/2 [PC=8 in clause 1]
  [3] $toplevel:$query_loop/0 [PC=101 in clause 1]
  [2] $toplevel:$runtoplevel/0 [PC=45 in clause 1]
  [1] $toplevel:$toplevel/0 [PC=3 in clause 1]
  [0] system:$c_call_prolog/0 [PC=0 in top query clause]

Please try to reproduce this and let me know if I can supply an additional information. If you send me your SSH public key, I will create an account for you to reproduce this.

[FunctionAnalysis]

Using the server.pl file, calls directly to server/1 fail, but after using the graphical debugger and trace option I followed the steps of calling server/0.

Just going through the navigator of the server.pl file, I see that server/0 and client/0 loop with exception handling calling the ssl.pl with server/1 and server/3.

With what limited things I know about ssl, and comparing it to the issue of certificate verification when relying on root certificates, my guess is that the client fails to verify, so the server ends up calling server/0.

When you call server/0 after consulting server.pl it enters recursion. Since it is meant to go to ssl.pl after the ssl_init module is called, it could be that the server basically ends up "talking to itself" which is why it hangs.

Since this happens at low thrash values, the negotiation is failing, but with any "root" only attempts the "root" will be the only one to accept the connection.

Another thing could be how ports are being handled. I only see one use of a port(Port) and with SSL the required https port is 443, but most hard-coded ports in the files are 2443 which is UDP (ignoring any 1111 ports which would need to obviously be switched by an admin).

So it looks like the UDP default port of 2443 is getting instantiated in place of a multi-threading with 443.

[JanWielemaker]

Built and ran this on Fedora 24. Stopped it after 300,000 runs. No hangs, no leaks, all just fine. To some extend this is a bit strange to me as I think the code should have been as below, closing the SSL streams
rather than the raw streams. I added a print message call just be be sure. No errors are printed. This
code runs just fine as well.

            stream_pair(StreamPair, PlainRead, PlainWrite),
            catch(ssl_negotiate(SSL, PlainRead, PlainWrite, Read, Write),
                  Essl,
                  print_message(error, Essl)),
            catch(call_cleanup(close(Write), close(Read)), E,
          print_message(error, E)),
            false.

[JanWielemaker]

@FunctionAnalysis I can't relate your post to this thread. There is only a run/0, no server/0, client/0, etc.

[JanWielemaker]

@triska A hanging tcp_accept/3 is a little weird. There is either some issue in the Prolog layer around sockets or there is some system reason to block. As you've got it in gdb, please run the gdb bt command against the hanging thread. That should give the C stack from which we can deduce whether it is hanging on a Prolog lock, the accept() system call or whatever. With some luck you can figure out what. GDB bluffers guide:

• info threads lists all threads
• thread N switches to a thread
• bt prints a backtrace
• up and down go a frame up or down
• p <expression> prints the value of an expression

With the source code at hand, that typically suffices to get a clue.

[FunctionAnalysis]

Looking at the latest ssl folder, and downloading the zip available in the official SWIPL repository, there are seven prolog source files. I was looking through all of them viewed in the navigator tool.

There is a call using the server module which itself calls a server/0.

I will post screen captures to illustrate what I am referring to.

On Sep 20, 2016, at 08:58, Jan Wielemaker notifications@github.com wrote:

@FunctionAnalysis I can't relate your post to this thread. There is only a run/0, no server/0, client/0, etc.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[FunctionAnalysis]

In the interest of my time, here is a capture of the main ssl package on the official repo.

[triska]

@JanWielemaker : There was a typo in the script invocation (I have now fixed it), please use the following to thrash the server (note the correct server port):

./thrash.sh 8371

I can reproduce the hang on a fresh Debian 8.5 install, please send me your public key if you want full root access to the system.

The commands you posted yield:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7f2d9d6f5700 (LWP 3163) "swipl" 0x00007f2d9cf92cb0 in __accept_nocancel () at ../sysdeps/unix/syscall-template.S:81
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f2d9d6f5700 (LWP 3163))]
#0  0x00007f2d9cf92cb0 in __accept_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
81  in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0  0x00007f2d9cf92cb0 in __accept_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f2d9b9180e6 in nbio_accept (master=,
    addr=addr@entry=0x7ffd240a4d60, addrlen=addrlen@entry=0x7ffd240a4d5c)
    at nonblockio.c:2138
#2  0x00007f2d9b916d43 in pl_accept (Master=, Slave=271,
    Peer=272) at socket.c:480
#3  0x00007f2d9d1df0c5 in PL_next_solution (qid=3, qid@entry=22)
    at pl-vmi.c:3678
#4  0x00007f2d9d21453f in query_loop (goal=goal@entry=316421, loop=0)
    at pl-pro.c:115
#5  0x00007f2d9d214cdb in prologToplevel (goal=316421) at pl-pro.c:428
#6  0x00007f2d9d24ead4 in PL_initialise (argc=1, argc@entry=2,
    argv=, argv@entry=0x7ffd240a5678) at pl-init.c:913
#7  0x0000000000400831 in main (argc=2, argv=0x7ffd240a5678) at pl-main.c:118

[JanWielemaker]

With the correct params I can at least reproduce your issue. First of all, the closing is not correct, so we need some work to fix that. Eventually the test program is now this:

:- use_module(library(ssl)).

run :-
    ssl_context(server, SSL,
            [ key_file('server.key'),
              certificate_file('server.crt'),
              cipher_list('DEFAULT:!DES:!RSA'),
              close_parent(true)
            ]),
        tcp_socket(Socket),
    tcp_setopt(Socket, reuseaddr),
    Port = 8371,
        tcp_bind(Socket, localhost:Port),
    current_prolog_flag(pid, PID),
        format("Process ~d: listening on port: ~w\n", [PID, Port]),
        tcp_listen(Socket, 5),
        tcp_open_socket(Socket, AcceptFd, _),
        between(1, infinite, Run),
            tcp_accept(AcceptFd, Client, _PeerName),
            tcp_open_socket(Client, StreamPair),
            stream_pair(StreamPair, PlainRead, PlainWrite),
            (   catch(ssl_negotiate(SSL, PlainRead, PlainWrite, Read, Write),
              Essl,
              print_message(error, Essl))
        ->  (   var(Essl)
        ->  format('Connected ~D~n', [Run]),
            catch(call_cleanup(close(Write), close(Read)), E,
              print_message(error, E))
        ;   format('Failed ~D~n', [Run]),
            catch(close(StreamPair), Ec,
              print_message(error, Ec))
        )
        ;   format('Negotiate failed ~D~n', [Run])
        ),
        false.

:- initialization
    thread_create(run, _, []).

lsfd :-
    current_prolog_flag(pid, PID),
    format(string(Cmd), "ls -l /proc/~w/fd", [PID]),
    shell(Cmd).

The thread is not needed, but frees the toplevel so we can look around. Using this, trashing no longer leaks memory and file handles. It runs for quite a while before pausing. If you wait for the ./thrash.sh to stop, you can indeed find that Prolog is trapped in accept(). If you stop ./thrash.sh and just call the sslscan without grep, you'll find that it gets an error in bind(). That is before it tries to contact the Prolog server and that is why the Prolog server gets no connection attempts and hangs in accept(). Nothing wrong there.

The issue is probably caused by sockets that are not closed correctly and wait for their lingering period. You can see that by just waiting. At some point it resumes. As far as I can see, Prolog is closing all sockets nicely, so it might be sslscan that doesn't do so. You may try the thrash loop against another server.

[triska]

OK, I think the test case I constructed is not sufficient to elicit the hang I observe when running the HTTPS server in practice.

Yesterday, I observed the actual issue again, after 5 days of continuous uptime. I have run GDB on the hanging process and obtained:

(gdb) info threads
  Id   Target Id         Frame
  7    Thread 0x7f1a17f3e700 (LWP 1522) "swipl" 0x00007f1a1bccaddb in __libc_recv (fd=4,
    buf=buf@entry=0x7f1a1038dc80, n=n@entry=4096, flags=-1, flags@entry=0)
    at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
  6    Thread 0x7f19d3fff700 (LWP 1543) "swipl" 0x00007f1a1bccaddb in __libc_recv (fd=10,
    buf=buf@entry=0x7f19ec04ca70, n=n@entry=4096, flags=-1, flags@entry=0)
    at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
  5    Thread 0x7f19d27fc700 (LWP 1546) "swipl" 0x00007f1a1bccaddb in __libc_recv (fd=7,
    buf=buf@entry=0x7f19f004ae10, n=n@entry=4096, flags=-1, flags@entry=0)
    at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
  4    Thread 0x7f19ca7fc700 (LWP 1553) "swipl" 0x00007f1a1bccaddb in __libc_recv (fd=5,
    buf=buf@entry=0x7f19d800cf40, n=n@entry=4096, flags=-1, flags@entry=0)
    at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
  3    Thread 0x7f19c9ffb700 (LWP 1554) "swipl" 0x00007f1a1bccaccd in accept ()
    at ../sysdeps/unix/syscall-template.S:81
  2    Thread 0x7f19c97fa700 (LWP 1555) "swipl" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1    Thread 0x7f1a1c42c700 (LWP 1520) "swipl" pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f1a1c42c700 (LWP 1520))]
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
238 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S
(gdb) bt
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007f1a1bf7607f in dispatch_cond_wait () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#2  0x00007f1a1bf7b643 in get_message () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#3  0x00007f1a1bf7b9c1 in pl_thread_get_message1_va ()
   from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#4  0x00007f1a1bf17a92 in PL_next_solution () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#5  0x00007f1a1bf4c3fe in query_loop () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#6  0x00007f1a1bf4cbcb in prologToplevel () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#7  0x00007f1a1bf86914 in PL_initialise () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#8  0x0000000000400831 in main ()

I have also tried to obtain a backtrace with PL_backtrace, but I think this caused the program to terminate:

(gdb) call PL_backtrace(25,1)
Program received signal SIGPIPE, Broken pipe.
0x00007f1a1bccaa7d in write () at ../sysdeps/unix/syscall-template.S:81
81  ../sysdeps/unix/syscall-template.S: No such file or directory.
The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use "set unwindonsignal on".
Evaluation of the expression containing the function
(PL_backtrace) will be abandoned.
When the function is done executing, GDB will silently stop.
(gdb) bt
#0  0x00007f1a1bccaa7d in write () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f1a1bfa97bf in S__flushbuf () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#2  0x00007f1a1bfab6b9 in Sflush () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#3  0x00007f1a1bfab99d in Svdprintf () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#4  0x00007f1a1bfaba4c in Sdprintf () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#5  0x00007f1a1bf675fe in writeContextFrame () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#6  0x00007f1a1bf676d2 in PL_backtrace () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#7  
#8  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#9  0x00007f1a1bf7607f in dispatch_cond_wait () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#10 0x00007f1a1bf7b643 in get_message () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#11 0x00007f1a1bf7b9c1 in pl_thread_get_message1_va ()
   from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#12 0x00007f1a1bf17a92 in PL_next_solution () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#13 0x00007f1a1bf4c3fe in query_loop () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#14 0x00007f1a1bf4cbcb in prologToplevel () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#15 0x00007f1a1bf86914 in PL_initialise () from /usr/local/lib/swipl-7.3.25/lib/x86_64-linux/libswipl.so.7.3
#16 0x0000000000400831 in main ()

Does this help in any way? Please let me know what else I can try to locate the root cause of this.

[JanWielemaker]

This may be perfectly ok as a state. What I do see is only 4 worker threads. That is very low for a real server. I typically use at least 16, depending in the load. Note that if keep alive is used (default), each worker will remain waiting for a second for new requests before dropping the line and become part of the available queue.

If it permanently blocks the threads hanging in recv are somehow doing something not sensitive to timeout. That can be legitimate if you modify the timeout and read e.g. posted data.

What we do need is a PL_backtrace() that returns a string, as the backtrace is commonly send to
an unknown stream :( Anyway, the above may give you a clue. Are we dealing with POST? How is t he data read?

[triska]

It is strange: I am running this program via a systemd service file, using --workers=32!

The salient part of the service file is:

[Service]
UMask=022
Environment=LANG=en_US.utf8
Restart=on-abort
StartLimitInterval=60
LimitNOFILE=1000
DefaultStartLimitBurst=5
WorkingDirectory=...
ExecStart=/usr/local/bin/swipl ... \
   --no-fork --user=... --https --certfile=...  --keyfile=...\
   --pidfile=/var/run/proloxy.pid --workers=32 --syslog=proloxy \
   --cipherlist=... --keep_alive_timeout=600

The program is my HTTP-proxy Proloxy, which only relays the requests.

One interesting piece of information, which may be a coincidence however: I see from the log files that yesterday, in the hours before the issue occurred, there were 4 HEAD requests, one at 5:38, one at 15:01, one at 15:03 and one at 15:19. The issue occurred after that, probably between 19:00 and 20:00 o'clock. Could it be related to the HEAD requests?

All other requests yesterday were GET requests. On the days before yesterday, I had also several HEAD requests, and a few POST requests that are related to Pengines. The vast majority of all requests were always GET.

[JanWielemaker]

A keep_alive_timeout of 600 will need many more workers or spawn jobs on new threads. Still, there are two mysteries what happened with the 28 workers and are the remaining 4 in keep alive timeout or not? No clue whether HEAD requests are part of the story, but that should be easy to test, no?