apr_socket_connect(): Connection reset by peer (104)

[vans163]

The client often gets socket receive errors when it sends many requests concurrently.

This did not happen on the 0.1 tag.

I think this is related to gen_statem now returning {stop, {shutdown, tcp_closed}, S} a lot more often, before 0.1 would call transport_close on the socket, but the process would stay alive. Now we both call transport_close and shutdown.

Maybe the shutdown happens before the connection can send all data?

Currently this is how we handle a http request without a "Connection: keep-alive|upgrade" header.

        {http_close, RespBin, D2} ->
            ?TRANSPORT_SEND(Socket, RespBin),
            ?TRANSPORT_CLOSE(Socket),
            {stop, {shutdown, tcp_closed}, D2};

[vans163]

Benchmarking using wrk (stargate:launch_demo()) reveals this to be true.

root@freebsd:~/wrk2 # ./wrk -t8 -c100 -d30s -R25000 http://127.0.0.1:8000/
Running 30s test @ http://127.0.0.1:8000/
8 threads and 100 connections
Thread calibration: mean lat.: 1.186ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.176ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.193ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.179ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.182ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.194ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.190ms, rate sampling interval: 10ms
Thread calibration: mean lat.: 1.191ms, rate sampling interval: 10ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.16ms 476.97us 3.17ms 64.68%
Req/Sec 3.30k 260.19 4.11k 68.99%
749275 requests in 30.00s, 40.73MB read
Socket errors: connect 0, read 16894, write 0, timeout 0
Requests/sec: 24977.47
Transfer/sec: 1.36MB

16894 socket errors on read. Compared side by side with a nginx server that had 0 socket errors on read and 500 timeouts.

This is definitely related to the gen_statem process not fully passing its send buffer to the kernel before terminating,