https connections dropped from certain client libs

[jcsp]

It has been observed that between v22.2.x and v22.3.x, requests to our HTTP endpoints (admin api, pandaproxy) made with python requests continue to work, but when made with urllib.request they fail.

One difference between these libs is that requests defaults to Connection: keep-alive, while urllib.urlopen defaults to Connection: close. They also send different Accept and Accept-Encoding headers by default.

If we compare the version of seastar used in v22.2.x with the one in v22.3.x...

git diff bdf4c58630f06aaffa02c98edc1192a2d8db2fe8..vectorizedio/v22.3.x src/http/

There are changes to handling of Connection header.

The upstream commits are

commit dd9577d7e03e33da6337542b2493445ce774df1f
Author: Jianyong Chen <baluschch@gmail.com>
Date:   Wed Aug 10 04:47:22 2022 +0800

    httpd: move the logic of keepalive to a separate method.
    
    For better readability.
    
    Signed-off-by: Jianyong Chen <baluschch@gmail.com>
    Message-Id: <20220809204722.18439-2-baluschch@gmail.com>

commit c14f9315dc9a1a1f21f2ff32499b7d3fee0ecf55
Author: Jianyong Chen <baluschch@gmail.com>
Date:   Wed Aug 10 04:47:20 2022 +0800

    httpd: compare the Connection header value in a case-insensitive manner.
    
    As per RFC 9110(section 7.6.1), connection options are case-insensitive.
    
    Signed-off-by: Jianyong Chen <baluschch@gmail.com>
    Message-Id: <20220809204722.18439-1-baluschch@gmail.com>

[jcsp]

See:

• Closing a TLS socket's input stream unexpectedly breaks its output stream as well scylladb/seastar#906

I think we are hitting a long standing seastar bug where "Connection: Close" never worked for TLS-wrapped http requests. When we seen close, the read fiber closes the input stream, but TLS sockets close both sides when either is closed, so the response then cannot be sent.

Non-TLS connections do not have the same limitation, which is why this issue only comes up when using TLS.

[jcsp]

Reverting the fixes to Connection header parsing works in practice + is what we will do for v22.3.x, but maybe we can fix this more cleanly upstream for v23.1.x: scylladb/seastar#1391