in_http: HTTP/1.1 keep-alive missing when HTTP/2 is enabled

[iandrewt]

Bug Report

Describe the bug
Since fluent-bit 3.0, HTTP/1.1 keep-alive has stopped working by default, as the new HTTP library does not seem to support it.
We have our fluent-bit HTTP input behind NGINX to load balance across multiple instances, and as NGINX doesn't support HTTP/2 for backend connections, this results in a new TCP handshake for every transaction.

To Reproduce

• Steps to reproduce the problem:

$  fluent-bit -i http -o stdout

$ curl -v --http1.1 -d '{"hello":"world"}' -H 'Content-Type: application/json' 'http://localhost:9880/?[0-1]'
* Host localhost:9880 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9880...
* connect to ::1 port 9880 from ::1 port 54810 failed: Connection refused
*   Trying 127.0.0.1:9880...
* Connected to localhost (127.0.0.1) port 9880
> POST /?0 HTTP/1.1
> Host: localhost:9880
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 17
> 
* upload completely sent off: 17 bytes
< HTTP/1.1 201 Created
< content-length: 0
< 
* Connection #0 to host localhost left intact
* Found bundle for host: 0x651731f03220 [serially]
* Connection 0 seems to be dead <- note the connection is being recreated
* Closing connection
* Hostname localhost was found in DNS cache
*   Trying [::1]:9880...
* connect to ::1 port 9880 from ::1 port 54818 failed: Connection refused
*   Trying 127.0.0.1:9880...
* Connected to localhost (127.0.0.1) port 9880
> POST /?1 HTTP/1.1
> Host: localhost:9880
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 17
> 
* upload completely sent off: 17 bytes
< HTTP/1.1 201 Created
< content-length: 0
< 
* Connection #1 to host localhost left intact

Expected behavior

$ fluent-bit -i http -p http2=false -o stdout

$ curl -v --http1.1 -d '{"hello":"world"}' -H 'Content-Type: application/json' 'http://localhost:9880/?[0-1]'
* Host localhost:9880 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9880...
* connect to ::1 port 9880 from ::1 port 40160 failed: Connection refused
*   Trying 127.0.0.1:9880...
* Connected to localhost (127.0.0.1) port 9880
> POST /?0 HTTP/1.1
> Host: localhost:9880
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 17
> 
* upload completely sent off: 17 bytes
< HTTP/1.1 201 Created 
< Server: Fluent Bit v3.0.3
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Found bundle for host: 0x630e26273220 [serially]
* Re-using existing connection with host localhost <- note the connection is being reused here
> POST /?1 HTTP/1.1
> Host: localhost:9880
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 17
> 
* upload completely sent off: 17 bytes
< HTTP/1.1 201 Created 
< Server: Fluent Bit v3.0.3
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

Your Environment

• Version used: 3.0.2 (yes the version in the working curl output above is 3.0.3 - I tested this from latest master as well as released 3.0.2)
• Configuration: fluent-bit -i http -o stdout
• Environment name and version (e.g. Kubernetes? What version?): N/A
• Server type and version: N/A
• Operating System and version: Arch Linux 6.8.7
• Filters and plugins: N/A

Additional context
I'm happy to work on this issue myself, would just need to learn a bit about how fluent-bit manages TCP connections. I've tried changing FLB_HTTP_SERVER_FLAG_KEEPALIVE to 1 here but that made fluent-bit crash on the second curl request

fluent-bit/include/fluent-bit/http_server/flb_http_server.h

Line 42 in e47718a

#define FLB_HTTP_SERVER_FLAG_KEEPALIVE (((uint64_t) 1) << 0)

[engine] caught signal (SIGSEGV)
#0  0x615705ba2a2a      in  flb_hash_table_add() at src/flb_hash_table.c:308
#1  0x61570616195e      in  flb_http_request_set_header() at src/flb_http_common.c:131
#2  0x615706188e8f      in  http1_session_process_request() at src/http_server/flb_http_server_http1.c:228
#3  0x615706189553      in  flb_http1_server_session_ingest() at src/http_server/flb_http_server_http1.c:491
#4  0x615706184997      in  flb_http_server_session_ingest() at src/http_server/flb_http_server.c:725
#5  0x6157061839b4      in  flb_http_server_session_read() at src/http_server/flb_http_server.c:126
#6  0x615706183f3b      in  flb_http_server_client_activity_event_handler() at src/http_server/flb_http_server.c:339
#7  0x615705ae57a7      in  flb_engine_start() at src/flb_engine.c:1041
#8  0x615705a7f381      in  flb_lib_worker() at src/flb_lib.c:674
#9  0x7fa5ebc65559      in  ???() at ???:0
#10 0x7fa5ebce2a5b      in  ???() at ???:0
#11 0xffffffffffffffff  in  ???() at ???:0
zsh: IOT instruction (core dumped)  bin/fluent-bit -vv -H -i http -o stdout