Fix bug with accessing already released connection

[EvgeniiMekhanik]

There was a problem with accessing already released connection when we send error response to client,
after error in processing http response. Reponse
processing can occurs on different cpus (moreover
we can process several responses in parallel on
several cpus). Problem we fail to process two or
more responses in parallel.
We call tfw_h2_error_resp->tfw_h2_send_err_resp
and free request inside tfw_h2_send_err_resp
and also decrement connection reference counter
because of it. Also we call tfw_h2_conn_terminate_close and send FIN to the client. If FIN from the client comes immediatly ss_conn_drop_guard_exit will
be called on another cpu in parallel. For one
response everything is ok, we drop connection
in ss_conn_drop_guard_exit and release it, but
if we also call tfw_h2_error_resp for second response we release connection inside tfw_h2_send_err_resp after sending response and access already deleted connection in tfw_h2_conn_terminate_close!

• To fix this problem we should increment connection reference counter before tfw_h2_error_resp and
  release it after ss_conn_drop_guard_exit.
• Also we have a problem with sk->security - we release this pointer in ss_conn_drop_guard_exit but this function can be called when connection stil has active processed responses and we can pass this responses to frang. So we should release sk->security when we totally release connection.

[EvgeniiMekhanik]

This PR fixes #2178 and possible fixes #2169 (need check)

[krizhanovsky]

The fix looks good for me, but why do we face a case when the same upstream connection is used by many CPUs concurrently? Didn't we recently fix the problem with TCP connections CPU locality , even for the loopback interface?

[EvgeniiMekhanik]

Thanks to @kingluo he pay my attention on our private discussion, that we also have a problem with tfw_cli_conn_send. I implement new commit to fix it. @kingluo please look my PR and if you see some other problem places please write me about it here!

[kingluo]

LGTM. Thanks.

[keshonok]

I understand the reason for the move, yet I wonder if there's a way to leave this function where it was, together with the opposite function. If there was it would be preferable.

[EvgeniiMekhanik]

I think about it, but we need to tfw_classify_conn_close, which defined in http_limits.h. http_limits.h include connection.h, so I don't see any way to stay it in header except forward tfw_classify_conn_close declaration. I prefer move this function to connection.c

[keshonok]

Please allow me to rephrase a little bit. :-) You are free to use this any way you want.

msg can be freed in ss_do_send() callback. If it's the last msg for this connection the ref counter becomes equal to 1. However if the client closes the connection and sends FIN, the connection ref counter is decremented to zero and the connection is freed. To avoid that, increment the connection ref counter here to guarantee access to cli_conn after sending data.

[EvgeniiMekhanik]

Yes fixed.

[EvgeniiMekhanik]

The fix looks good for me, but why do we face a case when the same upstream connection is used by many CPUs concurrently? Didn't we recently fix the problem with TCP connections CPU locality , even for the loopback interface?

Here we access client connection when we process response! Response processing can occurs and occurs on different CPU, because server socket is not a client socket!

[EvgeniiMekhanik]

It seems my second commit was wrong. We call tfw_cli_conn_send from two places:
tfw_h2_resp_fwd and we get connection before call tfw_cli_conn_send in this function!
__tfw_http_resp_fwd which is called for http1 only where we have no skb destructor and where we never destroy message on other cpu

[const-t]

LGTM