Requests queueing if there is no backend connection

[krizhanovsky]

tfw_sched_get_srv_conn() call in tfw_http_req_cache_cb() assumes that it's quite improbable that there is no connection to any of the backends, so if all the backend simultaneously reset connections Tempesta returns 502 response code. However, the recent tests show that the state is usual and must be correctly handled.

Firstly, Tempesta FW must use the same rescheduling mechanism, with requests eviction by the timeout, as for rescheduled requests that were ever sent to a serer. Secondly, a /proc/tempesta/servers/%group_name%/perfstat counter must be introduced.

Last, current rescheduling mechanism loops in trying a new server/connections for each request to be rescheduled, however at the point it knows precisely that there are no live connections. Instead it should wait until a request timer elapses or a new server connection established. In the first case a request must be deleted. In the second the connection should be tried, but gracefully, not for all the pending requests at once.

Somewhat linked with #687 since the message queues must be adjusted.

Maybe makes sense to implement #1454 before to facilitate testing and debugging of the current task.

Probably the problem can be solved with dynamic allocation a new server connection if all the current connections are busy #710 and handle the current static number of connections as a minimal provision.

[krizhanovsky]

No one well-doing HTTP proxy sends 50x if there is no backend connections, instead all of them, being TCP socket users, just naturally reduce client requests rate. Whereas Tempesta FW, being a part of the TCP/IP stack, handles all client requests replying with 50x or responses received from a backend servers. Thus, Tempesta FW must slow down client request rates and accurately process all the requests w/o errors. Consequently, the counter mentioned above just has no sense. See discussions: #488 (comment) #1012 (comment)

[krizhanovsky]

#702 is a test for the issue and there is pull request with some work on this #906 . Please finish the PR as well.

[krizhanovsky]

Browsers automatically retry safe request on receiving the 502 error code, so the issue concerns only non-safe requests being sent under too heavy load and/or failing backends. This is system administrator ability to disable connection resets on backend side. Thus, I decrease the task severity.