Handle idle connections #137

vasiliy-t · 2021-05-21T12:48:43Z

Problem statement

We have http server exposing metrics to telegraf. Telegraf scrapes metrics in 60 second intervals via http: it opens keep-alive connections but under certain conditions (have no repro, only in prod) it does not reuses established connections and opens new connection each time it scrapes metrics. This results in increasing number of open connections, every connection if a fiber which lives forever.

Possible solution

Introduce new http.server config options:

maximum number of idle connections, after reaching this limit http.server could start closing idle conns
idle timeout is the maximum amount of time an idle (keep-alive) connection will remain idle before closing

kyukhin · 2021-06-11T10:36:17Z

This is supposed to be fixed in httpng.

vasiliy-t · 2021-10-13T17:03:31Z

@kyukhin I did not agree with "wishlist". Lack of idle connections timeout has been exploding every cluster our customer have for a week and that's a lot of clusters. And the only solution we have is to configure Telegraf and have no way to protect from within tarantool. If it will be fixed in httpng then it must be ported to http - we have no resources to migrate every application we have to new http module.

Totktonada · 2021-10-14T12:23:51Z

It is in our backlog now. Further prioritization may be done via the product team.

ligurio · 2021-10-15T19:48:00Z

It is possible to set a timeout in a keep-alive header.
https://tools.ietf.org/id/draft-thomson-hybi-http-timeout-01.html#rfc.section.2
Now http module set keep-alive header without timeout, see https://github.com/tarantool/http/blob/master/http/server/init.lua#L238-L260

By default http module set keep-alive option in connections to make it persistent: $ tarantool examples/middleware.lua started entering the event loop HEAD / ... $ curl -I http://127.0.0.1:8080 HTTP/1.1 404 Not found Content-length: 0 Server: Tarantool http (tarantool v1.10.6-0-g5372cd2fa) Connection: keep-alive $ However sometimes we need persistent connection and close it instead. It is possible with option keepalive_disable. Option accepts a map where key is a user agent name with assigned non-nil value. Example: local httpd = http_server.new('127.0.0.1', 8080, { log_requests = true, log_errors = true, keepalive_disable = { ['curl/7.68.0'] = true }, }) 1. https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_disable Fixes #137

By default http module set keep-alive option in connections to make it persistent: $ tarantool examples/middleware.lua started entering the event loop HEAD / ... $ curl -I http://127.0.0.1:8080 HTTP/1.1 404 Not found Content-length: 0 Server: Tarantool http (tarantool v1.10.6-0-g5372cd2fa) Connection: keep-alive $ However sometimes we need persistent connection and close it instead. It is possible with option keepalive_disable. Option accepts a map where key is a user agent name with assigned non-nil value. Note: nginx web-server has a similar option, see [1]. Unfortunately http client [1] builtin into Tarantool supports only HTTP protocol version 1.1, so version 1.0 is untested. Example: local httpd = http_server.new('127.0.0.1', 8080, { log_requests = true, log_errors = true, keepalive_disable = { ['curl/7.68.0'] = true }, }) 1. https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_disable 2. https://www.tarantool.io/en/doc/latest/reference/reference_lua/http/ Fixes #137

Part of #137

By default http module set keep-alive option in connections to make it persistent. However sometimes we need persistent connection and close it instead. It is possible with option keepalive_disable. Option accepts a map where key is a user agent name with assigned non-nil value. Note: nginx web-server has a similar option, see [1]. Unfortunately http client [1] builtin into Tarantool supports only HTTP protocol version 1.1, so version 1.0 is untested. Example: local httpd = http_server.new('127.0.0.1', 8080, { log_requests = true, log_errors = true, keepalive_disable = { ['curl/7.68.0'] = true }, }) 1. https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_disable 2. https://www.tarantool.io/en/doc/latest/reference/reference_lua/http/ Fixes #137

Part of #137

By default http module set keep-alive option in connections to make it persistent. However sometimes we need persistent connection and close it instead. It is possible with option keepalive_disable. Option accepts a map where key is a user agent name with assigned non-nil value. Note: nginx web-server has a similar option, see [1]. Unfortunately http client [1] builtin into Tarantool supports only HTTP protocol version 1.1, so version 1.0 is untested. Example: local httpd = http_server.new('127.0.0.1', 8080, { log_requests = true, log_errors = true, keepalive_disable = { ['curl/7.68.0'] = true }, }) 1. https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_disable 2. https://www.tarantool.io/en/doc/latest/reference/reference_lua/http/ Fixes #137

Part of #137

By default http module set keep-alive option in connections to make it persistent. However sometimes we need persistent connection and close it instead. It is possible with option keepalive_disable. Option accepts a map where key is a user agent name with assigned non-nil value. Note: nginx web-server has a similar option, see [1]. Unfortunately http client [1] builtin into Tarantool supports only HTTP protocol version 1.1, so version 1.0 is untested. Example: local httpd = http_server.new('127.0.0.1', 8080, { log_requests = true, log_errors = true, keepalive_disable = { ['curl/7.68.0'] = true }, }) 1. https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_disable 2. https://www.tarantool.io/en/doc/latest/reference/reference_lua/http/ Fixes #137

By default http module set keep-alive option in connections to make it persistent. However sometimes we need persistent connection and close it instead. It is possible with option keepalive_disable. Option accepts a map where key is a user agent name with assigned non-nil value. Note: nginx web-server has a similar option, see [1]. Unfortunately http client [1] builtin into Tarantool supports only HTTP protocol version 1.1, so version 1.0 is untested. Example: local httpd = http_server.new('127.0.0.1', 8080, { log_requests = true, log_errors = true, keepalive_disable = { ['curl/7.68.0'] = true }, }) 1. https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_disable 2. https://www.tarantool.io/en/doc/latest/reference/reference_lua/http/ Part of #137

The main problem with current implementation of HTTP server is it's cooperative scheduling. To control connections we need invent something that will periodically checks open connections and close idles with exceeded timeout. Before start processing a new connection we close idle connections with exceeded idle_timeout. In comparison to other solutions (like running checking function in a separate fiber) closing connections synchronously is much easier to implement and don't make module complex. Fixes #137

The main problem with current implementation of HTTP server is it's cooperative scheduling. To control connections we need invent something that will periodically checks open connections and close idles with exceeded timeout. In proposed implementation we close idle connections with exceeded idle_timeout in synchronous manner - before start processing a new connection. In comparison to other solutions (like running checking function in a separate fiber) closing connections synchronously is much easier to implement and don't make module complex. Fixes #137