ngx_http_upstream_check_module导致recv() failed (104: Connection reset by peer)

[mjwtc0722]

upstream配置：check interval=10000 rise=2 fall=2 timeout=5000 type=tcp;
nginx错误日志（部分）：
2017/06/16 14:28:54 [error] 5626#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:29:04 [error] 5624#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:29:14 [error] 5630#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:29:24 [error] 5619#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:29:34 [error] 5628#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:29:44 [error] 5629#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:29:54 [error] 5618#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:30:04 [error] 5621#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:30:14 [error] 5616#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:30:24 [error] 5623#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:30:34 [error] 5620#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:30:44 [error] 5631#0: recv() failed (104: Connection reset by peer)
2017/06/16 14:30:54 [error] 5627#0: recv() failed (104: Connection reset by peer)

nginx版本1.10.3
patch版本1.9.2+

[chobits]

这个日志显示读后端数据时，后端的连接异常断开了（发了TCP RST包给你的nginx），具体可以再抓包确认下

tcpdump -i any -s0 -A port <port>

[mjwtc0722]

首先感谢chobits的提示，根据抓包的结果分析，ngx_http_upstream_check_module这个模块，在使用tcp检测后端状态时，只进行了TCP的三次握手，没有主动断开这个连接，而是等待服务端来断开。当后端是nginx或者tomcat时（linux上），超时后后端会发fin包关闭这个连接。这个错误日志recv() failed (104: Connection reset by peer)是在后端为IIS的情况下抛出的，抓包发现IIS并不会发fin包来断开链接，而是在超时后发RST包重置连接，所以导致了这个问题。
从这个问题也反应出ngx_http_upstream_check_module这个模块还是需要完善下检测机制的，如果是在检测后端状态后主动关闭这个连接，应该就不会出现connect reset这个问题

[mjwtc0722]

通过修改源代码已经解决了该问题
static ngx_check_conf_t ngx_check_types[] = {
{ NGX_HTTP_CHECK_TCP,
ngx_string("tcp"),
ngx_null_string,
0,
ngx_http_upstream_check_peek_handler,
ngx_http_upstream_check_peek_handler,
NULL,
NULL,
NULL,
0,
1 },
将最后一行的1改为0即可，根据数据结构分析可得知，这个1代表启用keepalived，所以客户端才不会主动断开连接，因为这是tcp的端口连通性检查，不需要keepalived，将其改为0禁止keepalived即可
修改之后的代码如下：
static ngx_check_conf_t ngx_check_types[] = {
{ NGX_HTTP_CHECK_TCP,
ngx_string("tcp"),
ngx_null_string,
0,
ngx_http_upstream_check_peek_handler,
ngx_http_upstream_check_peek_handler,
NULL,
NULL,
NULL,
0,
0 },

[chenzhangfeng]

666

[weifan01]

通过修改源代码已经解决了该问题
static ngx_check_conf_t ngx_check_types[] = {
{ NGX_HTTP_CHECK_TCP,
ngx_string("tcp"),
ngx_null_string,
0,
ngx_http_upstream_check_peek_handler,
ngx_http_upstream_check_peek_handler,
NULL,
NULL,
NULL,
0,
1 },
将最后一行的1改为0即可，根据数据结构分析可得知，这个1代表启用keepalived，所以客户端才不会主动断开连接，因为这是tcp的端口连通性检查，不需要keepalived，将其改为0禁止keepalived即可
修改之后的代码如下：
static ngx_check_conf_t ngx_check_types[] = {
{ NGX_HTTP_CHECK_TCP,
ngx_string("tcp"),
ngx_null_string,
0,
ngx_http_upstream_check_peek_handler,
ngx_http_upstream_check_peek_handler,
NULL,
NULL,
NULL,
0,
0 },

👍，完美的解决了我的问题，感谢