bug: active healthy check occur error when upstream node is not exist

[jujiale]

Current Behavior

when have seveval upstream nodes and open active healthy check with tcp way. if one node does not exist(such as the node machine shutdown ). healthy check will not remove the node from upstream nodes.

Expected Behavior

even if the node is not exist. it also should be remove from the upstream nodes config table.

Error Logs

log(): [healthcheck] (upstream#/xxxx/xxx/upstreams/426538814077405003) unhealthy TIMEOUT increment (1/3) for '([172.xx.xx.xx:20020]'

Steps to Reproduce

1. add a route with three upstream nodes, such as A, B, C. add active healthycheck with tcp way.
2. two upstream nodes is ok, such as A, B, the thrid node（C） is not exist.(in here. not exist means the machine is shutdown. or is not exist. not the port's process is down )
3. send request to this route. it also hit the node C.
4. when invoke http://ip:port/v1/healthcheck. the return includes the node C

Environment

• APISIX version (run apisix version):2.12.0
• Operating system (run uname -a):centos7: 3.10.0-957.21.3.el7.x86_64
• OpenResty / Nginx version (run openresty -V or nginx -V):openresty/1.21.4.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):3.5.0
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Fijileijun]

Added More Error Logs：
As the same time， apisix output following error logs every 2 hours:
trying to increment a target that is not in the list ：172.xx.xx.xx:20020

[kingluo]

@jujiale
Please show your upstream configuration.
If the checker failed to connect the unreachable node (TIMEOUT error) more than upstream.checks.active.unhealthy.timeouts times, it will be marked as unhealthy. If so, future requests will not be forwarded to that node, and in the response of /v1/healthcheck, it will be not in the healthy_nodes array, but it will persist in the nodes array because it's not actually removed from the upstream configuration via admin API. It's two different things.

[jujiale]

duplicate with #10500