request help: ip changes, but apisix still requests the old address

[ffch]

Description

• 10.167.109.79:8766 , this is the old ip address，the new is 10.167.109.185:8766.
• When I requests the domain xxxx.com, the access.log still shows the old ip address"GET / HTTP/1.1" 200 903 0.008 "-" "python-requests/2.20.1" 10.167.109.79:8766 200 0.009 "
• The node information of etcd has been displayed as a new address(10.167.109.185:8766).
• Then I deleted the route and created a new one, but it didn't work, the access.log still shows the old ip address.
• I have no choice but to restart apisix.
• I have tried several times and it still has this problem

What is the reason for this problem? Is there a way to solve it?

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[tokers]

1. You should tell us how you update the upstream node.
2. You should add the APISIX version so that we can analyze if this is a bug introduced in an old version.
3. Show the minimum reproducible steps if possible.

[ffch]

1. APISIX version is 2.13.0, etcd version is 3.4.0.
2. I change the upstream nodes by setting discovery_type to zookeeper, then zookeeper nodes change, and nodes in etcd also changed, Only apisix is still requesting the old ip address。
3. All I do is update the nodes of zookeeper.

Is there a cache in apisix ? Or cache for a long time？

[tzssangglass]

As soon as the ETCD is updated, then the APISIX data should be updated as well. Can you do a quick verification?
Change the ip directly in ETCD to see if APISIX senses the change and uses the new IP.

[ffch]

I found that when the ip changed, the following error log is printed:

2022/08/05 11:15:24 [error] 61#61: *1995466 [lua] config_etcd.lua:573: failed to fetch data from etcd: failed to check item data of [/SERVICES/apisix/routes] err:property "upstream" validation failed: property "nodes" validation failed: object matches none of the required,  etcd key: /SERVICES/apisix/routes, context: ngx.timer
2022/08/05 11:15:24 [error] 56#56: *2034285 [lua] config_etcd.lua:573: failed to fetch data from etcd: failed to check item data of [/SERVICES/apisix/routes] err:property "upstream" validation failed: property "nodes" validation failed: object matches none of the required,  etcd key: /SERVICES/apisix/routes, context: ngx.timer

and the etcd is

/SERVICES/apisix/routes/419517589346059237
{
	"create_time": 1659581568,
	"host": "st-b.admin.srv",
	"id": "419517589346059237",
	"name": "后台",
	"status": 1,
	"update_time": 1659581568,
	"upstream": {
		"_discovery_type": "zookeeper",
		"_service_name": "admin-web",
		"discovery_type": "zookeeper",
		"keepalive_pool": {
			"idle_timeout": 60,
			"requests": 1000,
			"size": 320
		},
		"nodes": [
			{
				"host": "10.167.37.187",
				"port": 8766,
				"weight": 100
			}
		],
		"pass_host": "pass",
		"scheme": "http",
		"timeout": {
			"connect": 6,
			"read": 6,
			"send": 6
		},
		"type": "roundrobin"
	},
	"uri": "/*"
}

the same etcd route is is fine at startup.

[tokers]

@ffch The nodes and discovery_type fields are exclusive. You should either configure the upstream with explicit node sets or configuring it with service discovery.

[tzssangglass]

In fact, there may be a problem with the format of the nodes in other upstream configurations. You need to double check the formatting of nodes in all configurations.

[ffch]

really thankful，maybe this is the reason（The nodes and discovery_type fields are exclusive），I'm going to check my project。