When apisix performs routing and forwarding, it fails probabilistically

[liangpeihuahua]

Current Behavior

1. apisix 正常运转一个月后，没有进行任何的重启和路由改变，然发生路由的概率上失败，重点是这些路由全部都跑了一个月，跑得都是没有问题的
2. 具体的报错如图：
   

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1.正常使用突然路由不通，不知道是否是版本有bug, 已经试过好几次了

Environment

• APISIX version (run apisix version): 2.15.0
• Operating system (run uname -a):Linux apisix-c8dfcb955-lbbzd 5.4.241-150.347.amzn2.x86_64 change: added doc of how to load plugin. #1 SMP Mon Apr 24 23:27:26 UTC 2023 x86_64 Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.21.4.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.4
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[yuluo-yx]

Can you provide exact log information? @liangpeihuahua

[liangpeihuahua]

2023/08/23 09:56:55 [error] 47#47: *247095498 [lua] init.lua:540: http_access_phase(): failed to set upstream: no valid upstream node: nil, client: 192.********, server: _, request: "GET /****/originAvatarList?cookie=&pageSize=50&parentType=0&classifyType=4&componentType=0 HTTP/1.1", host: “************.com"

这个是其中的一个报错，但是路由是存在的，并且一个月是跑得好好的，就突然路由失败，报的这个错误

[liangpeihuahua]

然后我尝试重启apisix,重启完后服务就起不来了，报错如下：说明：我们的etcd的版本是3.5.4

kubectl logs -n  apisix-2 apisix-2-55d958b4d9-pdq9p 
Defaulted container "apisix" out of: apisix, wait-etcd (init)
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init

WARNING: using fixed Admin API token has security risk.
Please modify "admin_key" in conf/config.yaml .


/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
etcd cluster version 3.0.0 is less than the required version 3.4.0, please upgrade your etcd cluster

[Revolyssup]

@liangpeihuahua In the logs above, just above the error log there is an info log of a matched route. Can you paste that as well?

[liangpeihuahua]

2023/08/23 09:56:55 [info] 44#44: *247179503 [lua] init.lua:380: http_access_phase(): matched route: {"createdIndex":62,"modifiedIndex":62,"key":"\/apisix\/routes\/429273851986183272","value":{"create_time":1672839187,"upstream":{"pass_host":"pass","nodes":[{"host":"192.168.***","upstream_host":"192.168.****:11002","port":11002,"weight":50,"priority":0}],"keepalive_pool":{"requests":1000,"idle_timeout":60,"size":320},"original_nodes":[{"host":"192.168.*.*","port":11002,"weight":50}],"discovery_type":"kubernetes","timeout":{"connect":6,"read":60,"send":60},"hash_on":"header","key":"uid","scheme":"http","service_name":"bud-abc-server\/bud-ugcmap-abc-svc:11002","type":"chash","parent":{"createdIndex":62,"modifiedIndex":62,"key":"\/apisix\/routes\/429273851986183272","value":"table: 0x7f07e4b9e818","has_domain":false,"update_count":0,"orig_modifiedIndex":62,"clean_handlers":{}},"nodes_ref":"table: 0x7f07e072f7d8"},"labels":{"namespace":"bud-abc-server"},"update_time":1672839187,"host":"abc.abc.com","uris":["\/ugcmap\/*"],"name":"bud-abc-cgi","methods":["GET","POST","HEAD","PUT","PATCH","DELETE"],"priority":0,"id":"429273851986183272","status":1,"plugins":{"cors":{"allow_origins":"*","expose_headers":"*","allow_methods":"*","allow_credential":false,"max_age":5,"allow_headers":"*","disable":false},"request-validation":{"rejected_code":400,"header_schema":{"properties":{},"type":"object"}}}},"has_domain":false,"update_count":0,"orig_modifiedIndex":62,"clean_handlers":"table: 0x7f07e0f36a10"}, client: 192.168.*.*, server: _, request: "GET /ugcmap/originAvatarList?cookie=&pageSize=50&parentType=1&classifyType=18&componentType=0 HTTP/1.1", host: "abc.abc.com"
2023/08/23 09:56:55 [error] 44#44: *247179503 [lua] init.lua:540: http_access_phase(): failed to set upstream: no valid upstream node: nil, client: 192.168.*.*, server: _, request: "GET /ugcmap/originAvatarList?cookie=&pageSize=50&parentType=1&classifyType=18&componentType=0 HTTP/1.1", host: "abc.abc.com"

[Revolyssup]

@liangpeihuahua Looks like an issue with your etcd cluster. You have specified in the issue that your etcd version is 3.5.4 but here the etcd server returns the version 3.0.0 #10077 (comment)
Did at any point your etcd cluster was downgraded to a different version?

[liangpeihuahua]

etcd 的版本从来都没有改变过，也没有升降级，就是自己读取的时候会读出来是3.0.0，实际上就是3.5.4，这个可以确定的

[Revolyssup]

@liangpeihuahua What is your etcd.host in apisix configuration?

When this error takes place, what do you get when you curl http://<host>/version?

[liangpeihuahua]

etcd:
host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
- "http://apisix-etcd.apisix.svc.cluster.local:2379"
prefix: "/apisix" # apisix configurations prefix
timeout: 30 # 30 seconds
discovery:
kubernetes:
service:
host: ${KUBERNETES_SERVICE_HOST}
port: ${KUBERNETES_SERVICE_PORT}
schema: https