bug: Apisix forwarded traffic to the wrong enpodints in gRPC mode.

[nisainan]

Current Behavior

By observing the access log, the upstream endpoint displayed by apisix does not actually exist, and it is inconsistent with what is shown on the dashboard.

Expected Behavior

It should be routed to the correct endpoints

Error Logs

10.119.181.176:8088, 10.119.95.95:8088 , These two endpoints do not actually exist. I don't know why apisix forwarded it to these two endpoints. Below is the actual ednpints：

{
  "nodes": [
    {
      "host": "10.119.172.172",
      "port": 8088,
      "weight": 100
    },
    {
      "host": "10.119.105.172",
      "port": 8088,
      "weight": 100
    },
    {
      "host": "10.119.65.6",
      "port": 8088,
      "weight": 100
    },
    {
      "host": "10.119.253.188",
      "port": 8088,
      "weight": 100
    }
  ],
  "type": "roundrobin",
  "hash_on": "vars",
  "scheme": "grpc",
  "pass_host": "pass",
  "name": "xxxx",
  "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
  "labels": {
    "managed-by": "apisix-ingress-controller"
  }
}

This anomaly occurs in clusters every 1-2 hours.

Steps to Reproduce

just proxy grpc

Environment

• APISIX version (run apisix version): 3.13
• Operating system (run uname -a): debian
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

Hi @nisainan, could you please provide your ingress controller version and related route configuration? Are there any related error logs in the ingress controller or APISIX?