Addressing 'Upstream Not Found' Warning in APISIX

[rkazak07]

Issue description

Hello,
I installed Apisix. Even though I did not set up anything inside , I always get the following warnings. When I examined it in Debug mode, I saw that it was plugged into the Rabbitmq service ports running inside. I added these ports as upstream. Warnings have been stopped, but it sometimes sends warnings for different applications like redis, kafka... How can I proceed with the solution?
When I include the application with Apisix ingress, it works stably.

Note: I installed Apisix as daemonset

2024-04-07T18:36:43+08:00 warn apisix/cluster.go:1096 route not found {"id": "fb148866", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/fb148866", "cluster": "default"} 2024-04-07T19:54:01+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T19:54:01+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T14:59:10.711119455+03:00 2024-04-07T19:59:10+08:00 warn apisix/cluster.go:1096 route not found {"id": "1be8a9cd", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/1be8a9cd", "cluster": "default"} 2024-04-07T20:00:36+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T15:00:36.322330526+03:00 2024-04-07T20:00:36+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T20:00:37+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "b785aa59", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b785aa59", "cluster": "default"} 2024-04-07T20:00:37+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "c0829acf", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/c0829acf", "cluster": "default"} 2024-04-07T20:01:43+08:00 warn apisix/cluster.go:1096 route not found {"id": "19221c15", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/19221c15", "cluster": "default"} 2024-04-07T15:02:43.559923042+03:00 2024-04-07T20:02:43+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T15:02:43.719498269+03:00 2024-04-07T20:02:43+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T15:02:45.045476422+03:00 2024-04-07T20:02:45+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "b785aa59", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b785aa59", "cluster": "default"} 2024-04-07T20:02:45+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "c0829acf", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/c0829acf", "cluster": "default"} 2024-04-07T15:03:15.520423951+03:00 2024-04-07T20:03:15+08:00 warn apisix/cluster.go:1096 route not found {"id": "6dd4f7e8", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/6dd4f7e8", "cluster": "default"} 2024-04-07T15:04:15.849063365+03:00 2024-04-07T20:04:15+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"bff-tapp\": the object has been modified; please apply your changes to the latest version and try again", "name": "bff-tapp", "namespace": "dev-bff"} 2024-04-07T20:04:15+08:00 warn apisix/cluster.go:1096 route not found {"id": "eec00fc1", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/eec00fc1", "cluster": "default"} 2024-04-07T20:04:53+08:00 warn apisix/cluster.go:1096 route not found {"id": "beecb92d", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/beecb92d", "cluster": "default"} 2024-04-07T15:05:49.522312841+03:00 2024-04-07T20:05:49+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "b785aa59", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b785aa59", "cluster": "default"} 2024-04-07T20:05:49+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "c0829acf", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/c0829acf", "cluster": "default"} 2024-04-07T15:06:46.130431720+03:00 2024-04-07T20:06:46+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"bff-appd\": the object has been modified; please apply your changes to the latest version and try again", "name": "bff-appd", "namespace": "dev-bff"} 2024-04-07T15:06:46.168746503+03:00 2024-04-07T20:06:46+08:00 warn apisix/cluster.go:1096 route not found {"id": "37e34cb7", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/37e34cb7", "cluster": "default"} 2024-04-07T15:08:49.527680464+03:00 2024-04-07T20:08:49+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"ui-atapp\": the object has been modified; please apply your changes to the latest version and try again", "name": "ui-atapp", "namespace": "dev-ui"} 2024-04-07T15:08:49.739153847+03:00 2024-04-07T20:08:49+08:00 warn apisix/cluster.go:1096 route not found {"id": "70110b8a", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/70110b8a", "cluster": "default"} 2024-04-07T15:11:04.884775966+03:00 2024-04-07T20:11:04+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"ui-tapp\": the object has been modified; please apply your changes to the latest version and try again", "name": "ui-tapp", "namespace": "dev-ui"} 2024-04-07T15:11:04.928498466+03:00 2024-04-07T20:11:04+08:00 warn apisix/cluster.go:1096 route not found {"id": "e3bcc2f5", "url": "http://apisix-ad min.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/e3bcc2f5", "cluster": "default"} 2024-04-07T15:13:47.631911060+03:00 2024-04-07T20:13:47+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"security-idp-capp\": the object has been modified; please apply your changes to the latest version and try again", "name": "security-idp-capp", "namespace": "dev-security"} 2024-04-07T20:13:47+08:00 warn apisix/cluster.go:1096 route not found {"id": "b26166dd", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/b26166dd", "cluster": "default"} 2024-04-07T15:15:19.907450226+03:00 2024-04-07T20:15:19+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"security-idp-papp\": the object has been modified; please apply your changes to the latest version and try again", "name": "security-idp-papp", "namespace": "dev-security"} 2024-04-07T15:16:26.397665034+03:00 2024-04-07T20:16:26+08:00 warn apisix/cluster.go:1096 route not found {"id": "321f3c87", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/321f3c87", "cluster": "default"} 2024-04-07T15:22:35.869151820+03:00 2024-04-07T20:22:35+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T15:22:35.888250101+03:00 2024-04-07T20:22:35+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T20:31:10+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T20:31:10+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T20:33:18+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T20:33:18+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"}

Thanks,

Environment

Apisix: 3.8.0
Apisix Ingress Controller: 1.8.0
Apisix Dashboard: 3.0.0
K8S Version: 1.27.10

[github-actions]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.