apisix upsteam node cannot be deleted dynamically

[flyingfish7]

My configuration information is as follows

apisixroute Configuration:

---

apiVersion: v1
items:

• apiVersion: apisix.apache.org/v1
  kind: ApisixRoute
  metadata:
  creationTimestamp: "2020-07-17T05:51:42Z"
  generation: 7
  name: mall-apisix-ingress
  namespace: mall
  resourceVersion: "2771827"
  selfLink: /apis/apisix.apache.org/v1/namespaces/mall/apisixroutes/mall-apisix-ingress
  uid: 357ffbad-97d1-4f28-b801-247a46b4b226
  spec:
  rules:
  • host: ebs.test.com
    http:
    paths:
    • backend:
      serviceName: ebs-app-rest
      servicePort: 8100
      path: '*'

[root@m1 apisix-ingress]# kubectl describe svc ebs-app-rest -n mall
Name: ebs-app-rest
Namespace: mall
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"ebs-app-rest","namespace":"mall"},"spec":{"ports":[{"name":"http"...
Selector: app=ebs-app-rest
Type: ClusterIP
IP: 10.254.199.108
Port: http 8080/TCP
TargetPort: 8100/TCP
Endpoints: 172.30.88.170:8100
Session Affinity: None
Events:

172.30.88.169,172.30.88.143 is not automatically deleted！
How to solve the problem！

[membphis]

@gxthrj do you have time to look at this?

[gxthrj]

@flyingfish7
1.Please show me the version of APISIX you are using.
2.show me some Related logs in ingress-controller, if you have time.

[flyingfish7]

ingress log :
E0729 00:18:39.513125 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 00:27:23.164890 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 00:35:52.721001 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 00:44:37.609175 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 00:53:13.617555 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:02:06.590928 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:10:51.631760 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:19:34.168028 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:28:05.541115 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:36:59.147975 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:45:56.305179 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 01:54:37.576536 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 02:03:31.655371 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 02:12:40.624479 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.22:8102"}
E0729 02:13:17.171239 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.3:8102"}
E0729 02:13:17.330469 1 builder.go:189] solver upstream failed, patch node info to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes, err: status: 400, body: {"error_msg":"invalid configuration: additional properties forbidden, found 172.30.118.3:8102"}

curl http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000102/nodes

{"node":{"value":{"nodes":{"172.30.118.1:8102":100,"172.30.118.19:8102":100,"172.30.118.3:8102":100,"172.30.118.36:8102":100,"172.30.88.166:8102":100,"172.30.88.153:8102":100,"172.30.88.177:8102":100,"172.30.88.159:8102":100,"172.30.118.6:8102":100,"172.30.88.185:8102":100,"172.30.88.155:8102":100,"172.30.118.55:8102":100,"172.30.88.158:8102":100,"172.30.88.151:8102":100,"172.30.118.48:8102":100,"172.30.118.22:8102":100,"172.30.88.136:8102":100,"172.30.88.179:8102":100,"172.30.118.62:8102":100},"id":"00000000000000000102","hash_on":"vars","key":"/apisix/upstreams/00000000000000000102","desc":"mall_ebs-pc-rest_8102","type":"roundrobin"},"createdIndex":1985,"key":"/apisix/upstreams/00000000000000000102","modifiedIndex":1985},"action":"get"}

apisix version：

ingress version:

[gxthrj]

Currently ingress-controller does not support APISIX v1.4.*, because the version does not support patch upstream nodes directly.
Unfortunately, The APISIX version you are currently using does not support the patch upstream nodes interface, you can downgrade APISIX to v1.3, or wait for the upcoming APISIX v1.5(Will be released in a few days).

[flyingfish7]

I tried apisix1.3 and 1.5 version, but the following error will still appear in ingress!

E0806 02:40:41.250639 1 builder.go:181] solver upstream failed, update upstream to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000086, err: status: 400, body: {"error_msg":"invalid configuration: property "nodes" validation failed: expect object to have at least 1 properties"}
E0806 02:40:41.257964 1 builder.go:181] solver upstream failed, update upstream to etcd failed, err: http patch failed, url: http://apisix-gw-lb:9080/apisix/admin/upstreams/00000000000000000086, err: status: 400, body: {"error_msg":"invalid configuration: property "nodes" validation failed: expect object to have at least 1 properties"}

[membphis]

we can fix this case in APISIX, make it support the number of the upstream node can be 0.

[gxthrj]

If Apache APISIX do not support this feature, we can do some protect.

[gxthrj]

Close this issue now , we have support the feature the number of upstream node can be 0. Feel free to reopen if you have any question.