bug: Error: etcdserver: member not found

[sudhirs649]

Current Behavior

We are getting this error even though we upgraded the etcd 8.7 to etcd 9.7 but still facing the same error.
Our apisix pod image tag: 3.8.0-debian

Expected Behavior

all apsisix-etcd pod needs to be in running status

Error Logs

etcd 05:04:16.59
etcd 05:04:16.74 Welcome to the Bitnami etcd container
etcd 05:04:16.75 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 05:04:16.77 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 05:04:16.78
etcd 05:04:16.79 INFO ==> ** Starting etcd setup **
etcd 05:04:17.35 INFO ==> Validating settings in ETCD_* env vars..
etcd 05:04:17.37 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 05:04:17.55 INFO ==> Initializing etcd
etcd 05:04:17.66 INFO ==> Generating etcd config file using env variables
etcd 05:04:18.47 INFO ==> Detected data from previous deployments
etcd 05:04:20.50 INFO ==> Updating member in existing cluster
{"level":"warn","ts":"2024-07-02T05:04:22.015Z","logger":"etcd-client","caller":"v3@v3.5.7/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00055c000/accel-apisix-etcd-0.accel-apisix-etcd-headless.accel-apisix.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found

Steps to Reproduce

We have deployed through helm charts we are only using etcd as a dependency chart

Environment

• APISIX version (run apisix version)--> 3.8.0-debian
• ETCD package version --> etcd-9.7.3.tgz , image version docker.io/bitnami/etcd:3.5.7-debian-11-r14
• Operating system (run uname -a): linux , Kernel version 5.15.0-1061-azure
• OpenResty / Nginx version (run openresty -V or nginx -V): helm.sh/chart:ingress-nginx-4.10.0
• APISIX Dashboard version, if relevant: 3.0.0-alpine
• AKS cluster version - 1.28

[shreemaan-abhishek]

closing this because this is an etcd issue, not apisix. If not, please provide substantial evidences.

[sudhirs649]

@shreemaan-abhishek We are using APISIX incorporated ETCD, which is present APISIX helmet chart https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix/charts/etcd-9.7.3.tgz.

So it is only a part of APISIX. Please at least guide us on this issue

[personage-hub]

Faced with same issue.

[Joeydelarago]

I'm experiencing this issue with the latest chart on GKE.

I don't have a permanent solution, but this quick fix solves it temporarily:

1. Scale the stateful set down to 2
2. Delete the PVC data-apisix-etcd-X, where X is the number of the etcd failing
3. Scale back up to 3

Edit:
Doing a helm uninstall, deleting the PVCs and reinstalling also fixes the issue for me. I installed etcd seperately as described here to do this without taking down apisix.