bug: One of the three etcd nodes is broken (once every two weeks)

[Martinprobu]

Current Behavior

One of the three etcd nodes is broken (once every two weeks)

Expected Behavior

Three nodes should be operating normally

Error Logs

Steps to Reproduce

1.insatll the etcd cluster:
helm upgrade etcd-release --install --create-namespace --namespace etcd --version v8.5.11 --values apisix/etcd.yaml etcd-repo/etcd
2.install the apisix:
helm upgrade apisix-release --install --create-namespace --namespace apisix --version v0.11.4 --values apisix/apisix-new.values.yml apisix/apisix

Environment

• APISIX version (run apisix version): 2.15.0
• Operating system (run uname -a): Linux etcd-release-2 5.4.0-1089-azure
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.21.4.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): helm.sh/chart:etcd-8.5.11
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version): 3.8.0

[membphis]

@nic-chen do you have time to take a look at this issue?

[Martinprobu]

@nic-chen @membphis do you need any other information ? or need a calling? We're waiting online

[nic-chen]

hi, @Martinprobu

Could you upload more logs? The current logs posted do not have any helpful information.

By the way, what is your k8s runtime? Containerd or Docker

[Martinprobu]

k8s version : 1.24.3
in azure aks.

[Martinprobu]

this is apisix logs
apisix_20230106A_query_data.csv
(24hours).

this is etcd logs (124 houres)
etcd_20230106A_query_data.csv

[nic-chen]

this is apisix logs apisix_20230106A_query_data.csv (24hours).

this is etcd logs (124 houres) etcd_20230106A_query_data.csv

Looks like it has nothing to do with APISIX.
Since your situation is periodic, I suggest you to check your environment to see if there are some timed tasks or some timed events.
And you can refer to the following issues:

etcd-io/etcd#13670
etcd-io/etcd#14542
apache/apisix-helm-chart#282

[MarkCupitt]

I have actually noticed this in Kubernetes V1.26 .. apache/apisix:3.1.0-debian installed vial Helm Chart its not easily reproducible

[papdaniel]

We have the same issue in EKS (kubernetes version v1.24.7), apisix version 0.11.5. Crashes every second weeks. It is interesting that it starts to crash at around 5 am, as it mentioned here: etcd-io/etcd#14542

[ecoupal-believe]

Samething on EKS 1.23, not sure how to recover from this, any idea ?

[ecoupal-believe]

I found a workaround. I installed an external etcd cluster using the latest bitnami chart and set "removeMemberOnContainerTermination" to false. To work persistence must be enabled and you will also need to configure etcd of the apisix side.

[Martinprobu]

I found a workaround. I installed an external etcd cluster using the latest bitnami chart and set "removeMemberOnContainerTermination" to false. To work persistence must be enabled and you will also need to configure etcd of the apisix side.

Is it still valid until now ?