Unexpected Behavior When Shutting Down etcd Node – Raft Message Drop and Cluster Instability

[Kroseida]

Hi etcd team,

I'm running a 3-node etcd cluster and noticed some behavior that I'm not sure is expected. I'd appreciate your feedback on whether this is normal or if I'm missing something.

Problem

When I fully shut down one of the etcd nodes (i.e., the whole machine is powered off or unreachable at the network level), I see the following error repeatedly in a very short period:
dropped internal Raft message since sending buffer is full (overloaded network)

During this time, the cluster seems unstable and not fully functional.

However, if I only stop the etcd service on the node (but leave the machine itself reachable on the network), everything behaves normally—no errors, and the cluster stays "healthy", one node degraded. If I stop the etcd service on the node and later on shutdown the machine, the same errors occurs.

This seems counterintuitive: I would expect both cases to degrade the cluster in the same way (1 node down out of 3), but apparently the fully unreachable node causes more trouble.

Cluster Setup

I'm running 3 etcd nodes with the following configuration on each (example from one node):

ETCD_NAME=daystrom
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=http://172.25.10.10:2379,http://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=http://172.25.10.10:2380
ETCD_ADVERTISE_CLIENT_URLS=http://172.25.10.10:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.25.10.10:2380
ETCD_INITIAL_CLUSTER=daystrom=http://172.25.10.10:2380,memoryalpha=http://172.25.10.11:2380,z15=http://172.25.10.12:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster

ETCD_HEARTBEAT_INTERVAL=1000
ETCD_ELECTION_TIMEOUT=5000
ETCD_PEER_DIAL_TIMEOUT=5s
ETCD_PEER_READ_TIMEOUT=5s
ETCD_PEER_WRITE_TIMEOUT=5s
ETCD_CLIENT_READ_TIMEOUT=5s
ETCD_CLIENT_WRITE_TIMEOUT=5s

System:

CPU: Intel(R) N150
RAM: 16gb DD4
STORAGE: 512gb NVME
NET: 10GBit/s 

OS:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

Etcd:

Raw install on bare-metal Server
etcd Version: 3.5.0
Git SHA: 946a5a6f2
Go Version: go1.16.3
Go OS/Arch: linux/amd64

Is this behavior expected when a node is completely unreachable at the network level?
Should I tune any parameters to make the cluster more tolerant of such node failures?

Thanks in advance for your help!

Best regards,

[jmhbnz]

Hi @Kroseida - Thanks for your question. It looks like you are running etcd v3.5.0 which is a four year old release.

That version is not recommended to be run in production please refer to https://github.com/etcd-io/etcd/tree/main/CHANGELOG#v35-data-corruption-issue

We have fixed countless bugs and security issues in later versions of v3.5. Please upgrade to a later patch release as soon as possible and then report back if this issue persists, thanks.

[Kroseida]

Hi @jmhbnz,

Thanks for the heads-up! I went ahead and updated etcd to a more recent version, and that seems to have resolved the issue. The cluster remains stable now, even when a node is completely unreachable.

Appreciate for the support!

Best regards,
Kroseida