failed to publish local member to cluster through raft

[rahulbapumore]

What happened?

We are getting following error in etcd deployment continuously and etcd is not coming up, etcd deployed as 3 replicas.
{"caller":"etcdserver/server.go:2075","error":"etcdserver: request timed out","local-member-attributes":"{Name:etcd-0 ClientURLs:[https://etcd-0.etcd.footprint:2379]}","local-member-id":"a691154c733f3752","message":"failed to publish local member to cluster through raft","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"etcd-0"},"publish-timeout":"7s","request-path":"/0/members/a691154c733f3752/attributes","service_id":"etcd","severity":"warning","timestamp":"2023-05-17T14:24:56.527+00:00","version":"1.2.0"}

What did you expect to happen?

etcd should come up and serve the request

How can we reproduce it (as minimally and precisely as possible)?

we are trying to deleted 3 etcd pods multiple times(50 or more ) to reproduce it.

Anything else we need to know?

No response

Etcd version (please run commands below)

bash-4.4$ etcd --version
etcd Version: 3.5.7
Git SHA: 215b53c
Go Version: go1.17.13
Go OS/Arch: linux/amd64
bash-4.4$ etcdctl version
etcdctl version: 3.5.7
API version: 3.5
bash-4.4$

Etcd configuration (command line flags or environment variables)

bash-4.4$ env | grep ETCD
ETCD_INITIAL_CLUSTER_TOKEN=erc
ETCD_MAX_SNAPSHOTS=3
ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379/
ETCD_HEARTBEAT_INTERVAL=100
ETCD_AUTO_COMPACTION_RETENTION=100
ETCD_TRUSTED_CA_FILE=/data/combinedca/cacertbundle.pem
ENTRYPOINT_RESTART_ETCD=true
ETCDCTL_CERT=/run/sec/certs/client/clicert.pem
ETCD_LOG_LEVEL=info
ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380/
ETCD_AUTO_COMPACTION_MODE=revision
ETCD_LOGGER=zap
ETCD_CERT_FILE=/run/sec/certs/server/srvcert.pem
ETCD_PEER_AUTO_TLS=true
ETCD_DATA_DIR=/data
ETCD_CLIENT_CERT_AUTH=true
ETCDCTL_ENDPOINTS=erc.zmorrah:2379
ETCD_METRICS=basic
ETCDCTL_API=3
ETCD_SNAPSHOT_COUNT=5000
ETCD_MAX_WALS=3
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://erc-0.erc-peer.zmorrah.svc.cluster.local:2380/
ETCD_KEY_FILE=/run/sec/certs/server/srvprivkey.pem
ETCD_ELECTION_TIMEOUT=1000
ETCDCTL_CACERT=/data/combinedca/cacertbundle.pem
ETCD_NAME=erc-0
ETCD_QUOTA_BACKEND_BYTES=268435456
ETCD_ADVERTISE_CLIENT_URLS=https://erc-0.erc.zmorrah:2379/
ETCDCTL_KEY=/run/sec/certs/client/cliprivkey.pem

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

etcd pod 0 itself is not starting,
so could not capture

Relevant log output

{"caller":"etcdserver/server.go:2075","error":"etcdserver: request timed out","local-member-attributes":"{Name:erc-0 ClientURLs:[https://erc-0.erc.footprint:2379]}","local-member-id":"a691154c733f3752","message":"failed to publish local member to cluster through raft","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"publish-timeout":"7s","request-path":"/0/members/a691154c733f3752/attributes","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:24:56.527+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:56.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:56.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:56.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:56.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:56.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:57.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:57.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:57.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:57.840+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:57.840+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:59.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:59.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:59.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:59.040+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:24:59.040+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:24:59.505+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:24:59.506+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:24:59.528+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:24:59.528+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:00.239+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:00.239+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:00.239+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:00.239+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:00.239+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:01.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:01.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:01.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:01.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:01.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:02.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:02.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:02.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:02.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:02.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/server.go:2075","error":"etcdserver: request timed out","local-member-attributes":"{Name:erc-0 ClientURLs:[https://erc-0.erc.footprint:2379]}","local-member-id":"a691154c733f3752","message":"failed to publish local member to cluster through raft","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"publish-timeout":"7s","request-path":"/0/members/a691154c733f3752/attributes","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:03.528+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:03.840+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:03.840+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:03.840+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:03.840+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:03.840+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:04.505+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:04.506+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:04.528+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:04.529+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:05.040+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:05.040+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:05.040+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:05.041+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:05.041+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:06.239+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:06.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:06.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:06.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:06.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:07.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:07.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:07.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:07.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:07.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:08.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:08.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:08.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:08.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:08.639+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:09.505+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:09.506+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:09.529+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:09.529+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:09.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:09.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:09.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:09.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:09.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/server.go:2075","error":"etcdserver: request timed out","local-member-attributes":"{Name:erc-0 ClientURLs:[https://erc-0.erc.footprint:2379]}","local-member-id":"a691154c733f3752","message":"failed to publish local member to cluster through raft","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"publish-timeout":"7s","request-path":"/0/members/a691154c733f3752/attributes","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:10.528+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:11.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:11.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:11.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:11.040+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:11.040+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:12.244+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:12.244+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:12.244+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:12.245+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:12.245+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:13.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:13.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:13.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:13.439+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:13.439+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:14.506+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:14.507+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:14.530+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:14.530+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:14.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:14.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:14.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:14.639+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:14.640+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:15.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:15.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:15.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:15.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:15.839+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:17.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:17.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:17.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:17.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:17.039+00:00","version":"1.2.0"}
{"caller":"etcdserver/server.go:2075","error":"etcdserver: request timed out","local-member-attributes":"{Name:erc-0 ClientURLs:[https://erc-0.erc.footprint:2379]}","local-member-id":"a691154c733f3752","message":"failed to publish local member to cluster through raft","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"publish-timeout":"7s","request-path":"/0/members/a691154c733f3752/attributes","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:17.529+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:18.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:18.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:18.240+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:18.241+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:18.241+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 is starting a new election at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:19.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 became pre-candidate at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:19.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 received MsgPreVoteResp from a691154c733f3752 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:19.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to 12f5a7d62234b587 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:19.440+00:00","version":"1.2.0"}
{"caller":"etcdserver/zap_raft.go:77","logger":"raft","message":"a691154c733f3752 [logterm: 18, index: 364292] sent MsgPreVote request to d238741d6ba13917 at term 18","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"service_id":"erc","severity":"info","timestamp":"2023-05-17T14:25:19.440+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:19.506+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:19.508+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:19.530+00:00","version":"1.2.0"}

[rahulbapumore]

@chaochn47 @ahrtr

[chaochn47]

etcd pod 0 itself is not starting

You probably want to check the DNS set up because dns lookup failed ==

{
    "caller": "rafthttp/probing_status.go:68",
    "error": "dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable",
    "message": "prober detected unhealthy status",
    "metadata": {
        "container_name": "dced",
        "namespace": "footprint",
        "pod_name": "erc-0"
    },
    "remote-peer-id": "d238741d6ba13917",
    "round-tripper-name": "ROUND_TRIPPER_SNAPSHOT",
    "rtt": "0s",
    "service_id": "erc",
    "severity": "warning",
    "timestamp": "2023-05-17T14:25:19.530+00:00",
    "version": "1.2.0"
}

[rahulbapumore]

Hi @chaochn47 ,
Actually this error of lookup failure is getting printed continuously in the loop.
Despite network issue was solved still ETCD was printing that error message in loop.
Only way to recover from this situation was killing ETCD manually.
So Does this mean that ETCD once goes into this DNS lookup failure then there is no way that ETCD recovers automaticaly?
Ideally ETCD should be able to resolve DNS query in later tries, but here it is printing same error message again and again.
Could you please let us know that this is expected behavior of ETCD?

Thanks in advance

[rahulbapumore]

Hi Team,
Any thoughts on above comment?

[rahulbapumore]

@chaochn47 @ahrtr

[rahulbapumore]

Hi Team,
Any view on above comment?

[serathius]

cc @jmhbnz

[chaochn47]

Despite network issue was solved still ETCD was printing that error message in loop.

Could you please provide evidence that network issue was resolved? What was broken in the network set up and what is the fix?

Ideally ETCD should be able to resolve DNS query in later tries, but here it is printing same error message again and again.
Could you please let us know that this is expected behavior of ETCD?

Yes, it's the expected behavior of etcd. For each pair of peer and round-triper raft http has a prober to probe if the network from pod erc-0 to the peer is up every proberInterval which is 4s.

	ticker := time.NewTicker(probingInterval)

	go func() {
		pinned := 0
		for {
			select {
			case <-ticker.C:
				start := time.Now()
				req, err := http.NewRequest("GET", endpoints[pinned], nil)
				if err != nil {
					panic(err)
				}
				resp, err := p.tr.RoundTrip(req)
				if err == nil && resp.StatusCode != http.StatusOK {
					err = fmt.Errorf("got unexpected HTTP status code %s from %s", resp.Status, endpoints[pinned])
					resp.Body.Close()
				}
				if err != nil {
					s.recordFailure(err)
					pinned = (pinned + 1) % len(endpoints)
					continue
				}

				var hh Health
				d := json.NewDecoder(resp.Body)
				err = d.Decode(&hh)
				resp.Body.Close()
				if err != nil || !hh.OK {
					s.recordFailure(err)
					pinned = (pinned + 1) % len(endpoints)
					continue
				}

				s.record(time.Since(start), hh.Now)

https://github.com/xiang90/probing/blob/master/prober.go#L51-L84

If the network issue is resolved, then monitorProbingStatus should not continuously report DNS look up failures every 5s. However, from the log you provided, the statement of "the network issue is resolved" is not true.

Not to mention local member a691154c733f3752 never received MsgPreVoteResp from peer, stuck in term 18 and not able to tranform to candidate state. Those are expected as part of pre-vote feature to prevent network isolated member to abrupt active leader.

[rahulbapumore]

Hi @chaochn47 ,
I agree that it was printing below logs again and again but its also true that we just deleted pod and everything came up.
We did not fix anything in the network, but as you said something wrong with the network and we just deleted pod and all pods came up(etcd came up). From this we are saying that network issue was resolved.
See the scenario, etcd was not coming up properly and printing logs of DNS lookup failed continuously, and suddenly we deleted pod and whole deployment came up and etcd started again. So even if we consider there was some network issue for small period of time and etcd went into the loop of printing DNS lookup failed error. And once we deleted pod(inside which etcd is running), everything came up and etcd was up and running.
So my question is that whether etcd goes into loop even if it once face DNS lookup issue? Because if it were issue with network then immediately after deleting pod/etcd, deployment wouldn;'t have come up. But everything etcd was up and running once we restarted the pod.
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:19.506+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-1.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"12f5a7d62234b587","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:19.508+00:00","version":"1.2.0"}
{"caller":"rafthttp/probing_status.go:68","error":"dial tcp: lookup erc-2.erc-peer.footprint.svc.cluster.local on 10.96.0.10:53: dial udp 10.96.0.10:53: connect: network is unreachable","message":"prober detected unhealthy status","metadata":{"container_name":"dced","namespace":"footprint","pod_name":"erc-0"},"remote-peer-id":"d238741d6ba13917","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","rtt":"0s","service_id":"erc","severity":"warning","timestamp":"2023-05-17T14:25:19.530+00:00","version":"1.2.0"}

[chaochn47]

If you delete a pod and a new pod is scheduled to a new node with no network issues, the issue is likely to be resolved.

[chaochn47]

Thank you for your question, this support issue will be moved to our Discussion Forums.

We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases.

We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on GitHub is making it difficult for us to use issues to identify real bugs.

Members of the etcd community use Discussion Forums to field support requests. Before posting a new question, please search these for answers to similar questions, and also familiarize yourself with:

    [user documentation](https://etcd.io/docs/latest)
    [frequently asked questions](https://etcd.io/docs/v3.5/faq)

Again, thanks for using etcd and raising this question.

The etcd team