Skip to content

etcd panic on member rejoin after reset — tocommit(...) is out of range [lastIndex(...)] #20269

New issue
New issue
Open
Open
etcd panic on member rejoin after reset — tocommit(...) is out of range [lastIndex(...)]#20269
Labels
type/bug
@amolmishra23

Description

@amolmishra23
amolmishra23
opened on Jul 3, 2025

Bug report criteria

What happened?

During negative testing of our etcd cluster, we simulate failure scenarios by renaming the etcd binary to something else. Which simulates that etcd member is unhealthy. Our operator then resets the member and attempts to re-add it to the cluster. During this process, etcd crashes with the following panic:

panic: tocommit(10563786) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?

This particular scenario happened while trying to make LEARNER member unhealthy and attempting to re-add it back to the cluster after resetting the configuration of LEARNER member.

etcdserver: restarting local member
raft: <member-id> became follower at term 0
raft: newRaft <member-id> [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft: received a MsgHeartbeat message with higher term from <peer-id> [term: 18]
raft: <member-id> became follower at term 18
raft: panic: tocommit(10563786) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?

stacktrace:
go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf
    etcdserver/zap_raft.go:101
go.etcd.io/etcd/raft/v3.(*raftLog).commitTo
    raft/log.go:237
go.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat
    raft/raft.go:1508
go.etcd.io/etcd/raft/v3.stepFollower
    raft/raft.go:1434
go.etcd.io/etcd/raft/v3.(*raft).Step
    raft/raft.go:975
go.etcd.io/etcd/raft/v3.(*node).run
    raft/node.go:356
created by go.etcd.io/etcd/raft/v3.RestartNode
    raft/node.go:244

What did you expect to happen?

etcd should either:

  1. Recover gracefully and rejoin the cluster, or
  2. Provide a clearer error with guidance on how to safely re-add a member.

How can we reproduce it (as minimally and precisely as possible)?

  1. Stop etcd by renaming the binary (to simulate failure).
  2. Reset the LEARNER member (removing data dir, etc.).
  3. Operator attempts to re-add the LEARNER member to the cluster.
  4. etcd starts and crashes with the panic above.

Anything else we need to know?

  1. Is this behavior expected when a member is reset and re-added without a full cluster reconfiguration?
  2. Are there recommended steps for safely removing and re-adding a member to an existing cluster?
  3. Should we be using etcdctl member remove and etcdctl member add explicitly in this flow?
  4. If we end up in this situation, what is the recommended guideline to come out of it?

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.21
Git SHA: a17edfd
Go Version: go1.23.7
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.18
API version: 3.5

Etcd configuration (command line flags or environment variables)

advertise-client-urls: https://4554913594194925:25687
auto-compaction-mode: periodic
auto-compaction-retention: "1"
cipher-suites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
client-transport-security:
  auto-tls: false
  cert-file: /home/cert/svid.pem
  client-cert-auth: true
  client-cert-file: /home/cert/svid.pem
  client-key-file: /home/cert/svid_key.pem
  key-file: /home/cert/svid_key.pem
  trusted-ca-file: /home/cert/ca.crt
data-dir: /home/cohesity/data/athena/etcd/4554913594194925-2229
initial-advertise-peer-urls: https://4554913594194925:25688
initial-cluster: 4554913383358931-129=https://4554913383358931:25688,4554913925888955-140=https://4554913925888955:25688,4554914012410917-142=https://4554914012410917:25688,4554913594194925-2229=https://4554913594194925:25688
initial-cluster-state: existing
initial-cluster-token: "3450042329922491"
listen-client-urls: https://10.10.20.20:25687
listen-peer-urls: https://10.10.20.20:25688
name: 4554913594194925-2229
peer-transport-security:
  auto-tls: false
  cert-file: /home/cert/svid.pem
  client-cert-auth: true
  client-cert-file: /home/cert/svid.pem
  client-key-file: /home/cert/svid_key.pem
  key-file: /home/cert/svid_key.pem
  trusted-ca-file: /home/cert/ca.crt
strict-reconfig-check: true
tls-max-version: TLS1.3
tls-min-version: TLS1.2

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+-----------------------+--------------------------------+--------------------------------+------------+
|        ID        | STATUS  |         NAME          |           PEER ADDRS           |          CLIENT ADDRS          | IS LEARNER |
+------------------+---------+-----------------------+--------------------------------+--------------------------------+------------+
| 4c1f37b6aaeced42 | started |  4554913383358931-129 | https://4554913383358931:25688 | https://4554913383358931:25687 |      false |
| 6cde776fd87b3644 | started |  4554913925888955-140 | https://4554913925888955:25688 | https://4554913925888955:25687 |      false |
| 9401647e1bff63f3 | started |  4554914012410917-142 | https://4554914012410917:25688 | https://4554914012410917:25687 |      false |
| bf4c5f14c328e1d3 | started | 4554913594194925-2213 | https://4554913594194925:25688 | https://4554913594194925:25687 |       true |
+------------------+---------+-----------------------+--------------------------------+--------------------------------+------------+

$ etcdctl --endpoints=<member list> endpoint status -w table
{"level":"warn","ts":"2025-07-03T00:41:57.444158-0700","logger":"etcd-client","caller":"v3@v3.5.18/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000568000/4554913383358931:25687","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: read tcp 10.136.5.46:35506->10.136.5.46:25687: read: connection reset by peer\""}
Failed to get the status of endpoint https://4554913594194925:25687 (context deadline exceeded)
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|            ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://4554913383358931:25687 | 4c1f37b6aaeced42 |  3.5.21 |   12 MB |      true |      false |        18 |   10608011 |           10608011 |        |
| https://4554913925888955:25687 | 6cde776fd87b3644 |  3.5.21 |   12 MB |     false |      false |        18 |   10608011 |           10608011 |        |
| https://4554914012410917:25687 | 9401647e1bff63f3 |  3.5.21 |   12 MB |     false |      false |        18 |   10608011 |           10608011 |        |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
Error: exit status 1
Usage:
  etcdcli ctl exec -- <etcdctl-subcommand> [flags]

Flags:
  -h, --help             help for exec
      --includeLearner   Include learner endpoint for etcdctl utility

exit status 1

Relevant log output

Jul 03 00:22:45 test-005056895018-node-1 systemd[1]: Starting etcd key-value store...
░░ Subject: A start job for unit etcd-member.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit etcd-member.service has begun execution.
░░
░░ The job identifier is 744969058.
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"warn","ts":"2025-07-03T00:22:45.895309-0700","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.895720-0700","caller":"etcdmain/config.go:367","msg":"loaded server configuration, other configuration command line flags and environment variables will be ignored if provided","path":"/home/test/data/etcd/etcd.conf"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.895742-0700","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/home/test/etcd/v3.5/etcd","--config-file","/home/test/data/etcd/etcd.conf"]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.895803-0700","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/home/test/etcd/4554913594194925-2227","dir-type":"member"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"warn","ts":"2025-07-03T00:22:45.895819-0700","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.895830-0700","caller":"embed/etcd.go:140","msg":"configuring peer listeners","listen-peer-urls":["https://10.136.5.46:25688"]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.895894-0700","caller":"embed/etcd.go:528","msg":"starting with peer TLS","tls-info":"cert = /home/certsvid.pem, key = /home/certsvid_key.pem, client-cert=/home/certsvid.pem, client-key=/home/certsvid_key.pem, trusted-ca = /home/certca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.896910-0700","caller":"embed/etcd.go:148","msg":"configuring client listeners","listen-client-urls":["https://10.136.5.46:25687"]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.897035-0700","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.5.21","git-sha":"a17edfd","go-version":"go1.23.7","go-os":"linux","go-arch":"amd64","max-cpu-set":12,"max-cpu-available":12,"member-initialized":true,"name":"4554913594194925-2227","data-dir":"/home/test/etcd/4554913594194925-2227","wal-dir":"","wal-dir-dedicated":"","member-dir":"/home/test/etcd/4554913594194925-2227/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://4554913594194925:25688"],"listen-peer-urls":["https://10.136.5.46:25688"],"advertise-client-urls":["https://4554913594194925:25687"],"listen-client-urls":["https://10.136.5.46:25687"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"1h0m0s","auto-compaction-interval":"1h0m0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.898930-0700","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/home/test/etcd/4554913594194925-2227/member/snap/db","took":"1.596892ms"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.899536-0700","caller":"etcdserver/server.go:534","msg":"No snapshot found. Recovering WAL from scratch!"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.900126-0700","caller":"etcdserver/raft.go:541","msg":"restarting local member","cluster-id":"7bfb72fdf5963e6f","local-member-id":"bf4c5f14c328e1d3","commit-index":0}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.900187-0700","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"bf4c5f14c328e1d3 switched to configuration voters=()"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.900222-0700","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"bf4c5f14c328e1d3 became follower at term 0"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.900242-0700","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft bf4c5f14c328e1d3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"warn","ts":"2025-07-03T00:22:45.901164-0700","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.901595-0700","caller":"mvcc/kvstore.go:425","msg":"kvstore restored","current-rev":1}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.901643-0700","caller":"etcdserver/server.go:628","msg":"restore consistentIndex","index":0}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.901949-0700","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.902774-0700","caller":"etcdserver/server.go:875","msg":"starting etcd server","local-member-id":"bf4c5f14c328e1d3","local-server-version":"3.5.21","cluster-version":"to_be_decided"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.902903-0700","caller":"etcdserver/server.go:775","msg":"starting initial election tick advance","election-ticks":10}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.902986-0700","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/home/test/etcd/4554913594194925-2227/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.903298-0700","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/home/test/etcd/4554913594194925-2227/member/snap","suffix":"snap","max":5,"interval":"30s"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.903347-0700","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/home/test/etcd/4554913594194925-2227/member/wal","suffix":"wal","max":5,"interval":"30s"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.904184-0700","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.905842-0700","caller":"embed/etcd.go:762","msg":"starting with client TLS","tls-info":"cert = /home/certsvid.pem, key = /home/certsvid_key.pem, client-cert=/home/certsvid.pem, client-key=/home/certsvid_key.pem, trusted-ca = /home/certca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.905966-0700","caller":"embed/etcd.go:633","msg":"serving peer traffic","address":"10.136.5.46:25688"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.905995-0700","caller":"embed/etcd.go:603","msg":"cmux::serve","address":"10.136.5.46:25688"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.906013-0700","caller":"embed/etcd.go:292","msg":"now serving peer/client/metrics","local-member-id":"bf4c5f14c328e1d3","initial-advertise-peer-urls":["https://4554913594194925:25688"],"listen-peer-urls":["https://10.136.5.46:25688"],"advertise-client-urls":["https://4554913594194925:25687"],"listen-client-urls":["https://10.136.5.46:25687"],"listen-metrics-urls":[]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.906171-0700","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"bf4c5f14c328e1d3","remote-peer-id":"4c1f37b6aaeced42"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.906193-0700","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"bf4c5f14c328e1d3","remote-peer-id":"4c1f37b6aaeced42","remote-peer-urls":["https://4554913383358931:25688"]}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.906320-0700","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"bf4c5f14c328e1d3 [term: 0] received a MsgHeartbeat message with higher term from 4c1f37b6aaeced42 [term: 18]"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"info","ts":"2025-07-03T00:22:45.906344-0700","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"bf4c5f14c328e1d3 became follower at term 18"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: {"level":"panic","ts":"2025-07-03T00:22:45.906360-0700","logger":"raft","caller":"etcdserver/zap_raft.go:101","msg":"tocommit(10563786) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf\n\tgo.etcd.io/etcd/server/v3/etcdserver/zap_raft.go:101\ngo.etcd.io/etcd/raft/v3.(*raftLog).commitTo\n\tgo.etcd.io/etcd/raft/v3@v3.5.21/log.go:237\ngo.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat\n\tgo.etcd.io/etcd/raft/v3@v3.5.21/raft.go:1508\ngo.etcd.io/etcd/raft/v3.stepFollower\n\tgo.etcd.io/etcd/raft/v3@v3.5.21/raft.go:1434\ngo.etcd.io/etcd/raft/v3.(*raft).Step\n\tgo.etcd.io/etcd/raft/v3@v3.5.21/raft.go:975\ngo.etcd.io/etcd/raft/v3.(*node).run\n\tgo.etcd.io/etcd/raft/v3@v3.5.21/node.go:356"}
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: panic: tocommit(10563786) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: goroutine 183 [running]:
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000003c0, {0x0, 0x0, 0x0})
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x2dc
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.uber.org/zap.(*SugaredLogger).log(0xc0000ac4a0, 0x4, {0x118ff29?, 0x46e4bb?}, {0xc00028e320?, 0xf5ae80?, 0xc0001f9000?}, {0x0, 0x0, 0x0})
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.uber.org/zap@v1.17.0/sugar.go:227 +0xec
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.uber.org/zap.(*SugaredLogger).Panicf(...)
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.uber.org/zap@v1.17.0/sugar.go:159
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf(0xa130ca?, {0x118ff29?, 0xc0001f9160?}, {0xc00028e320?, 0xc00011ade0?, 0x10e31e0?})
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/server/v3/etcdserver/zap_raft.go:101 +0x45
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.etcd.io/etcd/raft/v3.(*raftLog).commitTo(0xc0002bfe30, 0xa130ca)
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/raft/v3@v3.5.21/log.go:237 +0xf3
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat(_, {0x8, 0xbf4c5f14c328e1d3, 0x4c1f37b6aaeced42, 0x12, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/raft/v3@v3.5.21/raft.go:1508 +0x39
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.etcd.io/etcd/raft/v3.stepFollower(_, {0x8, 0xbf4c5f14c328e1d3, 0x4c1f37b6aaeced42, 0x12, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/raft/v3@v3.5.21/raft.go:1434 +0x3b8
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.etcd.io/etcd/raft/v3.(*raft).Step(_, {0x8, 0xbf4c5f14c328e1d3, 0x4c1f37b6aaeced42, 0x12, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/raft/v3@v3.5.21/raft.go:975 +0x12d5
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: go.etcd.io/etcd/raft/v3.(*node).run(0xc000700120)
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/raft/v3@v3.5.21/node.go:356 +0x925
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]: created by go.etcd.io/etcd/raft/v3.RestartNode in goroutine 1
Jul 03 00:22:45 test-005056895018-node-1 etcd-member[81324]:         go.etcd.io/etcd/raft/v3@v3.5.21/node.go:244 +0x239
Jul 03 00:22:45 test-005056895018-node-1 systemd[1]: etcd-member.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ An ExecStart= process belonging to unit etcd-member.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 2.
Jul 03 00:22:45 test-005056895018-node-1 systemd[1]: etcd-member.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The unit etcd-member.service has entered the 'failed' state with result 'exit-code'.
Jul 03 00:22:45 test-005056895018-node-1 systemd[1]: Failed to start etcd key-value store.
░░ Subject: A start job for unit etcd-member.service has failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit etcd-member.service has finished with a failure.
░░
░░ The job identifier is 744969058 and the job result is failed.
[test@test-005056895018-node-1 ~]$

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions