Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rejoining the cluster with a different client/raft url breaks etcd #1430

Closed
kelseyhightower opened this issue Oct 27, 2014 · 10 comments
Closed

Rejoining the cluster with a different client/raft url breaks etcd #1430

kelseyhightower opened this issue Oct 27, 2014 · 10 comments

Comments

@kelseyhightower
Copy link
Contributor

Steps to reproduce:

Start a three node cluster:

etcd0

etcd -name etcd0 -data-dir data \
-listen-client-urls http://127.0.0.1:2379 \
-advertise-client-urls http://127.0.0.1:2379 \
-listen-peer-urls http://127.0.0.1:2380 \
-initial-advertise-peer-urls http://127.0.0.1:2380 \
-initial-cluster etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:2390,etcd2=http://127.0.0.1:2400 \
-initial-cluster-state new

etcd1

etcd -name etcd1 -data-dir data \
-listen-client-urls http://127.0.0.1:2389 \
-advertise-client-urls http://127.0.0.1:2389 \
-listen-peer-urls http://127.0.0.1:2390 \
-initial-advertise-peer-urls http://127.0.0.1:2390 \
-initial-cluster etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:2390,etcd2=http://127.0.0.1:2400 \
-initial-cluster-state new

etcd2

etcd -name etcd2 -data-dir data \
-listen-client-urls http://127.0.0.1:2399 \
-advertise-client-urls http://127.0.0.1:2399 \
-listen-peer-urls http://127.0.0.1:2400 \
-initial-advertise-peer-urls http://127.0.0.1:2400 \
-initial-cluster etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:2390,etcd2=http://127.0.0.1:2400 \
-initial-cluster-state new

At this point everything works fine.

Stop etcd0 and add it back with a different client url

etcd -name etcd0 -data-dir data \
-listen-client-urls http://127.0.0.1:2479 \
-advertise-client-urls http://127.0.0.1:2479 \
-listen-peer-urls http://127.0.0.1:2480

Logs from etcd0 etcd1 etcd2

etcd0

Initial cluster bootstrap

2014/10/27 10:20:53 etcdserver: start node bc3f316962a4ce82 in cluster 30679f72ae8cacd1
2014/10/27 10:20:53 Listening for peers on http://127.0.0.1:2380
2014/10/27 10:20:53 Listening for client requests on http://127.0.0.1:2379
2014/10/27 10:20:58 etcdserver: publish error: context deadline exceeded
2014/10/27 10:20:58 raft: leader changed from 0x0 to 0xbc3f316962a4ce82
2014/10/27 10:20:58 etcdserver: published {Name:etcd0 ClientURLs:[http://127.0.0.1:2379]} to the cluster

Restart with a different set of ips and ports for both raft and client urls

2014/10/27 10:22:11 etcdserver: restart member bc3f316962a4ce82 in cluster 30679f72ae8cacd1 at commit index 41
2014/10/27 10:22:11 Listening for peers on http://127.0.0.1:2480
2014/10/27 10:22:11 Listening for client requests on http://127.0.0.1:2479
2014/10/27 10:22:16 etcdserver: published {Name:etcd0 ClientURLs:[http://127.0.0.1:2479]} to the cluster

etcd1

2014/10/27 10:20:57 etcdserver: start node 23e2cbc42c3d0b18 in cluster 30679f72ae8cacd1
2014/10/27 10:20:57 Listening for peers on http://127.0.0.1:2390
2014/10/27 10:20:57 Listening for client requests on http://127.0.0.1:2389
2014/10/27 10:20:58 raft: leader changed from 0x0 to 0xbc3f316962a4ce82
2014/10/27 10:20:58 etcdserver: published {Name:etcd1 ClientURLs:[http://127.0.0.1:2389]} to the cluster
2014/10/27 10:21:16 raft: leader changed from 0xbc3f316962a4ce82 to 0x0
2014/10/27 10:21:16 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:14 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:15 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:17 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:18 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:19 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:20 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:22 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:23 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:25 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:28 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:30 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:31 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:33 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:35 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:38 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:39 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:41 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:42 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:43 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:44 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:46 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:47 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:48 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:50 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:51 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:52 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:54 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:56 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:58 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:59 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:00 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:02 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:03 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:04 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:06 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:08 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:10 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:13 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:14 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:15 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:17 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:19 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:20 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:21 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:23 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:24 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:26 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:27 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:28 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:29 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:31 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:32 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:34 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:35 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:36 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:37 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:39 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:40 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:42 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:43 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:45 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:46 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:48 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:49 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:51 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:54 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:56 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:58 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:24:00 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:24:01 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83

etcd2

2014/10/27 10:21:02 etcdserver: start node 3cffa4d6b5998f83 in cluster 30679f72ae8cacd1
2014/10/27 10:21:02 Listening for peers on http://127.0.0.1:2400
2014/10/27 10:21:02 Listening for client requests on http://127.0.0.1:2399
2014/10/27 10:21:02 raft: leader changed from 0x0 to 0xbc3f316962a4ce82
2014/10/27 10:21:02 etcdserver: published {Name:etcd2 ClientURLs:[http://127.0.0.1:2399]} to the cluster
2014/10/27 10:21:16 raft: leader changed from 0xbc3f316962a4ce82 to 0x0
2014/10/27 10:21:16 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:14 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:15 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:17 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:18 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:19 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:20 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:22 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:23 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:25 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:28 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:30 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:31 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:33 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:35 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:38 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:39 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:41 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:42 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:43 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:44 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:46 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:47 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:48 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:50 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:51 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:52 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:22:54 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:22:56 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:22:58 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:22:59 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:00 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:02 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:03 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:04 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:06 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:08 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:10 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:13 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:14 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:15 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:17 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:19 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:20 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:21 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:23 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:24 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:26 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:27 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:28 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:29 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:31 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:32 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:23:34 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
2014/10/27 10:23:35 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:36 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:37 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:39 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:40 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:42 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:43 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:45 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:46 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:48 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:49 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:51 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:54 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:23:56 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:23:58 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:24:00 raft: leader changed from 0x23e2cbc42c3d0b18 to 0x0
2014/10/27 10:24:01 raft: leader changed from 0x0 to 0x3cffa4d6b5998f83
2014/10/27 10:24:02 raft: leader changed from 0x3cffa4d6b5998f83 to 0x0
@kelseyhightower kelseyhightower changed the title Rejoining the cluster with a different client url breaks etcd Rejoining the cluster with a different client/raft url breaks etcd Oct 27, 2014
@kelseyhightower
Copy link
Contributor Author

Adding -initial-advertise-peer-urls http://127.0.0.1:2480 does not fix the issue.

@kelseyhightower
Copy link
Contributor Author

Running curl -L http://127.0.0.1:2389/v2/admin/members

{
   "members": [
      {
         "id": 2585853179480312600,
         "peerURLs": [
            "http://127.0.0.1:2390"
         ],
         "name": "etcd1",
         "clientURLs": [
            "http://127.0.0.1:2389"
         ]
      },
      {
         "id": 4395413003413589891,
         "peerURLs": [
            "http://127.0.0.1:2400"
         ],
         "name": "etcd2",
         "clientURLs": [
            "http://127.0.0.1:2399"
         ]
      },
      {
         "id": 13564614931359518338,
         "peerURLs": [
            "http://127.0.0.1:2380"
         ],
         "name": "etcd0"
      }
   ]
}

@kelseyhightower
Copy link
Contributor Author

If I stop etcd0, things start working again.

@yichengq
Copy link
Contributor

We don't support peer url update now. if you restart with

etcd -name etcd0 -data-dir data \
-listen-client-urls http://127.0.0.1:2479 \
-advertise-client-urls http://127.0.0.1:2479 \
-listen-peer-urls http://127.0.0.1:2380

, it works. We would come up with warning message printed out for this case.

@kelseyhightower
Copy link
Contributor Author

@unihorn I've tried to follow your suggestion and it made etcd panic:

2014/10/27 10:51:42 etcdserver: restart member bc3f316962a4ce82 in cluster 30679f72ae8cacd1 at commit index 41
2014/10/27 10:51:42 Listening for peers on http://127.0.0.1:2380
2014/10/27 10:51:42 Listening for client requests on http://127.0.0.1:2479
2014/10/27 10:51:42 raft: leader changed from 0x0 to 0x23e2cbc42c3d0b18
2014/10/27 10:51:42 etcdserver: published {Name:etcd0 ClientURLs:[http://127.0.0.1:2479]} to the cluster
panic: conflict with committed entry

goroutine 20 [running]:
runtime.panic(0x2c0200, 0xc208038ce0)
    /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/coreos/etcd/raft.(*raftLog).maybeAppend(0xc2080487e0, 0x3, 0x1, 0x8ad, 0xc208178000, 0x8ab, 0x9c7, 0x8ae, 0xc208045230)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/log.go:66 +0x15b
github.com/coreos/etcd/raft.(*raft).handleAppendEntries(0xc20800e630, 0x3, 0xbc3f316962a4ce82, 0x23e2cbc42c3d0b18, 0x67, 0x1, 0x3, 0xc208178000, 0x8ab, 0x9c7, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/raft.go:382 +0x78
github.com/coreos/etcd/raft.stepFollower(0xc20800e630, 0x3, 0xbc3f316962a4ce82, 0x23e2cbc42c3d0b18, 0x67, 0x1, 0x3, 0xc208178000, 0x8ab, 0x9c7, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/raft.go:484 +0x206
github.com/coreos/etcd/raft.(*raft).Step(0xc20800e630, 0x3, 0xbc3f316962a4ce82, 0x23e2cbc42c3d0b18, 0x67, 0x1, 0x3, 0xc208178000, 0x8ab, 0x9c7, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/raft.go:377 +0xe5
github.com/coreos/etcd/raft.(*node).run(0xc20803fe80, 0xc20800e630)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/node.go:239 +0x3f4
created by github.com/coreos/etcd/raft.RestartNode
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/node.go:173 +0x2c1

goroutine 16 [chan receive]:
main.main()
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/main.go:138 +0x2a2

goroutine 19 [finalizer wait]:
runtime.park(0x18810, 0x5b9930, 0x5b83e9)
    /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x5b9930, 0x5b83e9)
    /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 22 [select]:
github.com/coreos/etcd/etcdserver.(*EtcdServer).run(0xc2080009c0)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/server.go:302 +0x585
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).start
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/server.go:286 +0x1d1

goroutine 37 [runnable]:
net/http.(*conn).serve(0xc20804a080)
    /usr/local/go/src/pkg/net/http/server.go:1094
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

goroutine 24 [IO wait]:
net.runtime_pollWait(0x6871b0, 0x72, 0x0)
    /private/var/folders/00/0sdwh000h01000cxqpysvccm0035qk/T/makerelease910109054/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208068530, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208068530, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080684d0, 0x46fdb8, 0x0, 0x685290, 0x23)
    /usr/local/go/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20803c088, 0x8, 0x0, 0x0)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803c088, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:244 +0x4b
net/http.(*Server).Serve(0xc2080043c0, 0x6861e0, 0xc20803c088, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1698 +0x91
net/http.Serve(0x6861e0, 0xc20803c088, 0x686160, 0xc20807d440, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1576 +0x7c
main.func·001()
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/main.go:199 +0x14b
created by main.startEtcd
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/main.go:200 +0xb09

goroutine 25 [IO wait]:
net.runtime_pollWait(0x687100, 0x72, 0x0)
    /private/var/folders/00/0sdwh000h01000cxqpysvccm0035qk/T/makerelease910109054/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208068610, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208068610, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080685b0, 0x46fdb8, 0x0, 0x685290, 0x23)
    /usr/local/go/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20803c090, 0xc20803eca8, 0x0, 0x0)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803c090, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:244 +0x4b
net/http.(*Server).Serve(0xc2080045a0, 0x6861e0, 0xc20803c090, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1698 +0x91
net/http.Serve(0x6861e0, 0xc20803c090, 0x6873a8, 0xc208082380, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1576 +0x7c
main.func·002()
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/main.go:218 +0x183
created by main.startEtcd
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/main.go:219 +0xeb8

goroutine 26 [IO wait]:
net.runtime_pollWait(0x687050, 0x72, 0x0)
    /private/var/folders/00/0sdwh000h01000cxqpysvccm0035qk/T/makerelease910109054/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208028990, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208028990, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208028930, 0xc208062000, 0x1000, 0x1000, 0x0, 0x685290, 0x23)
    /usr/local/go/src/pkg/net/fd_unix.go:242 +0x34c
net.(*conn).Read(0xc20803c0f0, 0xc208062000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/net.go:122 +0xe7
net/http.(*liveSwitchReader).Read(0xc20804a328, 0xc208062000, 0x1000, 0x1000, 0xf9d, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:206 +0xaf
io.(*LimitedReader).Read(0xc208044fe0, 0xc208062000, 0x1000, 0x1000, 0x1d, 0x0, 0x0)
    /usr/local/go/src/pkg/io/io.go:399 +0xd0
bufio.(*Reader).fill(0xc208005440)
    /usr/local/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).ReadSlice(0xc208005440, 0xc20806e70a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/bufio/bufio.go:298 +0x22c
bufio.(*Reader).ReadLine(0xc208005440, 0x0, 0x0, 0x0, 0x5d5300, 0x0, 0x0)
    /usr/local/go/src/pkg/bufio/bufio.go:326 +0x69
net/textproto.(*Reader).readLineSlice(0xc20807c240, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/textproto/reader.go:55 +0x9d
net/textproto.(*Reader).ReadLine(0xc20807c240, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/textproto/reader.go:36 +0x4e
net/http.ReadRequest(0xc208005440, 0xc2080008f0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/request.go:556 +0xc7
net/http.(*conn).readRequest(0xc20804a300, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:577 +0x276
net/http.(*conn).serve(0xc20804a300)
    /usr/local/go/src/pkg/net/http/server.go:1132 +0x61e
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

goroutine 36 [runnable]:
syscall.Syscall(0x3, 0xb, 0xc208219e00, 0x10000, 0x10000, 0x0, 0x0)
    /usr/local/go/src/pkg/syscall/asm_darwin_amd64.s:20 +0x5
syscall.read(0xb, 0xc208219e00, 0x10000, 0x10000, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/syscall/zsyscall_darwin_amd64.go:952 +0x75
syscall.Read(0xb, 0xc208219e00, 0x10000, 0x10000, 0x8000, 0x0, 0x0)
    /usr/local/go/src/pkg/syscall/syscall_unix.go:136 +0x5c
net.(*netFD).Read(0xc208028000, 0xc208219e00, 0x10000, 0x10000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_unix.go:238 +0x273
net.(*conn).Read(0xc20803c018, 0xc208219e00, 0x10000, 0x10000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/net.go:122 +0xe7
net/http.(*liveSwitchReader).Read(0xc20804a028, 0xc208219e00, 0x10000, 0x10000, 0x8000, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:206 +0xaf
io.(*LimitedReader).Read(0xc2080440c0, 0xc208219e00, 0x10000, 0x10000, 0x10e01, 0x0, 0x0)
    /usr/local/go/src/pkg/io/io.go:399 +0xd0
bufio.(*Reader).Read(0xc208004000, 0xc208219e00, 0x10000, 0x10000, 0x10000, 0x0, 0x0)
    /usr/local/go/src/pkg/bufio/bufio.go:165 +0x13b
io.(*LimitedReader).Read(0xc208045700, 0xc208219e00, 0x10000, 0x10000, 0x1fe00, 0x0, 0x0)
    /usr/local/go/src/pkg/io/io.go:399 +0xd0
net/http.(*body).readLocked(0xc208170940, 0xc208219e00, 0x10000, 0x10000, 0x1fe00, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/transfer.go:577 +0x81
net/http.(*body).Read(0xc208170940, 0xc208219e00, 0x10000, 0x10000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/transfer.go:572 +0x11a
bytes.(*Buffer).ReadFrom(0xc2080293b0, 0x687620, 0xc208170940, 0xfe00, 0x0, 0x0)
    /usr/local/go/src/pkg/bytes/buffer.go:169 +0x20e
io/ioutil.readAll(0x687620, 0xc208170940, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/io/ioutil/ioutil.go:33 +0x1ae
io/ioutil.ReadAll(0x687620, 0xc208170940, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/io/ioutil/ioutil.go:42 +0x6c
github.com/coreos/etcd/etcdserver/etcdhttp.serverHandler.serveRaft(0x0, 0x686010, 0xc2080009c0, 0x6860a0, 0xc2080009c0, 0x0, 0x0, 0x686060, 0xc20803ecc0, 0x685b40, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/etcdhttp/http.go:275 +0x267
github.com/coreos/etcd/etcdserver/etcdhttp.serverHandler.(github.com/coreos/etcd/etcdserver/etcdhttp.serveRaft)·fm(0x6875e8, 0xc20804c460, 0xc208000340)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/etcdhttp/http.go:95 +0x5b
net/http.HandlerFunc.ServeHTTP(0xc208004d20, 0x6875e8, 0xc20804c460, 0xc208000340)
    /usr/local/go/src/pkg/net/http/server.go:1235 +0x40
net/http.(*ServeMux).ServeHTTP(0xc20807d440, 0x6875e8, 0xc20804c460, 0xc208000340)
    /usr/local/go/src/pkg/net/http/server.go:1511 +0x1a3
net/http.serverHandler.ServeHTTP(0xc2080043c0, 0x6875e8, 0xc20804c460, 0xc208000340)
    /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc20804a000)
    /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

goroutine 33 [IO wait]:
net.runtime_pollWait(0x686ef0, 0x72, 0x0)
    /private/var/folders/00/0sdwh000h01000cxqpysvccm0035qk/T/makerelease910109054/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208028df0, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208028df0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208028d90, 0xc2080b4000, 0x1000, 0x1000, 0x0, 0x685290, 0x23)
    /usr/local/go/src/pkg/net/fd_unix.go:242 +0x34c
net.(*conn).Read(0xc20803c150, 0xc2080b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/net.go:122 +0xe7
net/http.noteEOFReader.Read(0x6874f0, 0xc20803c150, 0xc20806eaa8, 0xc2080b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/transport.go:1203 +0x72
net/http.(*noteEOFReader).Read(0xc208045520, 0xc2080b4000, 0x1000, 0x1000, 0xc20801296c, 0x0, 0x0)
    <autogenerated>:124 +0xca
bufio.(*Reader).fill(0xc208005920)
    /usr/local/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc208005920, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/bufio/bufio.go:132 +0x101
net/http.(*persistConn).readLoop(0xc20806ea50)
    /usr/local/go/src/pkg/net/http/transport.go:782 +0x95
created by net/http.(*Transport).dialConn
    /usr/local/go/src/pkg/net/http/transport.go:600 +0x93f

goroutine 35 [select]:
github.com/coreos/etcd/raft.(*node).step(0xc20803fe80, 0x6872f8, 0x2, 0x3, 0xbc3f316962a4ce82, 0x23e2cbc42c3d0b18, 0x67, 0x1, 0x3, 0xc208142000, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/node.go:322 +0x1c9
github.com/coreos/etcd/raft.(*node).Step(0xc20803fe80, 0x6872f8, 0x2, 0x3, 0xbc3f316962a4ce82, 0x23e2cbc42c3d0b18, 0x67, 0x1, 0x3, 0xc208142000, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/node.go:303 +0x96
github.com/coreos/etcd/etcdserver.(*EtcdServer).Process(0xc2080009c0, 0x6872f8, 0x2, 0x3, 0xbc3f316962a4ce82, 0x23e2cbc42c3d0b18, 0x67, 0x1, 0x3, 0xc208142000, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/server.go:293 +0x106
github.com/coreos/etcd/etcdserver/etcdhttp.serverHandler.serveRaft(0x0, 0x686010, 0xc2080009c0, 0x6860a0, 0xc2080009c0, 0x0, 0x0, 0x686060, 0xc20803ecc0, 0x685b40, ...)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/etcdhttp/http.go:287 +0x631
github.com/coreos/etcd/etcdserver/etcdhttp.serverHandler.(github.com/coreos/etcd/etcdserver/etcdhttp.serveRaft)·fm(0x6875e8, 0xc20804c000, 0xc208001a00)
    /Users/philips/coreos/etcd-go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/etcdhttp/http.go:95 +0x5b
net/http.HandlerFunc.ServeHTTP(0xc208004d20, 0x6875e8, 0xc20804c000, 0xc208001a00)
    /usr/local/go/src/pkg/net/http/server.go:1235 +0x40
net/http.(*ServeMux).ServeHTTP(0xc20807d440, 0x6875e8, 0xc20804c000, 0xc208001a00)
    /usr/local/go/src/pkg/net/http/server.go:1511 +0x1a3
net/http.serverHandler.ServeHTTP(0xc2080043c0, 0x6875e8, 0xc20804c000, 0xc208001a00)
    /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc20804a500)
    /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

goroutine 31 [IO wait]:
net.runtime_pollWait(0x686fa0, 0x72, 0x0)
    /private/var/folders/00/0sdwh000h01000cxqpysvccm0035qk/T/makerelease910109054/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208028d80, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208028d80, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208028d20, 0xc2080b2000, 0x1000, 0x1000, 0x0, 0x685290, 0x23)
    /usr/local/go/src/pkg/net/fd_unix.go:242 +0x34c
net.(*conn).Read(0xc20803c140, 0xc2080b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/net.go:122 +0xe7
net/http.noteEOFReader.Read(0x6874f0, 0xc20803c140, 0xc20806e9f8, 0xc2080b2000, 0x1000, 0x1000, 0xc208005620, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/transport.go:1203 +0x72
net/http.(*noteEOFReader).Read(0xc208045460, 0xc2080b2000, 0x1000, 0x1000, 0xc20801296c, 0x0, 0x0)
    <autogenerated>:124 +0xca
bufio.(*Reader).fill(0xc208005860)
    /usr/local/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc208005860, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/bufio/bufio.go:132 +0x101
net/http.(*persistConn).readLoop(0xc20806e9a0)
    /usr/local/go/src/pkg/net/http/transport.go:782 +0x95
created by net/http.(*Transport).dialConn
    /usr/local/go/src/pkg/net/http/transport.go:600 +0x93f

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc20806e9a0)
    /usr/local/go/src/pkg/net/http/transport.go:885 +0x38f
created by net/http.(*Transport).dialConn
    /usr/local/go/src/pkg/net/http/transport.go:601 +0x957

goroutine 34 [select]:
net/http.(*persistConn).writeLoop(0xc20806ea50)
    /usr/local/go/src/pkg/net/http/transport.go:885 +0x38f
created by net/http.(*Transport).dialConn
    /usr/local/go/src/pkg/net/http/transport.go:601 +0x957

@yichengq
Copy link
Contributor

The panic happens because etcd0 uses the log that belongs to a different cluster from etcd1 and etcd2.
The log confirms this:
etcd0

 0     0:  norm    1969-12-31 16:00:00 -0800 PST
 1     1:  conf ConfChangeAddNode 23e2cbc42c3d0b18 {"id":2585853179480312600,"peerURLs":["http://127.0.0.1:2390"],"name":"etcd1"}
 1     2:  conf ConfChangeAddNode 3cffa4d6b5998f83 {"id":4395413003413589891,"peerURLs":["http://127.0.0.1:2400"],"name":"etcd2"}
 1     3:  conf ConfChangeAddNode bc3f316962a4ce82 {"id":13564614931359518338,"peerURLs":["http://127.0.0.1:2380"],"name":"etcd0"}
 4     4:  norm    1969-12-31 16:00:00 -0800 PST
 4     5:  norm PUT /0/members/bc3f316962a4ce82/attributes {"name":"etcd0","clientURLs":["http://127.0.0.1:2379"]} 1969-12-31 16:00:00 -0800 PST
 4     6:  norm SYNC   2014-10-27 10:20:58.281131529 -0700 PDT
 4     7:  norm PUT /0/members/23e2cbc42c3d0b18/attributes {"name":"etcd1","clientURLs":["http://127.0.0.1:2389"]} 1969-12-31 16:00:00 -0800 PST
 4     8:  norm SYNC   2014-10-27 10:20:58.574659565 -0700 PDT
 4     9:  norm SYNC   2014-10-27 10:20:59.078461309 -0700 PDT

etcd1

 0     0:  norm    1969-12-31 16:00:00 -0800 PST
 1     1:  conf ConfChangeAddNode 23e2cbc42c3d0b18 {"id":2585853179480312600,"peerURLs":["http://127.0.0.1:2390"],"name":"etcd1"}
 1     2:  conf ConfChangeAddNode 3cffa4d6b5998f83 {"id":4395413003413589891,"peerURLs":["http://127.0.0.1:2400"],"name":"etcd2"}
 1     3:  conf ConfChangeAddNode bc3f316962a4ce82 {"id":13564614931359518338,"peerURLs":["http://127.0.0.1:2380"],"name":"etcd0"}
 3     4:  norm    1969-12-31 16:00:00 -0800 PST
 3     5:  norm PUT /0/members/23e2cbc42c3d0b18/attributes {"name":"etcd1","clientURLs":["http://127.0.0.1:2389"]} 1969-12-31 16:00:00 -0800 PST
 3     6:  norm SYNC   2014-10-27 10:33:13.670332579 -0700 PDT
 3     7:  norm PUT /0/members/3cffa4d6b5998f83/attributes {"name":"etcd2","clientURLs":["http://127.0.0.1:2399"]} 1969-12-31 16:00:00 -0800 PST
 3     8:  norm SYNC   2014-10-27 10:33:13.968348664 -0700 PDT
 3     9:  norm SYNC   2014-10-27 10:33:14.464540892 -0700 PDT

etcd2:

 0     0:  norm    1969-12-31 16:00:00 -0800 PST
 1     1:  conf ConfChangeAddNode 23e2cbc42c3d0b18 {"id":2585853179480312600,"peerURLs":["http://127.0.0.1:2390"],"name":"etcd1"}
 1     2:  conf ConfChangeAddNode 3cffa4d6b5998f83 {"id":4395413003413589891,"peerURLs":["http://127.0.0.1:2400"],"name":"etcd2"}
 1     3:  conf ConfChangeAddNode bc3f316962a4ce82 {"id":13564614931359518338,"peerURLs":["http://127.0.0.1:2380"],"name":"etcd0"}
 3     4:  norm    1969-12-31 16:00:00 -0800 PST
 3     5:  norm PUT /0/members/23e2cbc42c3d0b18/attributes {"name":"etcd1","clientURLs":["http://127.0.0.1:2389"]} 1969-12-31 16:00:00 -0800 PST
 3     6:  norm SYNC   2014-10-27 10:33:13.670332579 -0700 PDT
 3     7:  norm PUT /0/members/3cffa4d6b5998f83/attributes {"name":"etcd2","clientURLs":["http://127.0.0.1:2399"]} 1969-12-31 16:00:00 -0800 PST
 3     8:  norm SYNC   2014-10-27 10:33:13.968348664 -0700 PDT
 3     9:  norm SYNC   2014-10-27 10:33:14.464540892 -0700 PDT

The SYNC committed displays different time, so these logs belong to two different cluster. They cannot be run together.

We have introduced -initial-cluster-name to point out different clusters. User could set different cluster name when reboot, and if they do so, etcd could detect cluster unmatch. We need to document that.

We need to build command to automate the process to locate this kind of bugs.

@yichengq yichengq mentioned this issue Oct 27, 2014
Closed
@xiang90
Copy link
Contributor

xiang90 commented Oct 27, 2014

@kelseyhightower @unihorn
To sum it up:

  1. the issue is caused by a misconfiguration: mixes log data of two different cluster.
  2. we have provided a way for user to avoid this problem by using unique cluster-name for bootstrap, so even the clusters with same configuration can have different clusterID.
  3. we need to publish the log analysis tool for users to debug this themselves.

@yichengq yichengq mentioned this issue Oct 27, 2014
Closed
@kelseyhightower
Copy link
Contributor Author

I guess the confusion comes in because I'm not sure how I created a new cluster. I just stopped and started the same node using the same data dir.

@kelseyhightower
Copy link
Contributor Author

Ok, this all makes sense now. Maybe along the way I ran bootstrap static again.

@yichengq
Copy link
Contributor

further plan in #1440 . Thanks for reporting!

@dlipovetsky dlipovetsky mentioned this issue Jun 22, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kelseyhightower @yichengq @xiang90