Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot restart after shutdown / Ctrl+C (etcdmain: database file does not match with snapshot) #5857

Closed
binwiederhier opened this issue Jul 3, 2016 · 17 comments · Fixed by #5862
Closed

Cannot restart after shutdown / Ctrl+C (etcdmain: database file does not match with snapshot) #5857

binwiederhier opened this issue Jul 3, 2016 · 17 comments · Fixed by #5862
Labels
type/bug
Milestone
v3.1.0

Comments

@binwiederhier
Copy link

I have a cluster of 3 and I'm testing failover scenarios right now. I simulate power outages (shutting of a VM) and service failures (Ctrl+C on etcd). As of now, shutting off an instance has terrible consequences, as it cannot recover by simply rebooting a machine / restarting the service:

$ etcd
// ...runs...
<Press Ctrl+C>

$ etcd
2016-07-03 02:05:21.753663 I | flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://rly4:2379
2016-07-03 02:05:21.754254 I | flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/data
2016-07-03 02:05:21.754550 I | flags: recognized and used environment variable ETCD_DEBUG=true
2016-07-03 02:05:21.754805 I | flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://rly4:2380
2016-07-03 02:05:21.755066 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=rly4=http://rly4:2380,rly1=http://rly1:2380,rly5=http://rly5:2380
2016-07-03 02:05:21.755317 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
2016-07-03 02:05:21.755585 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=rly-cluster-1
2016-07-03 02:05:21.755833 I | flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://rly4:2379,http://127.0.0.1:2379
2016-07-03 02:05:21.756038 I | flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://rly4:2380
2016-07-03 02:05:21.756276 I | flags: recognized and used environment variable ETCD_NAME=rly4
2016-07-03 02:05:21.756525 I | flags: recognized and used environment variable ETCD_WAL_DIR=/var/lib/etcd/wal
2016-07-03 02:05:21.756784 I | etcdmain: etcd Version: 3.0.0
2016-07-03 02:05:21.757038 I | etcdmain: Git SHA: 6f48bda
2016-07-03 02:05:21.757261 I | etcdmain: Go Version: go1.6.2
2016-07-03 02:05:21.757478 I | etcdmain: Go OS/Arch: linux/amd64
2016-07-03 02:05:21.757696 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-07-03 02:05:21.757950 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2016-07-03 02:05:21.761023 I | etcdmain: listening for peers on http://rly4:2380
2016-07-03 02:05:21.761065 I | etcdmain: listening for client requests on 127.0.0.1:2379
2016-07-03 02:05:21.763682 I | etcdmain: listening for client requests on rly4:2379
2016-07-03 02:05:21.804445 I | etcdserver: recovered store from snapshot at index 141365
2016-07-03 02:05:21.804844 I | etcdserver: name = rly4
2016-07-03 02:05:21.805118 I | etcdserver: data dir = /var/lib/etcd/data
2016-07-03 02:05:21.805389 I | etcdserver: member dir = /var/lib/etcd/data/member
2016-07-03 02:05:21.805661 I | etcdserver: dedicated WAL dir = /var/lib/etcd/wal
2016-07-03 02:05:21.805891 I | etcdserver: heartbeat = 100ms
2016-07-03 02:05:21.806169 I | etcdserver: election = 1000ms
2016-07-03 02:05:21.806428 I | etcdserver: snapshot count = 10000
2016-07-03 02:05:21.806696 I | etcdserver: advertise client URLs = http://rly4:2379
2016-07-03 02:05:21.807377 I | etcdserver: restarting member 89889c482441ec6e in cluster d0ce954dc5f082d0 at commit index 141549
2016-07-03 02:05:21.807753 I | raft: 89889c482441ec6e became follower at term 1966
2016-07-03 02:05:21.808041 I | raft: newRaft 89889c482441ec6e [peers: [89889c482441ec6e,d27323ab9b295f50,e790e6b697b3c219], term: 1966, commit: 141549, applied: 141365, lastindex: 141549, lastterm: 1966]
2016-07-03 02:05:21.808406 I | membership: added member d27323ab9b295f50 [http://rly5:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 02:05:21.808751 I | membership: added member e790e6b697b3c219 [http://rly1:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 02:05:21.809021 I | membership: added member 89889c482441ec6e [http://rly4:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 02:05:21.809295 I | membership: set the cluster version to 3.0 from store
2016-07-03 02:05:21.818080 I | etcdmain: stopping listening for client requests on rly4:2379
2016-07-03 02:05:21.818511 I | etcdmain: stopping listening for client requests on rly4:2379
2016-07-03 02:05:21.818707 I | etcdmain: stopping listening for peers on http://rly4:2380
2016-07-03 02:05:21.818894 C | etcdmain: database file (/var/lib/etcd/data/member/snap/db index 140014) does not match with snapshot (index 141365).

$ find
./data
./data/member
./data/member/snap
./data/member/snap/00000000000007ae-0000000000022835.snap
./data/member/snap/db
./wal
./wal/0000000000000000-0000000000000000.wal
./wal/0.tmp

The only way to recover from this right now is to delete the data+wal dir, remove the member, re-add it and then restart the daemon. However, I fear that if all fail at the same time (power failure), I will lose all data.
@binwiederhier binwiederhier changed the title Rebooting / Ctrl+C corrupts database (etcdmain: database file does not match with snapshot) Cannot restart after shutdown / Ctrl+C (etcdmain: database file does not match with snapshot) Jul 3, 2016
@binwiederhier
Copy link
Author

Related: After removing the data+wal dir, restarting (without removing the member first) prints a stack trace:

# etcd
2016-07-03 02:20:28.031273 I | flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://rly4:2379
2016-07-03 02:20:28.031752 I | flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/data
2016-07-03 02:20:28.032234 I | flags: recognized and used environment variable ETCD_DEBUG=true
2016-07-03 02:20:28.032705 I | flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://rly4:2380
2016-07-03 02:20:28.033143 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=rly4=http://rly4:2380,rly1=http://rly1:2380,rly5=http://rly5:2380
2016-07-03 02:20:28.033508 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
2016-07-03 02:20:28.035314 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=rly-cluster-1
2016-07-03 02:20:28.035679 I | flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://rly4:2379,http://127.0.0.1:2379
2016-07-03 02:20:28.035896 I | flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://rly4:2380
2016-07-03 02:20:28.036123 I | flags: recognized and used environment variable ETCD_NAME=rly4
2016-07-03 02:20:28.036379 I | flags: recognized and used environment variable ETCD_WAL_DIR=/var/lib/etcd/wal
2016-07-03 02:20:28.036688 I | etcdmain: etcd Version: 3.0.0
2016-07-03 02:20:28.036923 I | etcdmain: Git SHA: 6f48bda
2016-07-03 02:20:28.037145 I | etcdmain: Go Version: go1.6.2
2016-07-03 02:20:28.037365 I | etcdmain: Go OS/Arch: linux/amd64
2016-07-03 02:20:28.037586 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-07-03 02:20:28.040920 I | etcdmain: listening for peers on http://rly4:2380
2016-07-03 02:20:28.040959 I | etcdmain: listening for client requests on 127.0.0.1:2379
2016-07-03 02:20:28.043601 I | etcdmain: listening for client requests on rly4:2379
2016-07-03 02:20:28.057481 I | netutil: resolving rly1:2380 to 192.168.0.31:2380
2016-07-03 02:20:28.061152 I | netutil: resolving rly1:2380 to 192.168.0.31:2380
2016-07-03 02:20:28.064142 I | netutil: resolving rly4:2380 to 192.168.0.34:2380
2016-07-03 02:20:28.067372 I | netutil: resolving rly4:2380 to 192.168.0.34:2380
2016-07-03 02:20:28.070151 I | netutil: resolving rly5:2380 to 192.168.0.35:2380
2016-07-03 02:20:28.072927 I | netutil: resolving rly5:2380 to 192.168.0.35:2380
2016-07-03 02:20:28.077895 I | etcdserver: name = rly4
2016-07-03 02:20:28.078413 I | etcdserver: data dir = /var/lib/etcd/data
2016-07-03 02:20:28.078798 I | etcdserver: member dir = /var/lib/etcd/data/member
2016-07-03 02:20:28.079151 I | etcdserver: dedicated WAL dir = /var/lib/etcd/wal
2016-07-03 02:20:28.079490 I | etcdserver: heartbeat = 100ms
2016-07-03 02:20:28.079506 I | etcdserver: election = 1000ms
2016-07-03 02:20:28.079514 I | etcdserver: snapshot count = 10000
2016-07-03 02:20:28.079527 I | etcdserver: advertise client URLs = http://rly4:2379
2016-07-03 02:20:28.083747 I | etcdserver: starting member 89889c482441ec6e in cluster d0ce954dc5f082d0
2016-07-03 02:20:28.084129 I | raft: 89889c482441ec6e became follower at term 0
2016-07-03 02:20:28.084149 I | raft: newRaft 89889c482441ec6e [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-07-03 02:20:28.084157 I | raft: 89889c482441ec6e became follower at term 1
2016-07-03 02:20:28.097873 I | rafthttp: started HTTP pipelining with peer d27323ab9b295f50
2016-07-03 02:20:28.098338 I | rafthttp: started HTTP pipelining with peer e790e6b697b3c219
2016-07-03 02:20:28.098619 I | rafthttp: starting peer d27323ab9b295f50...
2016-07-03 02:20:28.099148 I | rafthttp: started HTTP pipelining with peer d27323ab9b295f50
2016-07-03 02:20:28.099685 I | rafthttp: started streaming with peer d27323ab9b295f50 (writer)
2016-07-03 02:20:28.105343 I | rafthttp: started peer d27323ab9b295f50
2016-07-03 02:20:28.105805 I | rafthttp: added peer d27323ab9b295f50
2016-07-03 02:20:28.106256 I | rafthttp: starting peer e790e6b697b3c219...
2016-07-03 02:20:28.106673 I | rafthttp: started HTTP pipelining with peer e790e6b697b3c219
2016-07-03 02:20:28.114568 I | rafthttp: started streaming with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 02:20:28.115871 I | rafthttp: started streaming with peer d27323ab9b295f50 (writer)
2016-07-03 02:20:28.116274 I | rafthttp: started streaming with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 02:20:28.116737 I | rafthttp: started peer e790e6b697b3c219
2016-07-03 02:20:28.117054 I | rafthttp: added peer e790e6b697b3c219
2016-07-03 02:20:28.117337 I | etcdserver: starting server... [version: 3.0.0, cluster version: to_be_decided]
2016-07-03 02:20:28.117781 I | rafthttp: started streaming with peer e790e6b697b3c219 (writer)
2016-07-03 02:20:28.118010 I | rafthttp: started streaming with peer e790e6b697b3c219 (writer)
2016-07-03 02:20:28.129196 I | rafthttp: started streaming with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 02:20:28.131685 I | rafthttp: started streaming with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 02:20:28.132625 I | raft: 89889c482441ec6e [term: 1] received a MsgHeartbeat message with higher term from e790e6b697b3c219 [term: 1966]
2016-07-03 02:20:28.133191 I | raft: 89889c482441ec6e became follower at term 1966
2016-07-03 02:20:28.133723 C | raft: tocommit(141549) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?
panic: tocommit(141549) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?

goroutine 91 [running]:
panic(0xd44e00, 0xc8201e4680)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820192780, 0x12d2e00, 0x5d, 0xc8201d8880, 0x2, 0x2)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raftLog).commitTo(0xc820200460, 0x228ed)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/log.go:191 +0x1a6
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).handleHeartbeat(0xc8201d52b0, 0x8, 0x89889c482441ec6e, 0xe790e6b697b3c219, 0x7ae, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:844 +0x44
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.stepFollower(0xc8201d52b0, 0x8, 0x89889c482441ec6e, 0xe790e6b697b3c219, 0x7ae, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:806 +0x1194
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).Step(0xc8201d52b0, 0x8, 0x89889c482441ec6e, 0xe790e6b697b3c219, 0x7ae, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:596 +0x66d
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*node).run(0xc8201b5da0, 0xc8201d52b0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:317 +0x8ce
created by github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.StartNode
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:204 +0x777

@binwiederhier
Copy link
Author

binwiederhier commented Jul 3, 2016 •
edited
Loading

Apparently simply deleting the database file helped too: rm data/member/snap/db.
However, the problem still persists. If it crashes, systemd cannot restart it.

Edit: This workaround does not work anymore on 3.0.1 due to #5841 😢

@xiang90
Copy link
Contributor

xiang90 commented Jul 3, 2016

@binwiederhier Removing the data-dir should not work. Removing data-dir is equal to member lost.

Shutting down machine should be fine. Can you provide more details about how to reproduce the restart failure?

@xiang90 xiang90 added this to the v3.1.0 milestone Jul 3, 2016
@binwiederhier
Copy link
Author

Thanks for responding so quickly. Over night (standby-[8hrs]->wakeup) I have observed that my failing member ("rly4") healed itself, so it is now possible to kill/restart etcd without this nasty error. I presume there is some regular background process that writes the db file or something.

However, after I removed the member again (kill/Ctrl+C/"systemctl stop etcd" -> "etcdctl member remove .." -> "rm -rf data+wal") and re-added it, it was not possible to kill+restart it without the error above (database file does not match ...).

I attached all relevant files in this archive (issue5857.zip), I hope this helps. rly1+rly5 can be restarted fine, rly4 is the problem child.

@xiang90
Copy link
Contributor

xiang90 commented Jul 3, 2016

@binwiederhier Thanks. Should fix shortly if I can reproduce.

@xiang90
Copy link
Contributor

xiang90 commented Jul 3, 2016

@binwiederhier Do you still keep the log of rly4 from the very beginning (the first startup after re-joined it)

@xiang90 xiang90 mentioned this issue Jul 3, 2016
Merged
@binwiederhier
Copy link
Author

@xiang90 Since I can reproduce this realiably, here's a log that shows rly4 beging started as a fresh member, then Ctrl+C, then restarted:

root@rly4:/var/lib/etcd# etcd
2016-07-03 13:19:25.897416 I | flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://rly4:2379
2016-07-03 13:19:25.897898 I | flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/data
2016-07-03 13:19:25.898158 I | flags: recognized and used environment variable ETCD_DEBUG=true
2016-07-03 13:19:25.898460 I | flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://rly4:2380
2016-07-03 13:19:25.898721 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=rly4=http://rly4:2380,rly1=http://rly1:2380,rly5=http://rly5:2380
2016-07-03 13:19:25.898978 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
2016-07-03 13:19:25.899219 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=rly-cluster-1
2016-07-03 13:19:25.899425 I | flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://rly4:2379,http://127.0.0.1:2379
2016-07-03 13:19:25.899680 I | flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://rly4:2380
2016-07-03 13:19:25.899872 I | flags: recognized and used environment variable ETCD_NAME=rly4
2016-07-03 13:19:25.900122 I | flags: recognized and used environment variable ETCD_WAL_DIR=/var/lib/etcd/wal
2016-07-03 13:19:25.900372 I | etcdmain: etcd Version: 3.0.1
2016-07-03 13:19:25.900609 I | etcdmain: Git SHA: a4a52cb
2016-07-03 13:19:25.900821 I | etcdmain: Go Version: go1.6.2
2016-07-03 13:19:25.901037 I | etcdmain: Go OS/Arch: linux/amd64
2016-07-03 13:19:25.901256 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-07-03 13:19:25.904187 I | etcdmain: listening for peers on http://rly4:2380
2016-07-03 13:19:25.904246 I | etcdmain: listening for client requests on 127.0.0.1:2379
2016-07-03 13:19:25.906843 I | etcdmain: listening for client requests on rly4:2379
2016-07-03 13:19:25.917585 I | netutil: resolving rly1:2380 to 192.168.0.31:2380
2016-07-03 13:19:25.920841 I | netutil: resolving rly1:2380 to 192.168.0.31:2380
2016-07-03 13:19:25.923722 I | netutil: resolving rly4:2380 to 192.168.0.34:2380
2016-07-03 13:19:25.926564 I | netutil: resolving rly4:2380 to 192.168.0.34:2380
2016-07-03 13:19:25.930104 I | netutil: resolving rly5:2380 to 192.168.0.35:2380
2016-07-03 13:19:25.933113 I | netutil: resolving rly5:2380 to 192.168.0.35:2380
2016-07-03 13:19:25.937387 I | etcdserver: name = rly4
2016-07-03 13:19:25.937728 I | etcdserver: data dir = /var/lib/etcd/data
2016-07-03 13:19:25.938013 I | etcdserver: member dir = /var/lib/etcd/data/member
2016-07-03 13:19:25.938307 I | etcdserver: dedicated WAL dir = /var/lib/etcd/wal
2016-07-03 13:19:25.938583 I | etcdserver: heartbeat = 100ms
2016-07-03 13:19:25.938856 I | etcdserver: election = 1000ms
2016-07-03 13:19:25.939123 I | etcdserver: snapshot count = 10000
2016-07-03 13:19:25.939392 I | etcdserver: advertise client URLs = http://rly4:2379
2016-07-03 13:19:25.942993 I | etcdserver: starting member 3d7025c80a6d09dc in cluster d0ce954dc5f082d0
2016-07-03 13:19:25.943449 I | raft: 3d7025c80a6d09dc became follower at term 0
2016-07-03 13:19:25.943726 I | raft: newRaft 3d7025c80a6d09dc [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-07-03 13:19:25.943739 I | raft: 3d7025c80a6d09dc became follower at term 1
2016-07-03 13:19:25.959642 I | rafthttp: started HTTP pipelining with peer d27323ab9b295f50
2016-07-03 13:19:25.960345 I | rafthttp: started HTTP pipelining with peer e790e6b697b3c219
2016-07-03 13:19:25.961099 I | rafthttp: starting peer d27323ab9b295f50...
2016-07-03 13:19:25.961817 I | rafthttp: started HTTP pipelining with peer d27323ab9b295f50
2016-07-03 13:19:25.962873 I | rafthttp: started streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:25.971886 I | rafthttp: started peer d27323ab9b295f50
2016-07-03 13:19:25.975956 I | rafthttp: added peer d27323ab9b295f50
2016-07-03 13:19:25.976448 I | rafthttp: starting peer e790e6b697b3c219...
2016-07-03 13:19:25.976820 I | rafthttp: started HTTP pipelining with peer e790e6b697b3c219
2016-07-03 13:19:25.978109 I | rafthttp: started peer e790e6b697b3c219
2016-07-03 13:19:25.986587 I | rafthttp: added peer e790e6b697b3c219
2016-07-03 13:19:25.987369 I | etcdserver: starting server... [version: 3.0.1, cluster version: to_be_decided]
2016-07-03 13:19:25.987928 I | rafthttp: started streaming with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:25.989916 I | rafthttp: started streaming with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:25.990703 I | rafthttp: started streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:25.991217 I | rafthttp: peer e790e6b697b3c219 became active
2016-07-03 13:19:25.991500 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 writer)
2016-07-03 13:19:25.991750 I | rafthttp: started streaming with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:25.992138 I | rafthttp: started streaming with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:25.992547 I | raft: 3d7025c80a6d09dc [term: 1] received a MsgApp message with higher term from e790e6b697b3c219 [term: 2028]
2016-07-03 13:19:25.992862 I | raft: 3d7025c80a6d09dc became follower at term 2028
2016-07-03 13:19:25.993108 D | raft: 3d7025c80a6d09dc [logterm: 0, index: 163513] rejected msgApp [logterm: 2028, index: 163513] from e790e6b697b3c219
2016-07-03 13:19:25.993557 I | raft: raft.node: 3d7025c80a6d09dc elected leader e790e6b697b3c219 at term 2028
2016-07-03 13:19:25.994201 I | rafthttp: peer d27323ab9b295f50 became active
2016-07-03 13:19:25.994753 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 writer)
2016-07-03 13:19:25.995559 I | rafthttp: started streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:25.998998 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream Message writer)
2016-07-03 13:19:26.000923 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:26.001236 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:26.001471 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:26.001766 I | rafthttp: started streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:26.002013 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream Message writer)
2016-07-03 13:19:26.002256 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:26.033311 I | rafthttp: receiving database snapshot [index:163574, from e790e6b697b3c219] ...
2016-07-03 13:19:26.037462 I | snap: saved database snapshot to disk [total bytes: 32768]
2016-07-03 13:19:26.037780 I | rafthttp: received and saved database snapshot [index: 163574, from: e790e6b697b3c219] successfully
2016-07-03 13:19:26.038108 I | raft: 3d7025c80a6d09dc [commit: 0, lastindex: 0, lastterm: 0] starts to restore snapshot [index: 163574, term: 2028]
2016-07-03 13:19:26.038373 I | raft: log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 163574, term: 2028]
2016-07-03 13:19:26.038637 I | raft: 3d7025c80a6d09dc restored progress of 3d7025c80a6d09dc [next = 163575, match = 163574, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]
2016-07-03 13:19:26.038948 I | raft: 3d7025c80a6d09dc restored progress of d27323ab9b295f50 [next = 163575, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]
2016-07-03 13:19:26.039245 I | raft: 3d7025c80a6d09dc restored progress of e790e6b697b3c219 [next = 163575, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]
2016-07-03 13:19:26.039493 I | raft: 3d7025c80a6d09dc [commit: 163574] restored snapshot [index: 163574, term: 2028]
2016-07-03 13:19:26.040006 I | etcdserver: applying snapshot at index 0...
2016-07-03 13:19:26.040190 I | etcdserver: restoring mvcc store...
2016-07-03 13:19:26.040249 I | etcdserver: finished restoring mvcc store
2016-07-03 13:19:26.040259 I | etcdserver: recovering lessor...
2016-07-03 13:19:26.040503 I | etcdserver: closing old backend...
2016-07-03 13:19:26.045398 I | etcdserver: finished recovering lessor
2016-07-03 13:19:26.046285 I | etcdserver: recovering alarms...
2016-07-03 13:19:26.059099 I | etcdserver: raft applied incoming snapshot at index 163574
2016-07-03 13:19:26.059942 I | etcdserver: finished closing old backend
2016-07-03 13:19:26.060396 I | etcdserver: finished recovering alarms
2016-07-03 13:19:26.060422 I | etcdserver: recovering auth store...
2016-07-03 13:19:26.060444 I | etcdserver: finished recovering auth store
2016-07-03 13:19:26.060456 I | etcdserver: recovering store v2...
2016-07-03 13:19:26.087969 I | etcdserver: finished recovering store v2
2016-07-03 13:19:26.089217 I | etcdserver: recovering cluster configuration...
2016-07-03 13:19:26.089657 I | membership: added member 3d7025c80a6d09dc [http://rly4:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 13:19:26.089938 I | membership: added member d27323ab9b295f50 [http://rly5:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 13:19:26.090176 I | membership: added member e790e6b697b3c219 [http://rly1:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 13:19:26.090415 I | membership: set the cluster version to 3.0 from store
2016-07-03 13:19:26.090650 I | etcdserver: finished recovering cluster configuration
2016-07-03 13:19:26.091011 I | etcdserver: removing old peers from network...
2016-07-03 13:19:26.091316 I | rafthttp: stopping peer d27323ab9b295f50...
2016-07-03 13:19:26.091721 I | rafthttp: closed the TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 writer)
2016-07-03 13:19:26.092122 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:26.093099 I | rafthttp: closed the TCP streaming connection with peer d27323ab9b295f50 (stream Message writer)
2016-07-03 13:19:26.093583 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:26.094055 I | rafthttp: stopped HTTP pipelining with peer d27323ab9b295f50
2016-07-03 13:19:26.094530 W | rafthttp: lost the TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:26.094668 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:26.094740 W | rafthttp: lost the TCP streaming connection with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:26.094871 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:26.094882 I | rafthttp: stopped peer d27323ab9b295f50
2016-07-03 13:19:26.094893 I | rafthttp: removed peer d27323ab9b295f50
2016-07-03 13:19:26.095017 I | rafthttp: stopping peer e790e6b697b3c219...
2016-07-03 13:19:26.095128 I | rafthttp: closed the TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 writer)
2016-07-03 13:19:26.095259 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:26.095341 I | rafthttp: closed the TCP streaming connection with peer e790e6b697b3c219 (stream Message writer)
2016-07-03 13:19:26.095754 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:26.095781 I | rafthttp: stopped HTTP pipelining with peer e790e6b697b3c219
2016-07-03 13:19:26.096040 W | rafthttp: lost the TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:26.096555 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:26.096631 W | rafthttp: lost the TCP streaming connection with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:26.096776 E | rafthttp: failed to read e790e6b697b3c219 on stream Message (net/http: request canceled)
2016-07-03 13:19:26.096786 I | rafthttp: peer e790e6b697b3c219 became inactive
2016-07-03 13:19:26.096796 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:26.096804 I | rafthttp: stopped peer e790e6b697b3c219
2016-07-03 13:19:26.096813 I | rafthttp: removed peer e790e6b697b3c219
2016-07-03 13:19:26.096819 I | etcdserver: finished removing old peers from network
2016-07-03 13:19:26.096825 I | etcdserver: adding peers from new cluster configuration into network...
2016-07-03 13:19:26.096844 I | rafthttp: starting peer d27323ab9b295f50...
2016-07-03 13:19:26.096867 I | rafthttp: started HTTP pipelining with peer d27323ab9b295f50
2016-07-03 13:19:26.097921 I | rafthttp: started peer d27323ab9b295f50
2016-07-03 13:19:26.102746 I | rafthttp: added peer d27323ab9b295f50
2016-07-03 13:19:26.103311 I | rafthttp: starting peer e790e6b697b3c219...
2016-07-03 13:19:26.103518 I | rafthttp: started HTTP pipelining with peer e790e6b697b3c219
2016-07-03 13:19:26.105365 I | rafthttp: started peer e790e6b697b3c219
2016-07-03 13:19:26.105936 I | rafthttp: added peer e790e6b697b3c219
2016-07-03 13:19:26.106365 I | etcdserver: finished adding peers from new cluster configuration into network...
2016-07-03 13:19:26.106708 I | etcdserver: finished applying incoming snapshot at index 0
2016-07-03 13:19:26.107215 I | etcdserver: published {Name:rly4 ClientURLs:[http://rly4:2379]} to cluster d0ce954dc5f082d0
2016-07-03 13:19:26.107619 I | rafthttp: started streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:26.107670 I | rafthttp: started streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:26.107714 I | rafthttp: started streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:26.107755 I | rafthttp: started streaming with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:26.108118 I | rafthttp: started streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:26.108172 I | rafthttp: started streaming with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:26.108936 I | rafthttp: started streaming with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:26.109652 I | etcdmain: ready to serve client requests
2016-07-03 13:19:26.110598 N | etcdmain: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2016-07-03 13:19:26.112926 E | etcdmain: failed to notify systemd for readiness: No socket
2016-07-03 13:19:26.114487 E | etcdmain: forgot to set Type=notify in systemd service file?
2016-07-03 13:19:26.114795 I | rafthttp: started streaming with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:26.115437 I | rafthttp: peer e790e6b697b3c219 became active
2016-07-03 13:19:26.115888 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:26.116297 I | rafthttp: peer d27323ab9b295f50 became active
2016-07-03 13:19:26.116602 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:26.117167 I | etcdmain: ready to serve client requests
2016-07-03 13:19:26.117827 N | etcdmain: serving insecure client requests on rly4:2379, this is strongly discouraged!
2016-07-03 13:19:26.119359 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:26.119899 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:26.198899 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream Message writer)
2016-07-03 13:19:26.201975 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream Message writer)
2016-07-03 13:19:26.214577 I | rafthttp: established a TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 writer)
2016-07-03 13:19:26.216231 I | rafthttp: established a TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 writer)
2016-07-03 13:19:26.488664 I | api: enabled capabilities for version 3.0

// <Ctrl+C> here (works with "kill" or "init 0" too)

2016-07-03 13:19:44.440332 N | osutil: received interrupt signal, shutting down...
2016-07-03 13:19:44.443139 I | rafthttp: stopped HTTP pipelining with peer d27323ab9b295f50
2016-07-03 13:19:44.444660 I | rafthttp: stopped HTTP pipelining with peer e790e6b697b3c219
2016-07-03 13:19:44.445343 I | rafthttp: stopping peer d27323ab9b295f50...
2016-07-03 13:19:44.445906 I | rafthttp: closed the TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 writer)
2016-07-03 13:19:44.448025 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:44.448261 I | rafthttp: closed the TCP streaming connection with peer d27323ab9b295f50 (stream Message writer)
2016-07-03 13:19:44.448284 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (writer)
2016-07-03 13:19:44.448667 I | rafthttp: stopped HTTP pipelining with peer d27323ab9b295f50
2016-07-03 13:19:44.448844 W | rafthttp: lost the TCP streaming connection with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:44.448891 E | rafthttp: failed to read d27323ab9b295f50 on stream MsgApp v2 (net/http: request canceled)
2016-07-03 13:19:44.448905 I | rafthttp: peer d27323ab9b295f50 became inactive
2016-07-03 13:19:44.448922 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (stream MsgApp v2 reader)
2016-07-03 13:19:44.449072 W | rafthttp: lost the TCP streaming connection with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:44.449113 D | rafthttp: failed to read d27323ab9b295f50 on stream Message (net/http: request canceled)
2016-07-03 13:19:44.449130 I | rafthttp: stopped streaming with peer d27323ab9b295f50 (stream Message reader)
2016-07-03 13:19:44.449144 I | rafthttp: stopped peer d27323ab9b295f50
2016-07-03 13:19:44.449156 I | rafthttp: stopping peer e790e6b697b3c219...
2016-07-03 13:19:44.449536 I | rafthttp: closed the TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 writer)
2016-07-03 13:19:44.449552 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:44.449978 I | rafthttp: closed the TCP streaming connection with peer e790e6b697b3c219 (stream Message writer)
2016-07-03 13:19:44.449998 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (writer)
2016-07-03 13:19:44.450281 I | rafthttp: stopped HTTP pipelining with peer e790e6b697b3c219
2016-07-03 13:19:44.450454 W | rafthttp: lost the TCP streaming connection with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:44.450468 E | rafthttp: failed to read e790e6b697b3c219 on stream MsgApp v2 (net/http: request canceled)
2016-07-03 13:19:44.450475 I | rafthttp: peer e790e6b697b3c219 became inactive
2016-07-03 13:19:44.450485 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (stream MsgApp v2 reader)
2016-07-03 13:19:44.450586 W | rafthttp: lost the TCP streaming connection with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:44.450600 I | rafthttp: stopped streaming with peer e790e6b697b3c219 (stream Message reader)
2016-07-03 13:19:44.450608 I | rafthttp: stopped peer e790e6b697b3c219

root@rly4:/var/lib/etcd# etcd
2016-07-03 13:20:04.077239 I | flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://rly4:2379
2016-07-03 13:20:04.077866 I | flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/data
2016-07-03 13:20:04.078272 I | flags: recognized and used environment variable ETCD_DEBUG=true
2016-07-03 13:20:04.078740 I | flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://rly4:2380
2016-07-03 13:20:04.079118 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=rly4=http://rly4:2380,rly1=http://rly1:2380,rly5=http://rly5:2380
2016-07-03 13:20:04.079499 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
2016-07-03 13:20:04.079886 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=rly-cluster-1
2016-07-03 13:20:04.080222 I | flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://rly4:2379,http://127.0.0.1:2379
2016-07-03 13:20:04.080463 I | flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://rly4:2380
2016-07-03 13:20:04.080827 I | flags: recognized and used environment variable ETCD_NAME=rly4
2016-07-03 13:20:04.081121 I | flags: recognized and used environment variable ETCD_WAL_DIR=/var/lib/etcd/wal
2016-07-03 13:20:04.081417 I | etcdmain: etcd Version: 3.0.1
2016-07-03 13:20:04.081677 I | etcdmain: Git SHA: a4a52cb
2016-07-03 13:20:04.081925 I | etcdmain: Go Version: go1.6.2
2016-07-03 13:20:04.082172 I | etcdmain: Go OS/Arch: linux/amd64
2016-07-03 13:20:04.082419 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-07-03 13:20:04.082735 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2016-07-03 13:20:04.086894 I | etcdmain: listening for peers on http://rly4:2380
2016-07-03 13:20:04.086955 I | etcdmain: listening for client requests on 127.0.0.1:2379
2016-07-03 13:20:04.089557 I | etcdmain: listening for client requests on rly4:2379
2016-07-03 13:20:04.126639 I | etcdserver: recovered store from snapshot at index 163574
2016-07-03 13:20:04.127086 I | etcdserver: name = rly4
2016-07-03 13:20:04.127338 I | etcdserver: data dir = /var/lib/etcd/data
2016-07-03 13:20:04.127572 I | etcdserver: member dir = /var/lib/etcd/data/member
2016-07-03 13:20:04.127818 I | etcdserver: dedicated WAL dir = /var/lib/etcd/wal
2016-07-03 13:20:04.128145 I | etcdserver: heartbeat = 100ms
2016-07-03 13:20:04.128384 I | etcdserver: election = 1000ms
2016-07-03 13:20:04.128616 I | etcdserver: snapshot count = 10000
2016-07-03 13:20:04.129060 I | etcdserver: advertise client URLs = http://rly4:2379
2016-07-03 13:20:04.130203 I | etcdserver: restarting member 3d7025c80a6d09dc in cluster d0ce954dc5f082d0 at commit index 163611
2016-07-03 13:20:04.130682 I | raft: 3d7025c80a6d09dc became follower at term 2028
2016-07-03 13:20:04.130985 I | raft: newRaft 3d7025c80a6d09dc [peers: [3d7025c80a6d09dc,d27323ab9b295f50,e790e6b697b3c219], term: 2028, commit: 163611, applied: 163574, lastindex: 163611, lastterm: 2028]
2016-07-03 13:20:04.131373 I | membership: added member 3d7025c80a6d09dc [http://rly4:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 13:20:04.131698 I | membership: added member d27323ab9b295f50 [http://rly5:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 13:20:04.131983 I | membership: added member e790e6b697b3c219 [http://rly1:2380] to cluster d0ce954dc5f082d0 from store
2016-07-03 13:20:04.132274 I | membership: set the cluster version to 3.0 from store
2016-07-03 13:20:04.144457 I | etcdmain: stopping listening for client requests on rly4:2379
2016-07-03 13:20:04.145401 I | etcdmain: stopping listening for client requests on rly4:2379
2016-07-03 13:20:04.146479 I | etcdmain: stopping listening for peers on http://rly4:2380
2016-07-03 13:20:04.147403 C | etcdmain: database file (/var/lib/etcd/data/member/snap/db index 160016) does not match with snapshot (index 163574).

@xiang90
Copy link
Contributor

xiang90 commented Jul 3, 2016 •
edited
Loading

@binwiederhier

try to apply patch in #5862. update the binary of all members including rly1 and rly5. Then try to see if you still can reproduce it?

@binwiederhier
Copy link
Author

Sure thing. My first encounter with "go". Have to install all the deps to get it to build :-)

@binwiederhier
Copy link
Author

I'm trying to build a new version with your patch to test if it works, but I have no idea how to. ./build is just giving me this: https://gist.github.com/binwiederhier/5b42f5abc4829462970ccc9c45913b52 and I can't find any instructions.

@xiang90
Copy link
Contributor

xiang90 commented Jul 3, 2016

@binwiederhier It seems the GOPATH is not set correctly. You need to put etcd under GOPATH correctly to make the vendor work. etcd needs to be at $GOPATH/src/github.com/coreos/etcd

Can you do go env and paste the output?

@binwiederhier
Copy link
Author

binwiederhier commented Jul 3, 2016 •
edited
Loading

This is really hard. Do you have instructions how to set this stuff up from scratch? I feel I'm randomly poking around and I'm not getting anywhere. I managed to put things in the right place, but now when I run go get or go get ./... (I googled this, no idea what it does), I get this:

7/gopath/src/github.com/coreos/etcd:0a9d1c3 ✓ ❯❯❯ go get ./...
# golang.org/x/net/http2/hpack
../../../golang.org/x/net/http2/hpack/huffman.go:14: undefined: sync.Pool
# golang.org/x/sys/unix
../../../golang.org/x/sys/unix/syscall_solaris.go:38: clen redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:817
../../../golang.org/x/sys/unix/syscall_solaris.go:51: ParseDirent redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:830
../../../golang.org/x/sys/unix/syscall_solaris.go:77: Pipe redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux_amd64.go:110
../../../golang.org/x/sys/unix/syscall_solaris.go:89: (*SockaddrInet4).sockaddr redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:311
../../../golang.org/x/sys/unix/syscall_solaris.go:103: (*SockaddrInet6).sockaddr redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:325
../../../golang.org/x/sys/unix/syscall_solaris.go:118: (*SockaddrUnix).sockaddr redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:340
../../../golang.org/x/sys/unix/syscall_solaris.go:144: Getsockname redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:525
../../../golang.org/x/sys/unix/syscall_solaris.go:153: ImplementsGetwd redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:184
../../../golang.org/x/sys/unix/syscall_solaris.go:157: Getwd redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:188
../../../golang.org/x/sys/unix/syscall_solaris.go:178: Getgroups redeclared in this block
    previous declaration at ../../../golang.org/x/sys/unix/syscall_linux.go:201
../../../golang.org/x/sys/unix/syscall_solaris.go:178: too many errors
# github.com/coreos/etcd/pkg/httputil
pkg/httputil/httputil.go:18: req.Cancel undefined (type *http.Request has no field or method Cancel)
# github.com/coreos/etcd/pkg/fileutil
pkg/fileutil/lock_linux.go:37: undefined: syscall.Flock_t
pkg/fileutil/lock_linux.go:50: undefined: syscall.Flock_t
pkg/fileutil/lock_linux.go:51: undefined: syscall.FcntlFlock
pkg/fileutil/lock_linux.go:68: undefined: syscall.FcntlFlock
pkg/fileutil/lock_linux.go:89: undefined: syscall.FcntlFlock
# github.com/coreos/etcd/cmd/vendor/github.com/prometheus/common/model
cmd/vendor/github.com/prometheus/common/model/signature.go:33: undefined: sync.Pool
# github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/hpack
cmd/vendor/golang.org/x/net/http2/hpack/huffman.go:14: undefined: sync.Pool
# github.com/coreos/etcd/cmd/vendor/github.com/akrennmair/gopcap
cmd/vendor/github.com/akrennmair/gopcap/pcap.go:12:18: fatal error: pcap.h: No such file or directory
 #include <pcap.h>
                  ^
compilation terminated.
# github.com/coreos/etcd/cmd/vendor/golang.org/x/sys/unix
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:37: clen redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:784
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:50: ParseDirent redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:797
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:76: Pipe redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux_amd64.go:108
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:88: (*SockaddrInet4).sockaddr redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:291
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:102: (*SockaddrInet6).sockaddr redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:305
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:117: (*SockaddrUnix).sockaddr redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:320
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:141: Getsockname redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:492
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:153: ImplementsGetwd redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:164
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:155: Getwd redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:168
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:164: Getgroups redeclared in this block
    previous declaration at cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:181
cmd/vendor/golang.org/x/sys/unix/syscall_solaris.go:164: too many errors
# github.com/akrennmair/gopcap
../../akrennmair/gopcap/pcap.go:12:18: fatal error: pcap.h: No such file or directory
 #include <pcap.h>
                  ^
compilation terminated.
# github.com/ugorji/go/codec
../../ugorji/go/codec/encode.go:1377: undefined: sync.Pool
../../ugorji/go/codec/encode.go:1387: undefined: sync.Pool
../../ugorji/go/codec/helper.go:247: undefined: sync.Pool
# github.com/coreos/etcd/cmd/vendor/github.com/ugorji/go/codec
cmd/vendor/github.com/ugorji/go/codec/encode.go:1341: undefined: sync.Pool
cmd/vendor/github.com/ugorji/go/codec/encode.go:1351: undefined: sync.Pool

Edit: apt install libpcap-dev fixed the gopcap thing, the rest remains

@gyuho
Copy link
Contributor

gyuho commented Jul 3, 2016 •
edited
Loading

@binwiederhier You can clean up your GOPATH and use cd $GOPATH/src/github.com/coreos/etcd/cmd && godep restore

godep can be found at https://github.com/tools/godep.

instead of go get

@binwiederhier
Copy link
Author

binwiederhier commented Jul 3, 2016 •
edited
Loading

@gyuho I finally got it to build. Turns out the PPA version of go is needed. The one in Ubuntu 14.04 upstream does not work. Here are my instructions (if you want to add them somewhere):

# Install go on Ubuntu 14.04 (as per https://github.com/golang/go/wiki/Ubuntu)
$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable
$ sudo apt-get update
$ sudo apt-get install golang

# Install a weird dependency of something
$ sudo apt-get install libpcap-dev

# Checkout etcd
$ mkdir -p gopath/src/github.com/coreos
$ cd gopath
$ export GOPATH=$(pwd)
$ cd src/github.com/coreos
$ git clone git@github.com:coreos/etcd.git
$ cd etcd
$ go get     # or: "go get ./..." (Not sure which one did it)
$ ./build

# Binaries will be in cmd/*

@xiang90 I can confirm that your fix worked.

@xiang90
Copy link
Contributor

xiang90 commented Jul 3, 2016 •
edited
Loading

@gyuho Can you help to update the build section in our doc? Seems like it is currently difficult for people who are not familiar with go env. (godep is not needed in my opinion, ./build will setup the dependency correctly. they need to set gopath and get correct version of go installed.)

@gyuho
Copy link
Contributor

gyuho commented Jul 3, 2016

@xiang90 Ok will do.

@binwiederhier
Copy link
Author

Just wanted to say thanks. You guys have been great! Blazing fast responses, awesome tool!! Much appreciated 👍

gyuho added a commit to gyuho/etcd that referenced this issue Jul 5, 2016
@gyuho
README: add more instructions on building etcd
43bbbe3 
Address
- etcd-io#5857 (comment)
@gyuho gyuho mentioned this issue Jul 5, 2016
Merged
gyuho added a commit to gyuho/etcd that referenced this issue Jul 5, 2016
@gyuho
Documentation: add instruction on vendoring, build
ff5ddd0 
Addressing etcd-io#5857 (comment).
@philipn philipn mentioned this issue Jul 27, 2016
Closed
@mattymo mattymo mentioned this issue Aug 29, 2016
Closed
@tomasramanauskas tomasramanauskas mentioned this issue Oct 8, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Milestone
v3.1.0
Development

Successfully merging a pull request may close this issue.

etcdserver: commit before sending snapshot xiang90/etcd
3 participants
@binwiederhier @xiang90 @gyuho