Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: TestKVGetRetry #5601

Closed
gyuho opened this issue Jun 8, 2016 · 3 comments
Closed

test: TestKVGetRetry #5601

gyuho opened this issue Jun 8, 2016 · 3 comments
Labels
area/testing

Comments

@gyuho
Copy link
Contributor

gyuho commented Jun 8, 2016 

=== RUN   TestKVGetRetry
2016-06-08 21:07:24.721496 I | integration: launching node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:24.721783 I | integration: launching node7632688164621677138 (unix://localhost:node7632688164621677138.sock.bridge)
2016-06-08 21:07:24.721867 I | integration: launching node5947063075280645312 (unix://localhost:node5947063075280645312.sock.bridge)
2016-06-08 21:07:24.742611 I | etcdserver: name = node2127541176347271374
2016-06-08 21:07:24.742665 I | etcdserver: data dir = /tmp/etcd002323622
2016-06-08 21:07:24.742713 I | etcdserver: member dir = /tmp/etcd002323622/member
2016-06-08 21:07:24.742765 I | etcdserver: heartbeat = 10ms
2016-06-08 21:07:24.742799 I | etcdserver: election = 100ms
2016-06-08 21:07:24.742824 I | etcdserver: snapshot count = 0
2016-06-08 21:07:24.742860 I | etcdserver: advertise client URLs = http://127.0.0.1:21514
2016-06-08 21:07:24.742890 I | etcdserver: name = node5947063075280645312
2016-06-08 21:07:24.742943 I | etcdserver: data dir = /tmp/etcd733096136
2016-06-08 21:07:24.742989 I | etcdserver: member dir = /tmp/etcd733096136/member
2016-06-08 21:07:24.743018 I | etcdserver: heartbeat = 10ms
2016-06-08 21:07:24.743045 I | etcdserver: election = 100ms
2016-06-08 21:07:24.743068 I | etcdserver: snapshot count = 0
2016-06-08 21:07:24.743109 I | etcdserver: advertise client URLs = http://127.0.0.1:21518
2016-06-08 21:07:24.743173 I | etcdserver: initial advertise peer URLs = http://127.0.0.1:21517
2016-06-08 21:07:24.743236 I | etcdserver: initial cluster = node2127541176347271374=http://127.0.0.1:21513,node5947063075280645312=http://127.0.0.1:21517,node7632688164621677138=http://127.0.0.1:21515
2016-06-08 21:07:24.743321 I | etcdserver: initial advertise peer URLs = http://127.0.0.1:21513
2016-06-08 21:07:24.743393 I | etcdserver: name = node7632688164621677138
2016-06-08 21:07:24.743460 I | etcdserver: data dir = /tmp/etcd644281293
2016-06-08 21:07:24.743513 I | etcdserver: member dir = /tmp/etcd644281293/member
2016-06-08 21:07:24.743556 I | etcdserver: heartbeat = 10ms
2016-06-08 21:07:24.743596 I | etcdserver: election = 100ms
2016-06-08 21:07:24.743647 I | etcdserver: initial cluster = node2127541176347271374=http://127.0.0.1:21513,node5947063075280645312=http://127.0.0.1:21517,node7632688164621677138=http://127.0.0.1:21515
2016-06-08 21:07:24.743702 I | etcdserver: snapshot count = 0
2016-06-08 21:07:24.743758 I | etcdserver: advertise client URLs = http://127.0.0.1:21516
2016-06-08 21:07:24.743803 I | etcdserver: initial advertise peer URLs = http://127.0.0.1:21515
2016-06-08 21:07:24.743911 I | etcdserver: initial cluster = node2127541176347271374=http://127.0.0.1:21513,node5947063075280645312=http://127.0.0.1:21517,node7632688164621677138=http://127.0.0.1:21515
2016-06-08 21:07:24.764287 I | etcdserver: starting member 46fc68d58043bec5 in cluster 1de0725f07555f79
2016-06-08 21:07:24.764481 I | raft: 46fc68d58043bec5 became follower at term 0
2016-06-08 21:07:24.764536 I | raft: newRaft 46fc68d58043bec5 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-06-08 21:07:24.764576 I | raft: 46fc68d58043bec5 became follower at term 1
2016-06-08 21:07:24.767235 I | rafthttp: starting peer 70bca064fa6854d2...
2016-06-08 21:07:24.767696 I | rafthttp: started HTTP pipelining with peer 70bca064fa6854d2
2016-06-08 21:07:24.769338 I | rafthttp: started streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:24.770995 I | rafthttp: started streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:24.771199 I | etcdserver: starting member 70bca064fa6854d2 in cluster 1de0725f07555f79
2016-06-08 21:07:24.771558 I | raft: 70bca064fa6854d2 became follower at term 0
2016-06-08 21:07:24.771664 I | raft: newRaft 70bca064fa6854d2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-06-08 21:07:24.771737 I | raft: 70bca064fa6854d2 became follower at term 1
2016-06-08 21:07:24.772158 I | etcdserver: starting member d1e4193dc152cc39 in cluster 1de0725f07555f79
2016-06-08 21:07:24.772356 I | raft: d1e4193dc152cc39 became follower at term 0
2016-06-08 21:07:24.772439 I | raft: newRaft d1e4193dc152cc39 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-06-08 21:07:24.772492 I | raft: d1e4193dc152cc39 became follower at term 1
2016-06-08 21:07:24.773171 I | rafthttp: started peer 70bca064fa6854d2
2016-06-08 21:07:24.773245 I | rafthttp: started streaming with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:24.773428 I | rafthttp: added peer 70bca064fa6854d2
2016-06-08 21:07:24.773498 I | rafthttp: starting peer d1e4193dc152cc39...
2016-06-08 21:07:24.773867 I | rafthttp: started HTTP pipelining with peer d1e4193dc152cc39
2016-06-08 21:07:24.776804 I | rafthttp: started streaming with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:24.783899 I | rafthttp: starting peer 46fc68d58043bec5...
2016-06-08 21:07:24.784490 I | rafthttp: started HTTP pipelining with peer 46fc68d58043bec5
2016-06-08 21:07:24.793119 I | rafthttp: started peer 46fc68d58043bec5
2016-06-08 21:07:24.794648 I | rafthttp: added peer 46fc68d58043bec5
2016-06-08 21:07:24.794770 I | rafthttp: starting peer 70bca064fa6854d2...
2016-06-08 21:07:24.795128 I | rafthttp: started HTTP pipelining with peer 70bca064fa6854d2
2016-06-08 21:07:24.795579 I | rafthttp: started streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:24.795659 I | rafthttp: started streaming with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:24.796417 I | rafthttp: started streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:24.796480 I | rafthttp: started streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:24.796526 I | rafthttp: started streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:24.796879 I | rafthttp: started streaming with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:24.797104 I | rafthttp: started streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:24.797186 I | rafthttp: starting peer 46fc68d58043bec5...
2016-06-08 21:07:24.797724 I | rafthttp: started HTTP pipelining with peer 46fc68d58043bec5
2016-06-08 21:07:24.798435 I | rafthttp: started streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:24.799913 I | rafthttp: started streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:24.800121 I | rafthttp: started peer d1e4193dc152cc39
2016-06-08 21:07:24.800218 I | rafthttp: started streaming with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:24.800455 I | rafthttp: added peer d1e4193dc152cc39
2016-06-08 21:07:24.800556 I | etcdserver: set snapshot count to default 10000
2016-06-08 21:07:24.800623 I | etcdserver: starting server... [version: 3.0.0-beta.0+git, cluster version: to_be_decided]
2016-06-08 21:07:24.800958 I | rafthttp: started peer 70bca064fa6854d2
2016-06-08 21:07:24.801262 I | rafthttp: started streaming with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:24.801334 I | rafthttp: started streaming with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:24.801647 I | rafthttp: started streaming with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:24.801815 I | rafthttp: added peer 70bca064fa6854d2
2016-06-08 21:07:24.801889 I | etcdserver: set snapshot count to default 10000
2016-06-08 21:07:24.801938 I | etcdserver: starting server... [version: 3.0.0-beta.0+git, cluster version: to_be_decided]
2016-06-08 21:07:24.803612 I | rafthttp: started peer 46fc68d58043bec5
2016-06-08 21:07:24.803897 I | rafthttp: added peer 46fc68d58043bec5
2016-06-08 21:07:24.803966 I | rafthttp: starting peer d1e4193dc152cc39...
2016-06-08 21:07:24.804265 I | rafthttp: started streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:24.804322 I | rafthttp: started HTTP pipelining with peer d1e4193dc152cc39
2016-06-08 21:07:24.805946 I | integration: launched node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:24.806710 I | rafthttp: peer 46fc68d58043bec5 became active
2016-06-08 21:07:24.806795 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:24.807826 I | membership: added member 46fc68d58043bec5 [http://127.0.0.1:21513] to cluster 1de0725f07555f79
2016-06-08 21:07:24.808286 I | rafthttp: peer d1e4193dc152cc39 became active
2016-06-08 21:07:24.808368 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 writer)
2016-06-08 21:07:24.808506 I | raft: 46fc68d58043bec5 is starting a new election at term 1
2016-06-08 21:07:24.808588 I | raft: 46fc68d58043bec5 became candidate at term 2
2016-06-08 21:07:24.808645 I | raft: 46fc68d58043bec5 received vote from 46fc68d58043bec5 at term 2
2016-06-08 21:07:24.808756 I | raft: 46fc68d58043bec5 [logterm: 1, index: 3] sent vote request to 70bca064fa6854d2 at term 2
2016-06-08 21:07:24.808839 I | rafthttp: started streaming with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:24.808923 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream Message writer)
2016-06-08 21:07:24.808994 I | rafthttp: started streaming with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:24.809330 I | rafthttp: started streaming with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:24.809391 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:24.809644 I | rafthttp: started streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:24.809730 I | rafthttp: started streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:24.809830 I | rafthttp: started streaming with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:24.810115 I | raft: 46fc68d58043bec5 [logterm: 1, index: 3] sent vote request to d1e4193dc152cc39 at term 2
2016-06-08 21:07:24.810220 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:24.810304 I | integration: launched node7632688164621677138 (unix://localhost:node7632688164621677138.sock.bridge)
2016-06-08 21:07:24.810348 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:24.810425 I | membership: added member 46fc68d58043bec5 [http://127.0.0.1:21513] to cluster 1de0725f07555f79
2016-06-08 21:07:24.810526 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:24.811005 I | membership: added member 70bca064fa6854d2 [http://127.0.0.1:21517] to cluster 1de0725f07555f79
2016-06-08 21:07:24.811250 I | rafthttp: peer 70bca064fa6854d2 became active
2016-06-08 21:07:24.811299 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream Message writer)
2016-06-08 21:07:24.811421 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:24.811523 I | rafthttp: started peer d1e4193dc152cc39
2016-06-08 21:07:24.811807 I | rafthttp: added peer d1e4193dc152cc39
2016-06-08 21:07:24.811855 I | etcdserver: set snapshot count to default 10000
2016-06-08 21:07:24.811900 I | etcdserver: starting server... [version: 3.0.0-beta.0+git, cluster version: to_be_decided]
2016-06-08 21:07:24.812140 I | membership: added member d1e4193dc152cc39 [http://127.0.0.1:21515] to cluster 1de0725f07555f79
2016-06-08 21:07:24.812342 I | membership: added member 70bca064fa6854d2 [http://127.0.0.1:21517] to cluster 1de0725f07555f79
2016-06-08 21:07:24.812681 I | rafthttp: peer 46fc68d58043bec5 became active
2016-06-08 21:07:24.812736 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:24.821856 I | raft: d1e4193dc152cc39 [term: 1] received a MsgVote message with higher term from 46fc68d58043bec5 [term: 2]
2016-06-08 21:07:24.821949 I | raft: d1e4193dc152cc39 became follower at term 2
2016-06-08 21:07:24.822013 I | raft: d1e4193dc152cc39 [logterm: 1, index: 3, vote: 0] voted for 46fc68d58043bec5 [logterm: 1, index: 3] at term 2
2016-06-08 21:07:24.824369 I | membership: added member d1e4193dc152cc39 [http://127.0.0.1:21515] to cluster 1de0725f07555f79
2016-06-08 21:07:24.826509 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:24.827087 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:24.827373 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:24.827547 I | rafthttp: peer d1e4193dc152cc39 became active
2016-06-08 21:07:24.827639 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:24.829199 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:24.829636 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 writer)
2016-06-08 21:07:24.829826 I | rafthttp: peer 70bca064fa6854d2 became active
2016-06-08 21:07:24.829903 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 writer)
2016-06-08 21:07:24.830394 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream Message writer)
2016-06-08 21:07:24.830597 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:24.830692 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 writer)
2016-06-08 21:07:24.830910 I | raft: 46fc68d58043bec5 received vote from d1e4193dc152cc39 at term 2
2016-06-08 21:07:24.830959 I | raft: 46fc68d58043bec5 [quorum:2] has received 2 votes and 0 vote rejections
2016-06-08 21:07:24.831023 I | raft: 46fc68d58043bec5 became leader at term 2
2016-06-08 21:07:24.831073 I | raft: raft.node: 46fc68d58043bec5 elected leader 46fc68d58043bec5 at term 2
2016-06-08 21:07:24.831132 I | integration: launched node5947063075280645312 (unix://localhost:node5947063075280645312.sock.bridge)
2016-06-08 21:07:24.831193 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream Message writer)
2016-06-08 21:07:24.831256 I | raft: 70bca064fa6854d2 [logterm: 1, index: 3, vote: 0] ignored vote from 46fc68d58043bec5 [logterm: 1, index: 3] at term 1: lease is not expired (remaining ticks: 1)
2016-06-08 21:07:24.831331 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:24.831515 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:24.831739 I | membership: added member 46fc68d58043bec5 [http://127.0.0.1:21513] to cluster 1de0725f07555f79
2016-06-08 21:07:24.831870 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:24.832386 I | membership: added member 70bca064fa6854d2 [http://127.0.0.1:21517] to cluster 1de0725f07555f79
2016-06-08 21:07:24.832905 I | raft: raft.node: d1e4193dc152cc39 elected leader 46fc68d58043bec5 at term 2
2016-06-08 21:07:24.833013 I | raft: 70bca064fa6854d2 [term: 1] received a MsgApp message with higher term from 46fc68d58043bec5 [term: 2]
2016-06-08 21:07:24.833097 I | raft: 70bca064fa6854d2 became follower at term 2
2016-06-08 21:07:24.833157 I | raft: raft.node: 70bca064fa6854d2 elected leader 46fc68d58043bec5 at term 2
2016-06-08 21:07:24.833345 I | membership: added member d1e4193dc152cc39 [http://127.0.0.1:21515] to cluster 1de0725f07555f79
2016-06-08 21:07:24.835819 I | etcdserver: published {Name:node7632688164621677138 ClientURLs:[http://127.0.0.1:21516]} to cluster 1de0725f07555f79
2016-06-08 21:07:24.836284 I | etcdserver: published {Name:node2127541176347271374 ClientURLs:[http://127.0.0.1:21514]} to cluster 1de0725f07555f79
2016-06-08 21:07:24.836927 I | etcdserver: published {Name:node5947063075280645312 ClientURLs:[http://127.0.0.1:21518]} to cluster 1de0725f07555f79
2016-06-08 21:07:24.837341 I | etcdserver: setting up the initial cluster version to 3.0
2016-06-08 21:07:24.838355 N | membership: set the initial cluster version to 3.0
2016-06-08 21:07:24.838666 N | membership: set the initial cluster version to 3.0
2016-06-08 21:07:24.838847 N | membership: set the initial cluster version to 3.0
2016-06-08 21:07:24.851561 I | integration: stopping node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:24.852067 I | rafthttp: stopping peer 70bca064fa6854d2...
2016-06-08 21:07:24.852707 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:24.853027 I | rafthttp: closed the TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 writer)
2016-06-08 21:07:24.853084 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:24.853945 I | rafthttp: closed the TCP streaming connection with peer 70bca064fa6854d2 (stream Message writer)
2016-06-08 21:07:24.854002 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:24.854209 I | rafthttp: stopped HTTP pipelining with peer 70bca064fa6854d2
2016-06-08 21:07:24.854312 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:24.854400 W | rafthttp: lost the TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:24.854458 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:24.854671 W | rafthttp: lost the TCP streaming connection with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:24.854762 E | rafthttp: failed to read 70bca064fa6854d2 on stream Message (net/http: request canceled)
2016-06-08 21:07:24.854818 I | rafthttp: peer 70bca064fa6854d2 became inactive
2016-06-08 21:07:24.854894 I | rafthttp: stopped peer 70bca064fa6854d2
2016-06-08 21:07:24.854940 I | rafthttp: stopping peer d1e4193dc152cc39...
2016-06-08 21:07:24.855012 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:24.855319 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:24.855922 I | rafthttp: closed the TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 writer)
2016-06-08 21:07:24.855971 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:24.856330 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:24.856752 I | rafthttp: closed the TCP streaming connection with peer d1e4193dc152cc39 (stream Message writer)
2016-06-08 21:07:24.856808 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:24.856919 I | rafthttp: stopped HTTP pipelining with peer d1e4193dc152cc39
2016-06-08 21:07:24.857102 W | rafthttp: lost the TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:24.857160 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:24.857332 W | rafthttp: lost the TCP streaming connection with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:24.857390 E | rafthttp: failed to read d1e4193dc152cc39 on stream Message (net/http: request canceled)
2016-06-08 21:07:24.857428 I | rafthttp: peer d1e4193dc152cc39 became inactive
2016-06-08 21:07:24.857483 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:24.857537 I | rafthttp: stopped peer d1e4193dc152cc39
2016-06-08 21:07:24.865316 I | integration: stopped node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:24.954832 E | rafthttp: failed to dial 46fc68d58043bec5 on stream MsgApp v2 (dial tcp 127.0.0.1:21513: getsockopt: connection refused)
2016-06-08 21:07:24.955171 I | rafthttp: peer 46fc68d58043bec5 became inactive
2016-06-08 21:07:24.957382 E | rafthttp: failed to dial 46fc68d58043bec5 on stream MsgApp v2 (dial tcp 127.0.0.1:21513: getsockopt: connection refused)
2016-06-08 21:07:24.957669 I | rafthttp: peer 46fc68d58043bec5 became inactive
2016-06-08 21:07:24.965643 I | integration: restarting node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:24.966967 I | integration: launching node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:24.982317 I | etcdserver: name = node2127541176347271374
2016-06-08 21:07:24.982476 I | etcdserver: data dir = /tmp/etcd002323622
2016-06-08 21:07:24.982614 I | etcdserver: member dir = /tmp/etcd002323622/member
2016-06-08 21:07:24.982738 I | etcdserver: heartbeat = 10ms
2016-06-08 21:07:24.982853 I | etcdserver: election = 100ms
2016-06-08 21:07:24.982961 I | etcdserver: snapshot count = 0
2016-06-08 21:07:24.983135 I | etcdserver: advertise client URLs = http://127.0.0.1:21514
2016-06-08 21:07:24.983512 I | raft: 70bca064fa6854d2 is starting a new election at term 2
2016-06-08 21:07:24.983917 I | raft: 70bca064fa6854d2 became candidate at term 3
2016-06-08 21:07:24.984185 I | raft: 70bca064fa6854d2 received vote from 70bca064fa6854d2 at term 3
2016-06-08 21:07:24.984493 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9] sent vote request to 46fc68d58043bec5 at term 3
2016-06-08 21:07:24.984745 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9] sent vote request to d1e4193dc152cc39 at term 3
2016-06-08 21:07:24.984921 I | raft: raft.node: 70bca064fa6854d2 lost leader 46fc68d58043bec5 at term 3
2016-06-08 21:07:24.985231 I | raft: d1e4193dc152cc39 is starting a new election at term 2
2016-06-08 21:07:24.985571 I | raft: d1e4193dc152cc39 became candidate at term 3
2016-06-08 21:07:24.985783 I | raft: d1e4193dc152cc39 received vote from d1e4193dc152cc39 at term 3
2016-06-08 21:07:24.985989 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9] sent vote request to 46fc68d58043bec5 at term 3
2016-06-08 21:07:24.986198 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9] sent vote request to 70bca064fa6854d2 at term 3
2016-06-08 21:07:24.986367 I | raft: raft.node: d1e4193dc152cc39 lost leader 46fc68d58043bec5 at term 3
2016-06-08 21:07:24.986616 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9, vote: d1e4193dc152cc39] rejected vote from 70bca064fa6854d2 [logterm: 2, index: 9] at term 3
2016-06-08 21:07:24.986756 I | etcdserver: restarting member 46fc68d58043bec5 in cluster 1de0725f07555f79 at commit index 9
2016-06-08 21:07:24.987273 I | raft: 46fc68d58043bec5 became follower at term 2
2016-06-08 21:07:24.987525 I | raft: newRaft 46fc68d58043bec5 [peers: [], term: 2, commit: 9, applied: 0, lastindex: 9, lastterm: 2]
2016-06-08 21:07:24.988486 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9, vote: 70bca064fa6854d2] rejected vote from d1e4193dc152cc39 [logterm: 2, index: 9] at term 3
2016-06-08 21:07:24.988792 I | raft: 70bca064fa6854d2 received vote rejection from d1e4193dc152cc39 at term 3
2016-06-08 21:07:24.988969 I | raft: 70bca064fa6854d2 [quorum:2] has received 1 votes and 1 vote rejections
2016-06-08 21:07:24.989835 I | raft: d1e4193dc152cc39 received vote rejection from 70bca064fa6854d2 at term 3
2016-06-08 21:07:24.989992 I | raft: d1e4193dc152cc39 [quorum:2] has received 1 votes and 1 vote rejections
2016-06-08 21:07:24.994138 I | etcdserver: set snapshot count to default 10000
2016-06-08 21:07:24.994352 I | etcdserver: starting server... [version: 3.0.0-beta.0+git, cluster version: to_be_decided]
2016-06-08 21:07:24.997391 I | membership: added member 46fc68d58043bec5 [http://127.0.0.1:21513] to cluster 1de0725f07555f79
2016-06-08 21:07:24.998528 I | membership: added member 70bca064fa6854d2 [http://127.0.0.1:21517] to cluster 1de0725f07555f79
2016-06-08 21:07:24.998670 I | rafthttp: starting peer 70bca064fa6854d2...
2016-06-08 21:07:24.999458 I | rafthttp: started HTTP pipelining with peer 70bca064fa6854d2
2016-06-08 21:07:24.999770 I | raft: 46fc68d58043bec5 is starting a new election at term 2
2016-06-08 21:07:24.999925 I | raft: 46fc68d58043bec5 became candidate at term 3
2016-06-08 21:07:25.000022 I | raft: 46fc68d58043bec5 received vote from 46fc68d58043bec5 at term 3
2016-06-08 21:07:25.000131 I | raft: 46fc68d58043bec5 [logterm: 2, index: 9] sent vote request to 70bca064fa6854d2 at term 3
2016-06-08 21:07:25.001454 I | integration: launched node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:25.001603 I | integration: restarted node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:25.001718 I | rafthttp: started streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:25.003059 I | rafthttp: started streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:25.006542 I | rafthttp: started peer 70bca064fa6854d2
2016-06-08 21:07:25.007217 I | rafthttp: added peer 70bca064fa6854d2
2016-06-08 21:07:25.009019 I | rafthttp: started streaming with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:25.014010 I | rafthttp: started streaming with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:25.014138 I | membership: added member d1e4193dc152cc39 [http://127.0.0.1:21515] to cluster 1de0725f07555f79
2016-06-08 21:07:25.014651 I | rafthttp: starting peer d1e4193dc152cc39...
2016-06-08 21:07:25.015191 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9, vote: 70bca064fa6854d2] rejected vote from 46fc68d58043bec5 [logterm: 2, index: 9] at term 3
2016-06-08 21:07:25.015339 I | rafthttp: peer 70bca064fa6854d2 became active
2016-06-08 21:07:25.018849 W | rafthttp: closed an existing TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:25.019399 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:25.019766 I | rafthttp: peer 46fc68d58043bec5 became active
2016-06-08 21:07:25.019833 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:25.019879 W | rafthttp: closed an existing TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:25.019924 I | raft: 46fc68d58043bec5 received vote rejection from 70bca064fa6854d2 at term 3
2016-06-08 21:07:25.020009 I | raft: 46fc68d58043bec5 [quorum:2] has received 1 votes and 1 vote rejections
2016-06-08 21:07:25.020090 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:25.020370 I | rafthttp: started HTTP pipelining with peer d1e4193dc152cc39
2016-06-08 21:07:25.020572 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:25.021457 I | rafthttp: started streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:25.022519 I | rafthttp: started streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:25.024260 I | rafthttp: started streaming with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:25.024334 I | rafthttp: started streaming with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:25.024442 I | rafthttp: started peer d1e4193dc152cc39
2016-06-08 21:07:25.024741 I | rafthttp: added peer d1e4193dc152cc39
2016-06-08 21:07:25.025730 N | membership: set the initial cluster version to 3.0
2016-06-08 21:07:25.025826 I | rafthttp: peer d1e4193dc152cc39 became active
2016-06-08 21:07:25.025887 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:25.026264 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:25.027351 W | rafthttp: closed an existing TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:25.027581 W | rafthttp: closed an existing TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:25.027643 I | rafthttp: peer 46fc68d58043bec5 became active
2016-06-08 21:07:25.027688 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:38.601638 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:38.602162 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:38.602634 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 writer)
2016-06-08 21:07:38.603208 I | rafthttp: established a TCP streaming connection with peer 70bca064fa6854d2 (stream Message writer)
2016-06-08 21:07:38.603574 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:38.603942 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:38.604341 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 writer)
2016-06-08 21:07:38.604573 I | rafthttp: established a TCP streaming connection with peer d1e4193dc152cc39 (stream Message writer)
2016-06-08 21:07:38.604992 I | rafthttp: established a TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:38.605433 I | raft: 70bca064fa6854d2 is starting a new election at term 3
2016-06-08 21:07:38.605877 I | raft: 46fc68d58043bec5 is starting a new election at term 3
2016-06-08 21:07:38.606207 I | raft: 46fc68d58043bec5 became candidate at term 4
2016-06-08 21:07:38.606467 I | integration: terminating node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:38.606713 W | rafthttp: lost the TCP streaming connection with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:38.607033 E | rafthttp: failed to read d1e4193dc152cc39 on stream Message (read tcp 127.0.0.1:57605->127.0.0.1:21515: i/o timeout)
2016-06-08 21:07:38.607230 I | rafthttp: peer d1e4193dc152cc39 became inactive
2016-06-08 21:07:38.607387 E | etcdserver: publish error: etcdserver: request timed out
2016-06-08 21:07:38.607642 I | raft: 70bca064fa6854d2 became candidate at term 4
2016-06-08 21:07:38.607818 I | raft: 70bca064fa6854d2 received vote from 70bca064fa6854d2 at term 4
2016-06-08 21:07:38.608004 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9] sent vote request to 46fc68d58043bec5 at term 4
2016-06-08 21:07:38.608185 I | raft: d1e4193dc152cc39 is starting a new election at term 3
2016-06-08 21:07:38.608589 I | raft: d1e4193dc152cc39 became candidate at term 4
2016-06-08 21:07:38.608832 W | rafthttp: lost the TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:38.609278 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9] sent vote request to d1e4193dc152cc39 at term 4
2016-06-08 21:07:38.609638 I | raft: 46fc68d58043bec5 received vote from 46fc68d58043bec5 at term 4
2016-06-08 21:07:38.610059 I | raft: 46fc68d58043bec5 [logterm: 2, index: 9] sent vote request to 70bca064fa6854d2 at term 4
2016-06-08 21:07:38.610282 I | raft: d1e4193dc152cc39 received vote from d1e4193dc152cc39 at term 4
2016-06-08 21:07:38.611224 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9] sent vote request to 46fc68d58043bec5 at term 4
2016-06-08 21:07:38.611602 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9] sent vote request to 70bca064fa6854d2 at term 4
2016-06-08 21:07:38.611860 I | raft: 46fc68d58043bec5 [logterm: 2, index: 9] sent vote request to d1e4193dc152cc39 at term 4
2016-06-08 21:07:38.612117 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9, vote: d1e4193dc152cc39] rejected vote from 70bca064fa6854d2 [logterm: 2, index: 9] at term 4
2016-06-08 21:07:38.612359 I | raft: 46fc68d58043bec5 [logterm: 2, index: 9, vote: 46fc68d58043bec5] rejected vote from 70bca064fa6854d2 [logterm: 2, index: 9] at term 4
2016-06-08 21:07:38.612741 I | rafthttp: stopping peer 70bca064fa6854d2...
2016-06-08 21:07:38.614651 I | raft: 70bca064fa6854d2 [logterm: 2, index: 9, vote: 70bca064fa6854d2] rejected vote from d1e4193dc152cc39 [logterm: 2, index: 9] at term 4
2016-06-08 21:07:38.615032 I | rafthttp: closed the TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 writer)
2016-06-08 21:07:38.615208 I | raft: 70bca064fa6854d2 received vote rejection from d1e4193dc152cc39 at term 4
2016-06-08 21:07:38.615556 I | raft: 70bca064fa6854d2 [quorum:2] has received 1 votes and 1 vote rejections
2016-06-08 21:07:38.616253 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:38.617541 I | rafthttp: closed the TCP streaming connection with peer 70bca064fa6854d2 (stream Message writer)
2016-06-08 21:07:38.618279 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:38.618571 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:38.618880 I | raft: d1e4193dc152cc39 received vote rejection from 70bca064fa6854d2 at term 4
2016-06-08 21:07:38.619203 I | raft: d1e4193dc152cc39 [quorum:2] has received 1 votes and 1 vote rejections
2016-06-08 21:07:38.619519 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:38.619781 I | rafthttp: stopped HTTP pipelining with peer 70bca064fa6854d2
2016-06-08 21:07:38.620721 W | rafthttp: lost the TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:38.621077 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:38.621859 W | rafthttp: lost the TCP streaming connection with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:38.622164 E | rafthttp: failed to read 70bca064fa6854d2 on stream Message (net/http: request canceled)
2016-06-08 21:07:38.622320 I | rafthttp: peer 70bca064fa6854d2 became inactive
2016-06-08 21:07:38.622586 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:38.622858 I | rafthttp: stopped peer 70bca064fa6854d2
2016-06-08 21:07:38.623054 I | rafthttp: stopping peer d1e4193dc152cc39...
2016-06-08 21:07:38.624613 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:38.625830 I | rafthttp: closed the TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 writer)
2016-06-08 21:07:38.626105 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:38.627264 W | rafthttp: lost the TCP streaming connection with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:38.628489 I | rafthttp: closed the TCP streaming connection with peer d1e4193dc152cc39 (stream Message writer)
2016-06-08 21:07:38.628762 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:38.629213 I | rafthttp: stopped HTTP pipelining with peer d1e4193dc152cc39
2016-06-08 21:07:38.629566 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:38.629990 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:38.630198 I | rafthttp: stopped peer d1e4193dc152cc39
2016-06-08 21:07:38.651897 I | etcdserver: aborting publish because server is stopped
2016-06-08 21:07:38.670884 I | integration: terminated node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)
2016-06-08 21:07:38.671027 I | integration: terminating node7632688164621677138 (unix://localhost:node7632688164621677138.sock.bridge)
2016-06-08 21:07:38.671900 I | rafthttp: stopping peer 46fc68d58043bec5...
2016-06-08 21:07:38.673321 I | rafthttp: closed the TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:38.673456 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:38.674600 I | rafthttp: closed the TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:38.674750 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:38.674975 I | rafthttp: stopped HTTP pipelining with peer 46fc68d58043bec5
2016-06-08 21:07:38.675247 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:38.675381 I | rafthttp: stopped peer 46fc68d58043bec5
2016-06-08 21:07:38.675479 I | rafthttp: stopping peer 70bca064fa6854d2...
2016-06-08 21:07:38.675626 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:38.676928 I | rafthttp: closed the TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 writer)
2016-06-08 21:07:38.677256 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:38.677424 W | rafthttp: lost the TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:38.678078 W | rafthttp: lost the TCP streaming connection with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:38.678406 I | rafthttp: closed the TCP streaming connection with peer 70bca064fa6854d2 (stream Message writer)
2016-06-08 21:07:38.678501 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (writer)
2016-06-08 21:07:38.678689 I | rafthttp: stopped HTTP pipelining with peer 70bca064fa6854d2
2016-06-08 21:07:38.679016 W | rafthttp: lost the TCP streaming connection with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:38.679121 E | rafthttp: failed to read 70bca064fa6854d2 on stream MsgApp v2 (net/http: request canceled)
2016-06-08 21:07:38.679194 I | rafthttp: peer 70bca064fa6854d2 became inactive
2016-06-08 21:07:38.679293 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (stream MsgApp v2 reader)
2016-06-08 21:07:38.679578 W | rafthttp: lost the TCP streaming connection with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:38.679697 I | rafthttp: stopped streaming with peer 70bca064fa6854d2 (stream Message reader)
2016-06-08 21:07:38.679783 I | rafthttp: stopped peer 70bca064fa6854d2
2016-06-08 21:07:38.701653 I | integration: terminated node7632688164621677138 (unix://localhost:node7632688164621677138.sock.bridge)
2016-06-08 21:07:38.701805 I | integration: terminating node5947063075280645312 (unix://localhost:node5947063075280645312.sock.bridge)
2016-06-08 21:07:38.702654 I | rafthttp: stopping peer 46fc68d58043bec5...
2016-06-08 21:07:38.704018 I | rafthttp: closed the TCP streaming connection with peer 46fc68d58043bec5 (stream MsgApp v2 writer)
2016-06-08 21:07:38.704170 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:38.705385 I | rafthttp: closed the TCP streaming connection with peer 46fc68d58043bec5 (stream Message writer)
2016-06-08 21:07:38.705518 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (writer)
2016-06-08 21:07:38.705789 I | rafthttp: stopped HTTP pipelining with peer 46fc68d58043bec5
2016-06-08 21:07:38.705912 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (stream MsgApp v2 reader)
2016-06-08 21:07:38.706023 I | rafthttp: stopped streaming with peer 46fc68d58043bec5 (stream Message reader)
2016-06-08 21:07:38.706101 I | rafthttp: stopped peer 46fc68d58043bec5
2016-06-08 21:07:38.706160 I | rafthttp: stopping peer d1e4193dc152cc39...
2016-06-08 21:07:38.707150 I | rafthttp: closed the TCP streaming connection with peer d1e4193dc152cc39 (stream MsgApp v2 writer)
2016-06-08 21:07:38.707221 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:38.708307 I | rafthttp: closed the TCP streaming connection with peer d1e4193dc152cc39 (stream Message writer)
2016-06-08 21:07:38.708415 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (writer)
2016-06-08 21:07:38.708604 I | rafthttp: stopped HTTP pipelining with peer d1e4193dc152cc39
2016-06-08 21:07:38.708745 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (stream MsgApp v2 reader)
2016-06-08 21:07:38.708855 I | rafthttp: stopped peer d1e4193dc152cc39
2016-06-08 21:07:38.709115 I | rafthttp: stopped streaming with peer d1e4193dc152cc39 (stream Message reader)
2016-06-08 21:07:38.728897 I | integration: terminated node5947063075280645312 (unix://localhost:node5947063075280645312.sock.bridge)
==================
WARNING: DATA RACE
Read by goroutine 238:
  testing.tRunner.func1()
      /usr/local/golang/1.6.2/go/src/testing/testing.go:461 +0x166
  runtime.call32()
      /usr/local/golang/1.6.2/go/src/runtime/asm_amd64.s:472 +0x3d
  testing.(*common).Fatalf()
      /usr/local/golang/1.6.2/go/src/testing/testing.go:389 +0x94
  github.com/coreos/etcd/clientv3/integration.TestKVGetRetry()
      /home/runner/workspace/src/github.com/coreos/etcd/gopath/src/github.com/coreos/etcd/clientv3/integration/kv_test.go:570 +0x95e
  testing.tRunner()
      /usr/local/golang/1.6.2/go/src/testing/testing.go:473 +0xdc

Previous write by goroutine 66:
  [failed to restore the stack]

Goroutine 238 (running) created at:
  testing.RunTests()
      /usr/local/golang/1.6.2/go/src/testing/testing.go:582 +0xae2
  testing.(*M).Run()
      /usr/local/golang/1.6.2/go/src/testing/testing.go:515 +0x11d
  github.com/coreos/etcd/clientv3/integration.TestMain()
      /home/runner/workspace/src/github.com/coreos/etcd/gopath/src/github.com/coreos/etcd/clientv3/integration/main_test.go:15 +0x2e
  main.main()
      github.com/coreos/etcd/clientv3/integration/_test/_testmain.go:150 +0x20a

Goroutine 66 (finished) created at:
  github.com/coreos/etcd/clientv3/integration.TestKVGetRetry()
      /home/runner/workspace/src/github.com/coreos/etcd/gopath/src/github.com/coreos/etcd/clientv3/integration/kv_test.go:563 +0x7c4
  testing.tRunner()
      /usr/local/golang/1.6.2/go/src/testing/testing.go:473 +0xdc
==================
--- FAIL: TestKVGetRetry (14.01s)
    kv_test.go:570: timed out waiting for get
    kv_test.go:548: grpc: the client connection is closing
@gyuho gyuho changed the title TestKVGetRetry fails test: TestKVGetRetry Jun 8, 2016
@gyuho
Copy link
Contributor Author

gyuho commented Jun 8, 2016

Can't reproduce that race condition in my machine. Looked at the logs, and it seemed like it just needed more time for leader election?

2016-06-08 21:07:24.985783 I | raft: d1e4193dc152cc39 received vote from d1e4193dc152cc39 at term 3
2016-06-08 21:07:24.985989 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9] sent vote request to 46fc68d58043bec5 at term 3
2016-06-08 21:07:24.986198 I | raft: d1e4193dc152cc39 [logterm: 2, index: 9] sent vote request to 70bca064fa6854d2 at term 3
2016-06-08 21:07:24.986367 I | raft: raft.node: d1e4193dc152cc39 lost leader 46fc68d58043bec5 at term 3

But no leader got elected within 5 seconds, so it timed out (test errored).

2016-06-08 21:07:25.001603 I | integration: restarted node2127541176347271374 (unix://localhost:node2127541176347271374.sock.bridge)

@heyitsanthony
Copy link
Contributor

@gyuho it could be that it's killing the leader and so it's dropping the (linearized) get request. I think the fix is to force it to use/kill a follower instead of always using member 0 (which could be a leader).

@gyuho
Copy link
Contributor Author

gyuho commented Jun 8, 2016

@heyitsanthony Yeah it was killing the leader. I will fix the test to shut down the follower. Thanks.

gyuho added a commit to gyuho/etcd that referenced this issue Jun 8, 2016
@gyuho
clientv3/integration: WaitLeader to follower
09279a8 
Fix etcd-io#5601.
gyuho added a commit to gyuho/etcd that referenced this issue Jun 8, 2016
@gyuho
clientv3/integration: WaitLeader to follower
840993b 
Fix etcd-io#5601.
@gyuho gyuho mentioned this issue Jun 8, 2016
Merged
gyuho added a commit to gyuho/etcd that referenced this issue Jun 8, 2016
@gyuho
clientv3/integration: WaitLeader to follower
444a3e7 
Fix etcd-io#5601.
gyuho added a commit to gyuho/etcd that referenced this issue Jun 8, 2016
@gyuho
clientv3/integration: WaitLeader to follower
7530731 
Fix etcd-io#5601.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/testing
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gyuho @heyitsanthony