Skip to content

[Regression][2] libcurl does not finish CURLOPT_UPLOAD request after a connection lost without data tranfer #13740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ligurio opened this issue May 22, 2024 · 6 comments
Assignees

Comments

@ligurio
Copy link
Contributor

ligurio commented May 22, 2024

I did this

There was an issue in Tarantool 1 due to regression in libcurl (#11769), it was reported on 23 Aug, 2023. Issue was fixed in upstream on 4 Sept, 2023 and fix was confirmed by our tests for Tarantool. Unfortunately, this fix was ruined by refactoring made in commit d7b6ce6 ("lib: replace readwrite with write_resp") [3] on 1 Dec 2023, and now it is reproduced again. Then it was fixed in #13260 and regression test was added within fix. However, commit 270a25c that fixes #13337 broke fix again.

I expected the following

Fix of the original problem.

curl/libcurl version

270a25c

operating system

Ubuntu 22.04 amd64

Footnotes

  1. https://github.com/tarantool/tarantool/issues/9086 2

@icing
Copy link
Contributor

icing commented May 22, 2024

Sorry to hear that the problem seems to have resurfaced. As you remember, we added a specific test case for the situation and that has been succeeding ever since.

If we have exactly that problem again, the test cases just does not seem to cover the complete scenario that your application runs into. I will try to run the original reported client #11769 with an etcd to see if that still fails.

@icing
Copy link
Contributor

icing commented May 22, 2024

I have run the example exactly as described in #11769 and the test client terminates after killing the etcd server.

Do you have more information about the problems you see?

@ligurio
Copy link
Contributor Author

ligurio commented May 22, 2024

@icing Stefan, I have a full log with messages produced by etcd and curl built with debug.

curl version: d18bba68d678fdce9aeb47c01ac6e705931129d2

log.txt
2024-05-22 15:33:50.282 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 startup
Tarantool version is 3.2.0-entrypoint-39-gb4d586239a
�[34metcd #2 | �[0mWARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.�[0m
�[34metcd #2 | �[0m	A future release of golang/protobuf will delete this package,�[0m
�[34metcd #2 | �[0m	which has long been excluded from the compatibility promise.�[0m
�[34metcd #2 | �[0m�[0m
�[35metcd #1 | �[0mWARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.�[0m
�[35metcd #1 | �[0m	A future release of golang/protobuf will delete this package,�[0m
�[35metcd #1 | �[0m	which has long been excluded from the compatibility promise.�[0m
�[35metcd #1 | �[0m�[0m
�[36metcd #3 | �[0mWARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.�[0m
�[36metcd #3 | �[0m	A future release of golang/protobuf will delete this package,�[0m
�[36metcd #3 | �[0m	which has long been excluded from the compatibility promise.�[0m
�[36metcd #3 | �[0m�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286424 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://localhost:2381�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286475 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/tmp/Jme3bU/test2�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286494 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://localhost:2382�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286499 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=test1=http://localhost:2380,test2=http://localhost:2382,test3=http://localhost:2384�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286503 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286510 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://localhost:2381�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286515 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://localhost:2382�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286524 I | pkg/flags: recognized and used environment variable ETCD_NAME=test2�[0m
�[34metcd #2 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286557 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286561 I | etcdmain: etcd Version: 3.4.32�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286564 I | etcdmain: Git SHA: 9ac4e7ef0�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286566 I | etcdmain: Go Version: go1.21.9�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286568 I | etcdmain: Go OS/Arch: linux/amd64�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286570 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8�[0m
�[34metcd #2 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286593 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286581 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://localhost:2379�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286609 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/tmp/Jme3bU/test1�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286623 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://localhost:2380�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286627 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=test1=http://localhost:2380,test2=http://localhost:2382,test3=http://localhost:2384�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286630 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286635 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://localhost:2379�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286638 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://localhost:2380�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286645 I | pkg/flags: recognized and used environment variable ETCD_NAME=test1�[0m
�[35metcd #1 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286667 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286670 I | etcdmain: etcd Version: 3.4.32�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286672 I | etcdmain: Git SHA: 9ac4e7ef0�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286674 I | etcdmain: Go Version: go1.21.9�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286676 I | etcdmain: Go OS/Arch: linux/amd64�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286678 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8�[0m
�[35metcd #1 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286691 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286921 I | embed: name = test1�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286925 I | embed: data dir = /tmp/Jme3bU/test1�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286927 I | embed: member dir = /tmp/Jme3bU/test1/member�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286929 I | embed: heartbeat = 100ms�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286930 I | embed: election = 1000ms�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286932 I | embed: snapshot count = 100000�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:50.286938 I | embed: advertise client URLs = http://localhost:2379�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286907 I | embed: name = test2�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286911 I | embed: data dir = /tmp/Jme3bU/test2�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286913 I | embed: member dir = /tmp/Jme3bU/test2/member�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286916 I | embed: heartbeat = 100ms�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286921 I | embed: election = 1000ms�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286923 I | embed: snapshot count = 100000�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:50.286931 I | embed: advertise client URLs = http://localhost:2381�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288123 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://localhost:2383�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288149 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/tmp/Jme3bU/test3�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288161 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://localhost:2384�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288164 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=test1=http://localhost:2380,test2=http://localhost:2382,test3=http://localhost:2384�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288167 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288174 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://localhost:2383�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288177 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://localhost:2384�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288181 I | pkg/flags: recognized and used environment variable ETCD_NAME=test3�[0m
�[36metcd #3 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288199 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288202 I | etcdmain: etcd Version: 3.4.32�[0m
�[36metcd 2024-05-22 15:33:50.383 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 startup
#3 | �[0m2024-05-22 15:33:50.288203 I | etcdmain: Git SHA: 9ac4e7ef0�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288204 I | etcdmain: Go Version: go1.21.9�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288206 I | etcdmain: Go OS/Arch: linux/amd64�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288208 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8�[0m
�[36metcd #3 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288218 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288432 I | embed: name = test3�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288439 I | embed: data dir = /tmp/Jme3bU/test3�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288450 I | embed: member dir = /tmp/Jme3bU/test3/member�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288452 I | embed: heartbeat = 100ms�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288454 I | embed: election = 1000ms�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288456 I | embed: snapshot count = 100000�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:50.288463 I | embed: advertise client URLs = http://localhost:2383�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.301963 I | etcdserver: starting member 7e3bd17c66e004e8 in cluster de368b0e4577b2d6�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=()�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 became follower at term 0�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: newRaft 7e3bd17c66e004e8 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 became follower at term 1�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=(9096094204636955880)�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=(9096094204636955880 10276657743932975437)�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.301939 I | etcdserver: starting member fa252746f913ab7f in cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=()�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f became follower at term 0�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: newRaft fa252746f913ab7f [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f became follower at term 1�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=(9096094204636955880)�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=(9096094204636955880 10276657743932975437)�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.301939 I | etcdserver: starting member 8e9e05c52164694d in cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=()�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d became follower at term 0�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d became follower at term 1�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880)�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437)�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.303865 W | auth: simple token is not cryptographically signed�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.304194 W | auth: simple token is not cryptographically signed�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.304641 W | auth: simple token is not cryptographically signed�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.308368 I | rafthttp: starting peer 7e3bd17c66e004e8...�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.308421 I | rafthttp: started HTTP pipelining with peer 7e3bd17c66e004e8�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.310033 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.308602 I | rafthttp: starting peer 8e9e05c52164694d...�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.308692 I | rafthttp: started HTTP pipelining with peer 8e9e05c52164694d�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.309930 I | rafthttp: started streaming with peer 8e9e05c52164694d (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.310102 I | rafthttp: starting peer 7e3bd17c66e004e8...�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.310160 I | rafthttp: started HTTP pipelining with peer 7e3bd17c66e004e8�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.310740 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.311036 I | rafthttp: started streaming with peer 8e9e05c52164694d (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.311218 I | rafthttp: started peer 7e3bd17c66e004e8�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.311262 I | rafthttp: added peer 7e3bd17c66e004e8�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.311417 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.311285 I | rafthttp: starting peer fa252746f913ab7f...�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.312549 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.312573 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.313865 I | rafthttp: started HTTP pipelining with peer fa252746f913ab7f�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.313358 I | rafthttp: started peer 8e9e05c52164694d�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.313397 I | rafthttp: added peer 8e9e05c52164694d�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.313418 I | rafthttp: starting peer fa252746f913ab7f...�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.313534 I | rafthttp: started HTTP pipelining with peer fa252746f913ab7f�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.313682 I | rafthttp: started streaming with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.314743 I | rafthttp: started streaming with peer fa252746f913ab7f (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.314836 I | rafthttp: started streaming with peer fa252746f913ab7f (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.315053 I | rafthttp: started streaming with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.315589 I | rafthttp: started peer fa252746f913ab7f�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.315620 I | rafthttp: added peer fa252746f913ab7f�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.315634 I | etcdserver: starting server... [version: 3.4.32, cluster version: to_be_decided]�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.316326 I | rafthttp: started streaming with peer fa252746f913ab7f (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.316395 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test1/member/snap, suffix: snap.db, max: 5, interval: 30s�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.316452 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test1/member/snap, suffix: snap, max: 5, interval: 30s�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.316463 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test1/member/wal, suffix: wal, max: 5, interval: 30s�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.316785 I | etcdserver/membership: added member 7e3bd17c66e004e8 [http://localhost:2382] to cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.316922 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.317016 I | etcdserver/membership: added member fa252746f913ab7f [http://localhost:2384] to cluster de368b0e4577b2d6�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.313743 I | rafthttp: started streaming with peer 8e9e05c52164694d (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315005 I | rafthttp: started peer fa252746f913ab7f�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315017 I | rafthttp: started streaming with peer fa252746f913ab7f (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315035 I | rafthttp: added peer fa252746f913ab7f�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315042 I | rafthttp: started streaming with peer fa252746f913ab7f (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315049 I | etcdserver: starting server... [version: 3.4.32, cluster version: to_be_decided]�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315090 I | rafthttp: started streaming with peer fa252746f913ab7f (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315100 I | rafthttp: started streaming with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315744 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test2/member/snap, suffix: snap.db, max: 5, interval: 30s�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315783 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test2/member/snap, suffix: snap, max: 5, interval: 30s�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.315794 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test2/member/wal, suffix: wal, max: 5, interval: 30s�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.316619 I | etcdserver/membership: added member 7e3bd17c66e004e8 [http://localhost:2382] to cluster de368b0e4577b2d6�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.316767 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster de368b0e4577b2d6�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.316870 I | etcdserver/membership: added member fa252746f913ab7f [http://localhost:2384] to cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.315145 I | rafthttp: started peer 7e3bd17c66e004e8�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.315177 I | rafthttp: added peer 7e3bd17c66e004e8�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.315194 I | rafthttp: starting peer 8e9e05c52164694d...�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.315363 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.315394 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.315416 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.316674 I | rafthttp: started HTTP pipelining with peer 8e9e05c52164694d�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.317886 I | rafthttp: started streaming with peer 8e9e05c52164694d (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318039 I | rafthttp: started streaming with peer 8e9e05c52164694d (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318163 I | rafthttp: started peer 8e9e05c52164694d�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318189 I | rafthttp: added peer 8e9e05c52164694d�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318198 I | rafthttp: started streaming with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318231 I | rafthttp: started streaming with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318297 I | etcdserver: starting server... [version: 3.4.32, cluster version: to_be_decided]�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318922 I | rafthttp: peer 7e3bd17c66e004e8 became active�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318935 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318941 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318982 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test3/member/snap, suffix: snap.db, max: 5, interval: 30s�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.318994 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test3/member/snap, suffix: snap, max: 5, interval: 30s�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.319003 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test3/member/wal, suffix: wal, max: 5, interval: 30s�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.318239 I | embed: listening for peers on 127.0.0.1:2382�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.318754 I | rafthttp: peer fa252746f913ab7f became active�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.318769 I | rafthttp: peer 8e9e05c52164694d became active�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.318793 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.318800 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.318866 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.318885 I | rafthttp: peer 7e3bd17c66e004e8 became active�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319032 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319071 I | embed: listening for peers on 127.0.0.1:2380�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319640 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319769 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319808 I | rafthttp: peer fa252746f913ab7f became active�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319821 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream Message writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319827 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.319971 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.319526 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.319760 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.319829 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.319686 I | etcdserver/membership: added member 7e3bd17c66e004e8 [http://localhost:2382] to cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.319792 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.319879 I | rafthttp: peer 8e9e05c52164694d became active�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.319893 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.319900 I | etcdserver/membership: added member fa252746f913ab7f [http://localhost:2384] to cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.320080 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.321489 I | embed: listening for peers on 127.0.0.1:2384�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.321709 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.321728 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.321788 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.321805 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.321745 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.321770 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.321737 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.321800 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.366051 I | etcdserver: 7e3bd17c66e004e8 initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.366129 I | etcdserver: 8e9e05c52164694d initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.369464 I | etcdserver: fa252746f913ab7f initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 is starting a new election at term 1�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 became candidate at term 2�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 received MsgVoteResp from 7e3bd17c66e004e8 at term 2�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 [logterm: 1, index: 3] sent MsgVote request to 8e9e05c52164694d at term 2�[0m
�[34metcd #2 | �[0mraft2024/02024-05-22 15:33:52.512 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 startup
2024-05-22 15:33:52.513 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#3 startup
* !!! WARNING !!!
* This is a debug build of libcurl, do not use in production.
* STATE: INIT => SETUP handle 0x55d511b6e978; line 1966
* STATE: SETUP => CONNECT handle 0x55d511b6e978; line 1982
* Added connection 0. The cache now contains 1 members
* Host localhost:2379 was resolved.
* IPv4: 127.0.0.1
* STATE: CONNECT => CONNECTING handle 0x55d511b6e978; line 2022
*   Trying 127.0.0.1:2379...
* Connected to localhost (127.0.0.1) port 2379
* STATE: CONNECTING => PROTOCONNECT handle 0x55d511b6e978; line 2130
* STATE: PROTOCONNECT => DO handle 0x55d511b6e978; line 2159
* cr_exp100_read, start AWAITING_CONTINUE
> POST /v3/watch HTTP/1.1
Host: localhost:2379
Transfer-Encoding: chunked
Accept: */*
Connection: Keep-Alive
Keep-Alive: timeout=1
Expect: 100-continue

* STATE: DO => DID handle 0x55d511b6e978; line 2255
* STATE: DID => PERFORMING handle 0x55d511b6e978; line 2373
* cr_exp100_read, AWAITING_CONTINUE, not expired
* HTTP 1.1 or later with persistent connection
< HTTP/1.1 100 Continue
< 
* cr_exp100_read, pass through
2024-05-22 15:33:52.518 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:52.617 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:52.718 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:52.819 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:52.920 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.021 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.121 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.221 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.322 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.423 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.524 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.625 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.726 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.827 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:53.928 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.029 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.130 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.231 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.332 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.433 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.534 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.635 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.736 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.836 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:54.937 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.038 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.139 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.240 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.341 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.442 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.543 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.643 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.744 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.845 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:55.946 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.047 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.148 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.249 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.350 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.451 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.551 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.651 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.752 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.853 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:56.954 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.055 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.156 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.257 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.358 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.459 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.560 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.661 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.761 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.861 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:57.961 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.061 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.162 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.263 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.364 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.465 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.566 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.667 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.768 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.869 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:58.970 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.071 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.171 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.272 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.373 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.473 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
5/22 15:33:52 INFO: 7e3bd17c66e004e8 [logterm: 1, index: 3] sent MsgVote request to fa252746f913ab7f at term 2�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d [term: 1] received a MsgVote message with higher term from 7e3bd17c66e004e8 [term: 2]�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d became follower at term 2�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: 8e9e05c52164694d [logterm: 1, index: 3, vote: 0] cast MsgVote for 7e3bd17c66e004e8 [logterm: 1, index: 3] at term 2�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f [term: 1] received a MsgVote message with higher term from 7e3bd17c66e004e8 [term: 2]�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f became follower at term 2�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: fa252746f913ab7f [logterm: 1, index: 3, vote: 0] cast MsgVote for 7e3bd17c66e004e8 [logterm: 1, index: 3] at term 2�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 received MsgVoteResp from 8e9e05c52164694d at term 2�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 has received 2 MsgVoteResp votes and 0 vote rejections�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: 7e3bd17c66e004e8 became leader at term 2�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:33:52 INFO: raft.node: 7e3bd17c66e004e8 elected leader 7e3bd17c66e004e8 at term 2�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:33:52 INFO: raft.node: fa252746f913ab7f elected leader 7e3bd17c66e004e8 at term 2�[0m
�[35metcd #1 | �[0mraft2024/05/22 15:33:52 INFO: raft.node: 8e9e05c52164694d elected leader 7e3bd17c66e004e8 at term 2�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.508322 I | embed: ready to serve client requests�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.508407 I | etcdserver: published {Name:test2 ClientURLs:[http://localhost:2381]} to cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.508541 I | embed: ready to serve client requests�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.508681 I | etcdserver: published {Name:test3 ClientURLs:[http://localhost:2383]} to cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.508780 I | embed: ready to serve client requests�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.508882 I | etcdserver: published {Name:test1 ClientURLs:[http://localhost:2379]} to cluster de368b0e4577b2d6�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.509456 I | etcdserver: setting up the initial cluster version to 3.4�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.510359 N | embed: serving insecure client requests on 127.0.0.1:2383, this is strongly discouraged!�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.510639 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.510981 N | etcdserver/membership: set the initial cluster version to 3.4�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.511091 I | etcdserver/api: enabled capabilities for version 3.4�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.510465 N | embed: serving insecure client requests on 127.0.0.1:2381, this is strongly discouraged!�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.510535 N | etcdserver/membership: set the initial cluster version to 3.4�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:52.510767 I | etcdserver/api: enabled capabilities for version 3.4�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.510980 N | etcdserver/membership: set the initial cluster version to 3.4�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:52.511064 I | etcdserver/api: enabled capabilities for version 3.4�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.518022 N | pkg/osutil: received terminated signal, shutting down...�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.518086 W | embed: stopping insecure grpc server due to error: accept tcp 127.0.0.1:2379: use of closed network connection�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:52.518156 W | embed: stopped insecure grpc server due to error: accept tcp 127.0.0.1:2379: use of closed network connection�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.518538 I | etcdserver: skipped leadership transfer for stopping non-leader member�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.518665 I | rafthttp: stopping peer fa252746f913ab7f...�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.518907 I | rafthttp: closed the TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.518916 I | rafthttp: stopped streaming with peer fa252746f913ab7f (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519095 I | rafthttp: closed the TCP streaming connection with peer fa252746f913ab7f (stream Message writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519106 I | rafthttp: stopped streaming with peer fa252746f913ab7f (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519146 I | rafthttp: stopped HTTP pipelining with peer fa252746f913ab7f�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519235 W | rafthttp: lost the TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519246 E | rafthttp: failed to read fa252746f913ab7f on stream MsgApp v2 (context canceled)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519253 I | rafthttp: peer fa252746f913ab7f became inactive (message send to peer failed)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519286 I | rafthttp: stopped streaming with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519346 W | rafthttp: lost the TCP streaming connection with peer fa252746f913ab7f (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519357 I | rafthttp: stopped streaming with peer fa252746f913ab7f (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519365 I | rafthttp: stopped peer fa252746f913ab7f�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.519374 I | rafthttp: stopping peer 7e3bd17c66e004e8...�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520328 I | rafthttp: closed the TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520346 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520809 I | rafthttp: closed the TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520819 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520843 I | rafthttp: stopped HTTP pipelining with peer 7e3bd17c66e004e8�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520928 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520943 E | rafthttp: failed to read 7e3bd17c66e004e8 on stream MsgApp v2 (context canceled)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520953 I | rafthttp: peer 7e3bd17c66e004e8 became inactive (message send to peer failed)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.520980 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.521044 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.521057 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.521081 I | rafthttp: stopped peer 7e3bd17c66e004e8�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.521193 E | rafthttp: failed to find member 7e3bd17c66e004e8 in cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.521205 E | rafthttp: failed to find member fa252746f913ab7f in cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.521225 E | rafthttp: failed to find member fa252746f913ab7f in cluster de368b0e4577b2d6�[0m
�[35metcd #1 | �[0m2024-05-22 15:33:59.522008 E | rafthttp: failed to find member 7e3bd17c66e004e8 in cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:59.519019 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream 2024-05-22 15:33:59.574 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.675 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.776 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.877 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:33:59.978 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:34:00.079 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:34:00.180 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:34:00.281 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:34:00.382 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:34:00.482 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
* nread <= 0, server closed connection, bailing
2024-05-22 15:34:00.583 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:59.519236 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:59.521976 E | rafthttp: failed to dial 8e9e05c52164694d on stream Message (peer 8e9e05c52164694d failed to find local node fa252746f913ab7f)�[0m
�[36metcd #3 | �[0m2024-05-22 15:33:59.521990 I | rafthttp: peer 8e9e05c52164694d became inactive (message send to peer failed)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:59.519643 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:59.521357 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:59.521606 E | rafthttp: failed to dial 8e9e05c52164694d on stream MsgApp v2 (peer 8e9e05c52164694d failed to find local node 7e3bd17c66e004e8)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:59.521622 I | rafthttp: peer 8e9e05c52164694d became inactive (message send to peer failed)�[0m
�[34metcd #2 | �[0m2024-05-22 15:33:59.903597 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:00.512125 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:00.512156 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:04.514128 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:04.514154 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:05.643698 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:05.652120 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:05.652206 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:08.515638 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:08.515659 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:12.516419 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:12.516441 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:16.517349 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:16.517387 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.518420 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version":2024-05-22 15:34:20.602 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 startup
 dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.518443 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[32metcd #1 | �[0mWARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.�[0m
�[32metcd #1 | �[0m	A future release of golang/protobuf will delete this package,�[0m
�[32metcd #1 | �[0m	which has long been excluded from the compatibility promise.�[0m
�[32metcd #1 | �[0m�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.614973 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://localhost:2379�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615007 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/tmp/Jme3bU/test1�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615021 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://localhost:2380�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615025 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=test1=http://localhost:2380,test2=http://localhost:2382,test3=http://localhost:2384�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615028 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615032 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://localhost:2379�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615035 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://localhost:2380�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615042 I | pkg/flags: recognized and used environment variable ETCD_NAME=test1�[0m
�[32metcd #1 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615070 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615075 I | etcdmain: etcd Version: 3.4.32�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615077 I | etcdmain: Git SHA: 9ac4e7ef0�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615079 I | etcdmain: Go Version: go1.21.9�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615081 I | etcdmain: Go OS/Arch: linux/amd64�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615083 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615127 N | etcdmain: the server is already initialized as member before, starting as etcd member...�[0m
�[32metcd #1 | �[0m[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615134 W | embed: Running http and grpc server on single port. This is not recommended for production.�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615452 I | embed: name = test1�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615457 I | embed: data dir = /tmp/Jme3bU/test1�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615459 I | embed: member dir = /tmp/Jme3bU/test1/member�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615461 I | embed: heartbeat = 100ms�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615463 I | embed: election = 1000ms�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615465 I | embed: snapshot count = 100000�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615470 I | embed: advertise client URLs = http://localhost:2379�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615473 I | embed: initial advertise peer URLs = http://localhost:2380�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.615476 I | embed: initial cluster = �[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.619154 I | etcdserver: restarting member 8e9e05c52164694d in cluster de368b0e4577b2d6 at commit index 8�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: 8e9e05c52164694d switched to configuration voters=()�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: 8e9e05c52164694d became follower at term 2�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 8, applied: 0, lastindex: 8, lastterm: 2]�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.629582 W | auth: simple token is not cryptographically signed�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631402 I | etcdserver: starting server... [version: 3.4.32, cluster version: to_be_decided]�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631463 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test1/member/snap, suffix: snap.db, max: 5, interval: 30s�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631471 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test1/member/snap, suffix: snap, max: 5, interval: 30s�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631476 I | pkg/fileutil: started to purge file, dir: /tmp/Jme3bU/test1/member/wal, suffix: wal, max: 5, interval: 30s�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631701 I | etcdserver/membership: added member 7e3bd17c66e004e8 [http://localhost:2382] to cluster de368b0e4577b2d6�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631711 I | rafthttp: starting peer 7e3bd17c66e004e8...�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631728 I | rafthttp: started HTTP pipelining with peer 7e3bd17c66e004e8�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.631986 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632028 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632533 I | rafthttp: started peer 7e3bd17c66e004e8�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632547 I | rafthttp: added peer 7e3bd17c66e004e8�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632551 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632606 I | rafthttp: started streaming with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632615 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster de368b0e4577b2d6�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: 8e9e05c52164694d switched to configuration voters=(9096094204636955880 10276657743932975437 18024856269400288127)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632654 I | etcdserver/membership: added member fa252746f913ab7f [http://localhost:2384] to cluster de368b0e4577b2d6�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632658 I | rafthttp: starting peer fa252746f913ab7f...�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632667 I | rafthttp: started HTTP pipelining with peer fa252746f913ab7f�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.632806 I | rafthttp: started streaming with peer fa252746f913ab7f (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633186 I | rafthttp: started streaming with peer fa252746f913ab7f (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633602 I | rafthttp: peer 7e3bd17c66e004e8 became active�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633614 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633626 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633916 I | rafthttp: started peer fa252746f913ab7f�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633940 I | rafthttp: added peer fa252746f913ab7f�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633943 I | rafthttp: started streaming with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.633995 I | rafthttp: started streaming with peer fa252746f913ab7f (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634046 N | etcdserver/membership: set the initial cluster ver2024-05-22 15:34:20.703 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 startup
sion to 3.4�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634078 I | etcdserver/api: enabled capabilities for version 3.4�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634135 I | embed: listening for peers on 127.0.0.1:2380�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634292 I | rafthttp: peer fa252746f913ab7f became active�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634305 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream Message writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634348 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634364 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634371 I | rafthttp: established a TCP streaming connection with peer fa252746f913ab7f (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634400 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.634405 I | rafthttp: established a TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.633365 I | rafthttp: peer 8e9e05c52164694d became active�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.633375 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.633639 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.634384 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.634408 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.634270 I | rafthttp: peer 8e9e05c52164694d became active�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.634279 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.634294 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.634309 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.634328 I | rafthttp: established a TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.682165 I | etcdserver: 8e9e05c52164694d initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)�[0m
�[32metcd #1 | �[0mraft2024/05/22 15:34:20 INFO: raft.node: 8e9e05c52164694d elected leader 7e3bd17c66e004e8 at term 2�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.704955 I | embed: ready to serve client requests�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.705093 I | etcdserver: published {Name:test1 ClientURLs:[http://localhost:2379]} to cluster de368b0e4577b2d6�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.706399 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!�[0m
F2024-05-22 15:34:20.711 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
�[32metcd #1 | �[0m2024-05-22 15:34:20.710351 N | pkg/osutil: received terminated signal, shutting down...�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.710420 W | embed: stopping insecure grpc server due to error: accept tcp 127.0.0.1:2379: use of closed network connection�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.710492 W | embed: stopped insecure grpc server due to error: accept tcp 127.0.0.1:2379: use of closed network connection�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.710509 I | etcdserver: skipped leadership transfer for stopping non-leader member�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.710578 I | rafthttp: stopping peer 7e3bd17c66e004e8...�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.710699 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711033 I | rafthttp: closed the TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711036 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711166 I | rafthttp: closed the TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711169 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711181 I | rafthttp: stopped HTTP pipelining with peer 7e3bd17c66e004e8�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711235 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711240 E | rafthttp: failed to read 7e3bd17c66e004e8 on stream MsgApp v2 (context canceled)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711249 I | rafthttp: peer 7e3bd17c66e004e8 became inactive (message send to peer failed)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711264 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711297 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711301 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711304 I | rafthttp: stopped peer 7e3bd17c66e004e8�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711307 I | rafthttp: stopping peer fa252746f913ab7f...�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.711140 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711833 I | rafthttp: closed the TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711837 I | rafthttp: stopped streaming with peer fa252746f913ab7f (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711895 I | rafthttp: closed the TCP streaming connection with peer fa252746f913ab7f (stream Message writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711897 I | rafthttp: stopped streaming with peer fa252746f913ab7f (writer)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711905 I | rafthttp: stopped HTTP pipelining with peer fa252746f913ab7f�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711946 W | rafthttp: lost the TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711949 E | rafthttp: failed to read fa252746f913ab7f on stream MsgApp v2 (context canceled)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711950 I | rafthttp: peer fa252746f913ab7f became inactive (message send to peer failed)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711957 I | rafthttp: stopped streaming with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711970 W | rafthttp: lost the TCP streaming connection with peer fa252746f913ab7f (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.711972 I | rafthttp: stopped streaming with peer fa25272024-05-22 15:34:20.809 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#1 teardown
2024-05-22 15:34:20.811 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
46f913ab7f (stream Message reader)�[0m
�[32metcd #1 | �[0m2024-05-22 15:34:20.712049 I | rafthttp: stopped peer fa252746f913ab7f�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.711881 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.711938 W | rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.719239 E | rafthttp: failed to dial 8e9e05c52164694d on stream MsgApp v2 (dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.719247 I | rafthttp: peer 8e9e05c52164694d became inactive (message send to peer failed)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.720514 E | rafthttp: failed to dial 8e9e05c52164694d on stream MsgApp v2 (dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.720520 I | rafthttp: peer 8e9e05c52164694d became inactive (message send to peer failed)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.810577 N | pkg/osutil: received terminated signal, shutting down...�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.810683 W | embed: stopping insecure grpc server due to error: accept tcp 127.0.0.1:2381: use of closed network connection�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.810839 W | embed: stopped insecure grpc server due to error: accept tcp 127.0.0.1:2381: use of closed network connection�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.810911 I | etcdserver: 7e3bd17c66e004e8 starts leadership transfer from 7e3bd17c66e004e8 to fa252746f913ab7f�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: 7e3bd17c66e004e8 [term 2] starts to transfer leadership to fa252746f913ab7f�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: 7e3bd17c66e004e8 sends MsgTimeoutNow to fa252746f913ab7f immediately as fa252746f913ab7f already has up-to-date log�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f [term 2] received MsgTimeoutNow from 7e3bd17c66e004e8 and starts an election to get leadership.�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f became candidate at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f received MsgVoteResp from fa252746f913ab7f at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f [logterm: 2, index: 9] sent MsgVote request to 7e3bd17c66e004e8 at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f [logterm: 2, index: 9] sent MsgVote request to 8e9e05c52164694d at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: raft.node: fa252746f913ab7f lost leader 7e3bd17c66e004e8 at term 3�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: 7e3bd17c66e004e8 [term: 2] received a MsgVote message with higher term from fa252746f913ab7f [term: 3]�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: 7e3bd17c66e004e8 became follower at term 3�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: 7e3bd17c66e004e8 [logterm: 2, index: 9, vote: 0] cast MsgVote for fa252746f913ab7f [logterm: 2, index: 9] at term 3�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: raft.node: 7e3bd17c66e004e8 lost leader 7e3bd17c66e004e8 at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f received MsgVoteResp from 7e3bd17c66e004e8 at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f has received 2 MsgVoteResp votes and 0 vote rejections�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: fa252746f913ab7f became leader at term 3�[0m
�[36metcd #3 | �[0mraft2024/05/22 15:34:20 INFO: raft.node: fa252746f913ab7f elected leader fa252746f913ab7f at term 3�[0m
�[34metcd #2 | �[0mraft2024/05/22 15:34:20 INFO: raft.node: 7e3bd17c66e004e8 elected leader fa252746f913ab7f at term 3�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.813665 W | etcdserver: failed to reach the peerURL(http://localhost:2380) of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[36metc2024-05-22 15:34:20.910 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.010 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
d #3 | �[0m2024-05-22 15:34:20.813674 W | etcdserver: cannot get the version of member 8e9e05c52164694d (Get "http://localhost:2380/version": dial tcp 127.0.0.1:2380: connect: connection refused)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911126 I | etcdserver: 7e3bd17c66e004e8 finished leadership transfer from 7e3bd17c66e004e8 to fa252746f913ab7f (took 100.230596ms)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911197 I | rafthttp: stopping peer 8e9e05c52164694d...�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911433 I | rafthttp: closed the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911442 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911556 I | rafthttp: closed the TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911566 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911591 I | rafthttp: stopped HTTP pipelining with peer 8e9e05c52164694d�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911608 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911617 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911622 I | rafthttp: stopped peer 8e9e05c52164694d�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911627 I | rafthttp: stopping peer fa252746f913ab7f...�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911747 I | rafthttp: closed the TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911756 I | rafthttp: stopped streaming with peer fa252746f913ab7f (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911916 I | rafthttp: closed the TCP streaming connection with peer fa252746f913ab7f (stream Message writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911922 I | rafthttp: stopped streaming with peer fa252746f913ab7f (writer)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.911938 I | rafthttp: stopped HTTP pipelining with peer fa252746f913ab7f�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912024 W | rafthttp: lost the TCP streaming connection with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912037 E | rafthttp: failed to read fa252746f913ab7f on stream MsgApp v2 (context canceled)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912045 I | rafthttp: peer fa252746f913ab7f became inactive (message send to peer failed)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912087 I | rafthttp: stopped streaming with peer fa252746f913ab7f (stream MsgApp v2 reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912171 W | rafthttp: lost the TCP streaming connection with peer fa252746f913ab7f (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912184 I | rafthttp: stopped streaming with peer fa252746f913ab7f (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912193 I | rafthttp: stopped peer fa252746f913ab7f�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.911839 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.911970 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912506 E | rafthttp: failed to find member fa252746f913ab7f in cluster de368b0e4577b2d6�[0m
�[34metcd #2 | �[0m2024-05-22 15:34:20.912535 E | rafthttp: failed to find member fa252746f913ab7f in cluster de368b0e4577b2d6�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.912726 E | rafthttp: failed to dial 7e3bd17c66e004e8 on stream Message (peer 7e3bd17c66e004e8 failed to find local node fa252746f913ab7f)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:20.912744 I | rafthttp: peer 7e3bd17c66e004e8 became inactive (message send to peer failed)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:21.103884 W | 2024-05-22 15:34:21.110 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.211 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.312 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.413 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.514 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.615 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.716 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.817 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:21.917 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:22.018 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#2 teardown
2024-05-22 15:34:22.020 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#3 teardown
2024-05-22 15:34:22.119 [1488188] main/104/luatest/test.utils V> etcd_test | waiting for etcd#3 teardown
rafthttp: lost the TCP streaming connection with peer 8e9e05c52164694d (stream Message writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:21.304028 W | rafthttp: lost the TCP streaming connection with peer 7e3bd17c66e004e8 (stream Message writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.019677 N | pkg/osutil: received terminated signal, shutting down...�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.019802 W | embed: stopping insecure grpc server due to error: accept tcp 127.0.0.1:2383: use of closed network connection�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.019926 W | embed: stopped insecure grpc server due to error: accept tcp 127.0.0.1:2383: use of closed network connection�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.019962 W | etcdserver: fa252746f913ab7f failed to transfer leadership (etcdserver: unhealthy cluster)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.020037 I | rafthttp: stopping peer 7e3bd17c66e004e8...�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021777 I | rafthttp: closed the TCP streaming connection with peer 7e3bd17c66e004e8 (stream MsgApp v2 writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021791 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021806 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021822 I | rafthttp: stopped HTTP pipelining with peer 7e3bd17c66e004e8�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021861 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021875 I | rafthttp: stopped streaming with peer 7e3bd17c66e004e8 (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021881 I | rafthttp: stopped peer 7e3bd17c66e004e8�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.021886 I | rafthttp: stopping peer 8e9e05c52164694d...�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023400 I | rafthttp: closed the TCP streaming connection with peer 8e9e05c52164694d (stream MsgApp v2 writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023408 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023421 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (writer)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023436 I | rafthttp: stopped HTTP pipelining with peer 8e9e05c52164694d�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023444 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (stream MsgApp v2 reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023456 I | rafthttp: stopped streaming with peer 8e9e05c52164694d (stream Message reader)�[0m
�[36metcd #3 | �[0m2024-05-22 15:34:22.023463 I | rafthttp: stopped peer 8e9e05c52164694d�[0m

�[1m
Failed tests:
-------------
�[0m
1) integration.client.http.test_stream_version_specific_behavior_on_disconnect�[1m�[31m
...es/MRG/etcd-client/test/integration/client/http_test.lua:130: expected: a value evaluating to true, actual: false�[0m
stack traceback:
	...es/MRG/etcd-client/test/integration/client/http_test.lua:130: in function 'integration.client.http.test_stream_version_specific_behavior_on_disconnect'
	...
	[C]: in function 'xpcall'
Ran 1 tests in 31.854 seconds, �[1m�[32m0 succeeded�[0m, �[1m�[31m1 failed�[0m

�[1mFailed tests:
�[1m�[31m
integration.client.http.test_stream_version_specific_behavior_on_disconnect
�[0m

I'm not sure that you are interested in a reproducer because currently it is complicated: it involves tarantool (curl is builtin), etcd, proprietary Lua module etcd-client and our own test framework luatest.

@icing
Copy link
Contributor

icing commented May 22, 2024

Thanks, that log might give me an idea how to extend our test scenarios.

icing added a commit to icing/curl that referenced this issue May 22, 2024
adding edge cases, possible workaround
refs curl#13740

- add 2 variations on test_07_42 which PAUSEs uploads
  and response connections terminating either right away
  or after the 100-continue response
- when detecting the connection being closed in transfer.c
  readwrite_data(), clear ALL send bits in data->req.keepon.
  It no longer makes send to wait for a KEEP_SEND_PAUSE or HOLD.
- in the protocol client writer add the check for incomplete
  response bodies. When an EOS is seen and the length is known,
  check that and fail if bytes are missing.
@icing
Copy link
Contributor

icing commented May 22, 2024

I changed my test client and added more server edge behaviour in tests. I was able to reproduce a condition where curl would not terminate a transfer after the connection had been closed. This seems to match the output from your log files.

It would be great to know if #13750 works for you as well.

@ligurio
Copy link
Contributor Author

ligurio commented May 22, 2024

It would be great to know if #13750 works for you as well.

@icing Stefan, it fixes the problem. Many thanks!

@bagder bagder closed this as completed in 30de937 May 23, 2024
ligurio added a commit to tarantool/tarantool that referenced this issue Jun 18, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
commit 30de937bda0f ("transfer: conn close on paused upload")
fixes a regression catched on the previous bump [2]. The new
version brings a number of functional fixes, and updates CMake
module for building curl library.

Changes in CMake module:

- TBD

Changelog entry has been removed because duplicate entries about
bumps confuses end users.

Closes #9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_CHANGELOG=libcurl submodule bump
NO_TEST=libcurl submodule bump
ligurio added a commit to ligurio/tarantool that referenced this issue Jun 18, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes tarantool#9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump
Buristan pushed a commit to tarantool/tarantool that referenced this issue Jun 25, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes #9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump
Buristan pushed a commit to Buristan/tarantool that referenced this issue Jun 25, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes tarantool#9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

(cherry picked from commit 7192bf6)
Buristan pushed a commit to Buristan/tarantool that referenced this issue Jun 25, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes tarantool#9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

(cherry picked from commit 7192bf6)
Totktonada pushed a commit to tarantool/tarantool that referenced this issue Jun 25, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes #9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

(cherry picked from commit 7192bf6)
Totktonada pushed a commit to tarantool/tarantool that referenced this issue Jun 25, 2024
The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes #9612

1. https://curl.se/changes.html#8_8_0
2. curl/curl#13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

(cherry picked from commit 7192bf6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

3 participants