Skip to content
This repository has been archived by the owner on Oct 7, 2023. It is now read-only.

zetcd errors with etcd master branch #9

Closed
gyuho opened this issue Dec 21, 2016 · 1 comment
Closed

zetcd errors with etcd master branch #9

gyuho opened this issue Dec 21, 2016 · 1 comment

Comments

@gyuho
Copy link
Contributor

gyuho commented Dec 21, 2016

I have simple local etcd server + zetcd process and tried to send requests to zetcd but almost all requests error...

Here's how to reproduce

# build etcd
./install-zetcd.sh
./start-server-zetcd.sh
./test-zk.sh

https://github.com/gyuho/zookeeper-tests
https://github.com/gyuho/zookeeper-tests/blob/master/start-server-zetcd.sh
https://github.com/gyuho/zookeeper-tests/blob/master/test-zk.sh

@heyitsanthony
Copy link
Contributor

heyitsanthony commented Dec 28, 2016

Running the benchmark, I see a lot of timeouts:

2016/12/28 00:24:57 Recv loop terminated: err=read tcp 127.0.0.1:34170->127.0.0.1:2181: i/o timeout
2016/12/28 00:24:57 Send loop terminated: err=<nil>

The zk logs show two things zetcd doesn't implement:

Renegotiating timeouts; benchmark requests 1s, zk gives 4s:

2016-12-28 08:13:09,594 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@645] - Established session 0x159447d61890033 with negotiated timeout 4000 for client /172.17.0.1:51658

Limiting max connections to 60 (but zetcd accepts all 100):

2016-12-28 08:13:10,491 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@854] - Connection request from old client /172.17.0.1:51748; will be dropped if server is in r-o mode
2016-12-28 08:13:10,491 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900] - Client attempting to establish new session at /172.17.0.1:51748
2016-12-28 08:13:10,491 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@188] - Too many connections from /172.17.0.1 - max is 60

If I change the session timeout to 4s in zetcd like zk does, the time-outs all go away.

Potential fixes would be to support timeout renegotiation (probably worthwhile) and connection limiting (seems silly) like zk.

heyitsanthony pushed a commit to heyitsanthony/zetcd that referenced this issue Dec 28, 2016
Zookeeper will give timeouts greater than the requested timeout, leading
to zetcd connections timing out for respecting the request timeout.
Instead, lengthen the timeout if there are too many sessions and the
requested timeout is too short.

Fixes etcd-io#9
heyitsanthony pushed a commit to heyitsanthony/zetcd that referenced this issue Dec 28, 2016
Heavy contention on /zk/cver keys from RMW accesses causes STM to
retry too often. Instead, blindly write to cver keys and use the
etcd version for the znode stats.

Fixes etcd-io#9
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants