get requests intermittently timing out on v3.1.0-rc.1

[kragniz]

I've got a few CI jobs for python-etcd3, some of which run against etcd 3.1.0-rc.1. I've observed that the test for getting values out of etcd is flaky against v3.1.0-rc.1.

It's not really a problem for me, but there's a discrepancy between the reliability of this and v3.0.13 (the other version CI runs against).

Here's an example job that failed:
https://travis-ci.org/kragniz/python-etcd3/jobs/189153848

[gyuho]

Please see https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-does-the-etcd-warning-apply-entries-took-too-long-mean

[heyitsanthony]

@gyuho it was working fine with 3.0, though. It's getting request timeout errors, not only warnings-- this might be something different.

[xiang90]

@kragniz Can you easily reproduce this out of your test suite with a simple go or python program that I can compile/run on my local machine?

[kragniz]

Yeah, I'll create a little python script to try and reproduce.

[kragniz]

Here's the equivilant code that the test is running:

Save this somewhere like timeout_reproducer.py

import json
import subprocess
import os

from hypothesis import given
from hypothesis.strategies import characters

import etcd3

os.environ['ETCDCTL_API'] = '3'

def etcdctl(*args):
    args = ['etcdctl', '-w', 'json'] + list(args)
    print(" ".join(args))
    output = subprocess.check_output(args)
    return json.loads(output.decode('utf-8'))

@given(characters(blacklist_categories=['Cs', 'Cc']))
def test_get_key(string):
    etcd = etcd3.client()
    etcdctl('put', '/doot/a_key', string)
    returned, _ = etcd.get('/doot/a_key')
    assert returned == string.encode('utf-8')
    etcdctl('del', '--prefix', '')

if __name__ == '__main__':
    test_get_key()

run it with:

$ virtualenv -p python3 /tmp/etcd3env
$ source /tmp/etcd3env/bin/activate
$ pip install etcd3 hypothesis
$ python timeout_reproducer.py

[fanminshi]

@kragniz which etcdctl version are you running? I run your script with etcd(darwin) binaries downloaded from https://github.com/coreos/etcd/releases/tag/v3.1.0-rc.1. I didn't see any errors. does your script produce error everytime?

[kragniz]

Thanks!