Easily able to cause select to timeout

[harlowja]

I'm not currently sure the underlying issue but I am adding https://github.com/python-zk/kazoo/pull/249/files to kazoo and using 0.15.2 it easily times out continuously in its select reader/writer, https://github.com/python-zk/kazoo/blob/master/kazoo/protocol/connection.py#L219

When I switch to 0.14 (or 0.15.0) the problem goes away (no timeouts, nor select issues occur), was there a change that might have caused this?

Example way to run:

Checkout that branch,

Setup ZOOKEEPER_CLASSPATH so that kazoo can use a local zookeeper.

Then run:

test=TestEventletSemaphore
nosetests kazoo.tests.test_eventlet_handler:$test -x -s -v

Isolating down to which commit seems to work and which doesn't it appears to be one of these two that is the root of the issue...

Commit: 8f98f8b63443
Commit: da8771671468989

[harlowja]

If it matters:

Python 2.6.6 (r266:84292, Dec 19 2012, 19:54:30)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from eventlet.hubs import get_hub
get_hub()
eventlet.hubs.epolls.Hub object at 0x890dbcc

And

$ pip freeze | grep greenlet
greenlet==0.4.4

[harlowja]

From a little more investigation, the selects hub does not have this problem.

I am thinking that since kazoo does a select.select on a read socket:

https://github.com/python-zk/kazoo/blob/master/kazoo/protocol/connection.py#L524

And then if nothing is ready to read at that point it will switch into doing a potential write select loop.

https://github.com/python-zk/kazoo/blob/master/kazoo/protocol/connection.py#L290

Is the problem that it is using the same socket there (in the same greenthread) and not doing read operations in the second writing select, and this somehow doesn't work when using poll/epoll but does work using select?

[harlowja]

Seemed to have fixed it with a eventlet.sleep, so must have been starving other threads...

[temoto]

Hello, @harlowja I'm glad you have resolved the issue.
Though, it must not be necessary, eventlet.sleep(0) helps sometimes.

[harlowja]

Thanks.

python-zk/kazoo#249 is what I ended up with that worked if u are interested. It still a little odd that I had to do eventlet.sleep to make this work (so there might really still be a bug somewhere in those above commits), or it could just be that the kazoo unittest needed to yield more...

[temoto]

Thanks for sharing your code. It's not a little odd, it's a full blown bug you have discovered.

We have another problem possibly related to those commits you have mentioned in the first message. #137 So sadly I must say Eventlet 0.15.2 is unstable in this regard. We have fixed one issue at the cost of introducing more.

[harlowja]

Should I reopen this then? For more investigation?

[harlowja]

Btw, some logs that show the error (probably to kazoo specific, but might be useful).

http://paste.ubuntu.com/8426939/

[temoto]

Yes, let's have it open for motivation.