Weird issues when using logbook 0.9.1 + cassandra-driver 2.5.1

[winks]

Hi,
I just spent a lot of time trying to track down why my cassandra connection would drop on connect.
I fixed it by reordering the imports.
In this reproduce case it's only RotatingFileHandler, but I think it also breaks with Logger and StreamHandler.

requirements.txt
Logbook==0.9.1
cassandra-driver==2.5.1

# bug.py
import os.path

from logbook import RotatingFileHandler #, Logger, StreamHandler

# Import cassandra before logbook or break it all
from cassandra.cluster import Cluster
from cassandra import query

LOG_PATH = '/tmp'


format_string = (
    '[{record.time:%Y-%m-%d %H:%M:%S}]'
    '[{record.level_name}][{record.channel}]: {record.message}'
)

stream_log_handler = RotatingFileHandler(
    os.path.join(LOG_PATH, 'stream.log'),
    format_string=format_string,
    filter=lambda r, _: 's3_stream' in r.channel,
    backup_count=2
)
stream_log_handler.push_application()

cassandra_hosts = ['172.20.70.101']
keyspace = 'system'

cluster = Cluster(cassandra_hosts)
session = cluster.connect(keyspace)
session.row_factory = query.dict_factory

print session.execute('SELECT * FROM local;')

"""
Traceback (most recent call last):
  File "bug.py", line 29, in <module>
    session = cluster.connect(keyspace)
  File "/home/xxx/.virtualenvs/xxx/local/lib/python2.7/site-packages/cassandra/cluster.py", line 755, in connect
    self.control_connection.connect()
  File "/home/xxx/.virtualenvs/xxx/local/lib/python2.7/site-packages/cassandra/cluster.py", line 1868, in connect
    self._set_new_connection(self._reconnect_internal())
  File "/home/xxx/.virtualenvs/xxx/local/lib/python2.7/site-packages/cassandra/cluster.py", line 1903, in _reconnect_internal
    raise NoHostAvailable("Unable to connect to any servers", errors)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'172.20.70.101': OperationTimedOut('errors=None, last_host=None',)})
"""

# bug_gone.py
import os.path

# Import cassandra before logbook or break it all
from cassandra.cluster import Cluster
from cassandra import query

from logbook import RotatingFileHandler #, Logger, StreamHandler

LOG_PATH = '/tmp'


format_string = (
    '[{record.time:%Y-%m-%d %H:%M:%S}]'
    '[{record.level_name}][{record.channel}]: {record.message}'
)

stream_log_handler = RotatingFileHandler(
    os.path.join(LOG_PATH, 'stream.log'),
    format_string=format_string,
    filter=lambda r, _: 's3_stream' in r.channel,
    backup_count=2
)
stream_log_handler.push_application()

cassandra_hosts = ['172.20.70.101']
keyspace = 'system'

cluster = Cluster(cassandra_hosts)
session = cluster.connect(keyspace)
session.row_factory = query.dict_factory

print session.execute('SELECT * FROM local;')

"""
[{u'tokens': sortedset( ... some valid output
"""

[winks]

For reference: https://datastax-oss.atlassian.net/browse/PYTHON-355

[vmalloc]

@winks I'm not sure what can be done here, assuming this is indeed a race for who initialises gevent, we can't actually know what other libraries are loaded in addition to Logbook in all possible use cases...

[winks]

@vmalloc Yeah, that sounds reasonable. But for people who just use logbook and not dig into it, this dependency is probably as unknown as it was to me.

I guess it won't hurt to add a paragraph to the docs with a caveat that something could happen, e.g. with the cassandra-driver that also uses gevent?

[vmalloc]

Ok. Care to PR such a suggestion?

[winks]

Hey,
I had a deeper look at the documentation but I couldn't find any part that really hits what I searched for. "Troubleshooting" or "Caveats".

It is quite an edge case, I have to agree on that. I wouldn't have added more than

Please note that Logbook uses gevent and in rare cases if you're using it together with other libraries that use gevent, there have been rare race conditions which could sometimes be fixed by reordering the import statements. (Example: logbook 0.9.2 and cassandra-driver 2.5.1)

Please feel free to use this, adapt this, or not do anything at all. I'd be happy even if it helps one poor soul who searches this bugtracker or finds it in the docs :) And if I understood it correctly, it might even be fixed in cassandra-driver 2.6.0.

[brainstorm]

Wow, @winks great forensic work there... did it get fixed in cassandra-driver 2.6.0? Works for you now?

[brainstorm]

@winks, chime in if it ever get solved in newer versions, thanks for the issue anyhow! Closing.