Using locust to query Cassandra

[cscetbon]

Hi,

I'm trying to run queries on Cassandra using locust but get blocking issues when locust tries to connect to the cluster. It works well when I do it without locust. Any reason why locust would make it fail ? Any workaround ?

https://pastebin.com/QxuiEpu6
https://pastebin.com/BJFkdE2q

[heyman]

What does "blocking issues" mean?

The only thing I can think of on top of my head is if gevent's monkey patching somehow messes with the Cassanda drivers (though this seems to indicate that the cassandra lib should work fine with gevent: datastax/python-driver#46)

What happens if you add the following to the top of the non-locust python file?

from gevent import monkey
monkey.patch_all(thread=False)

[cscetbon]

@heyman, by blocking I mean the locust web interface can't refresh for instance. Adding your snippet doesn't help.

[heyman]

@cscetbon The snippet was not intended to fix the problem. I wanted you to add it to the top of the non locust version that's currently working, to see if that would cause the same problem as you have when using the Cassandra client with Locust. Then we would know that the problem is related to gevent's monkey patching.

Does the web interface load at all? Are there other issues (since you write "for instance")? For me to be able to understand the issue, you really need to be as detailed and specific as possible.

[cscetbon]

@HeyHugo I don't have any non locust file. Are you talking about the python module I use ? I have only one locust file, the one I sent you.

The interface loads, it's when I start the load with 4 users in my case and Hatch of 1 that it starts blocking. Reloading the app doesn't work anymore, I think if I wait I'll get a timeout error instead

[heyman]

@cscetbon I meant this one: https://pastebin.com/QxuiEpu6

To me it sounds as if this could be related to issues with the monkey patching not being used, e.g. if the Cassandra drivers doesn't use the builtin python socket lib for IO.

Do you see any exceptions/other output from Locust? What happens if you add some print statements to your code. Where does it block?

[cscetbon]

okay, so adding it to the non locust file doesn't have any impact.

Locust blocks on self.session = cluster.connect("demo")

The only output messages I get are :

[2017-04-03 10:44:14,284] ELIA.local/INFO/cassandra.policies: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
[2017-04-03 10:45:44,482] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4454472272) to 127.0.0.1
[2017-04-03 10:47:14,530] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4454685200) to 127.0.0.1
[2017-04-03 10:48:44,558] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4454686416) to 127.0.0.1
[2017-04-03 10:50:14,587] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4456207056) to 127.0.0.1

So I suppose that something in locust prevents it to connect.

[jlapier]

I'm having the same problem. Using the same code on the same host in a python script outside of locust, it connects fine. Using it inside locust gives me the exact same errors @cscetbon is seeing on the connect call. I can't figure out what's different about using the cluster client inside locust.

[manaspaldhe12]

Same issue here :|

[grucin]

I had some problems with connection to other database. This commit gemius@a705ba4 fixed my issue, but it does not guarantee that it is the same case.

[rslota]

Hi,

Same issue here. With locust any cassandra operation that requires connection to DBMS hangs the whole interpreter (sometimes it needs to be killed with -9). Without locust it works just fine. Also, adding the code snippet from @heyman to non-locust version doesn't affect the cassandra driver.

[rslota]

The patch from @grucin doesn't work for me.

[rslota]

Also, running:

from gevent import monkey
monkey.patch_all(thread=True)

does fix the issue. Why Locust doesn't patch threads? What are the implications of overriding this?

[cscetbon]

@rslota when I use it tests are executed but statistics are not updated. Is it better for you ?

[rslota]

@cscetbon ,

Yeah, when I patch at least threads (but currently in my code I just patch everything - monkey.patch_all()) everything works great. No stability issues and all statistics seem to be just fine. But... I don't load test HTTP, I have my own "drivers" to load several DBMS, so I fire success and failure events in locust manually (with my own time measurements), so maybe that's the reason I don't have issue you are seeing.

[heyman]

I really don't remember if there was a good reason for not patching threads (I should have left a comment). It was made in the very early days of the development, and it's very possible that we just simply didn't use any threads at the time. Therefore I suggest that we change it to patch everything since people have reported that it seems to work fine.

[heyman]

I've now changed so that we also patch threading. It's released in the latest version on PyPI (0.8a4).