New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using locust to query Cassandra #569

Closed
cscetbon opened this Issue Mar 31, 2017 · 16 comments

Comments

Projects
None yet
6 participants
@cscetbon

cscetbon commented Mar 31, 2017

Hi,

I'm trying to run queries on Cassandra using locust but get blocking issues when locust tries to connect to the cluster. It works well when I do it without locust. Any reason why locust would make it fail ? Any workaround ?

https://pastebin.com/QxuiEpu6
https://pastebin.com/BJFkdE2q

@heyman

This comment has been minimized.

Member

heyman commented Mar 31, 2017

What does "blocking issues" mean?

The only thing I can think of on top of my head is if gevent's monkey patching somehow messes with the Cassanda drivers (though this seems to indicate that the cassandra lib should work fine with gevent: datastax/python-driver#46)

What happens if you add the following to the top of the non-locust python file?

from gevent import monkey
monkey.patch_all(thread=False)
@cscetbon

This comment has been minimized.

cscetbon commented Mar 31, 2017

@heyman, by blocking I mean the locust web interface can't refresh for instance. Adding your snippet doesn't help.

@heyman

This comment has been minimized.

Member

heyman commented Apr 3, 2017

@cscetbon The snippet was not intended to fix the problem. I wanted you to add it to the top of the non locust version that's currently working, to see if that would cause the same problem as you have when using the Cassandra client with Locust. Then we would know that the problem is related to gevent's monkey patching.

Does the web interface load at all? Are there other issues (since you write "for instance")? For me to be able to understand the issue, you really need to be as detailed and specific as possible.

@cscetbon

This comment has been minimized.

cscetbon commented Apr 3, 2017

@HeyHugo I don't have any non locust file. Are you talking about the python module I use ? I have only one locust file, the one I sent you.

The interface loads, it's when I start the load with 4 users in my case and Hatch of 1 that it starts blocking. Reloading the app doesn't work anymore, I think if I wait I'll get a timeout error instead

@heyman

This comment has been minimized.

Member

heyman commented Apr 3, 2017

@cscetbon I meant this one: https://pastebin.com/QxuiEpu6

To me it sounds as if this could be related to issues with the monkey patching not being used, e.g. if the Cassandra drivers doesn't use the builtin python socket lib for IO.

Do you see any exceptions/other output from Locust? What happens if you add some print statements to your code. Where does it block?

@cscetbon

This comment has been minimized.

cscetbon commented Apr 3, 2017

okay, so adding it to the non locust file doesn't have any impact.

Locust blocks on self.session = cluster.connect("demo")

The only output messages I get are :

[2017-04-03 10:44:14,284] ELIA.local/INFO/cassandra.policies: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
[2017-04-03 10:45:44,482] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4454472272) to 127.0.0.1
[2017-04-03 10:47:14,530] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4454685200) to 127.0.0.1
[2017-04-03 10:48:44,558] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4454686416) to 127.0.0.1
[2017-04-03 10:50:14,587] ELIA.local/WARNING/cassandra.connection: Heartbeat failed for connection (4456207056) to 127.0.0.1

So I suppose that something in locust prevents it to connect.

@jlapier

This comment has been minimized.

jlapier commented May 11, 2017

I'm having the same problem. Using the same code on the same host in a python script outside of locust, it connects fine. Using it inside locust gives me the exact same errors @cscetbon is seeing on the connect call. I can't figure out what's different about using the cluster client inside locust.

@manaspaldhe12

This comment has been minimized.

manaspaldhe12 commented May 16, 2017

Same issue here :|

@grucin

This comment has been minimized.

grucin commented May 30, 2017

I had some problems with connection to other database. This commit gemius@a705ba4 fixed my issue, but it does not guarantee that it is the same case.

@rslota

This comment has been minimized.

rslota commented Jul 24, 2017

Hi,

Same issue here. With locust any cassandra operation that requires connection to DBMS hangs the whole interpreter (sometimes it needs to be killed with -9). Without locust it works just fine. Also, adding the code snippet from @heyman to non-locust version doesn't affect the cassandra driver.

@rslota

This comment has been minimized.

rslota commented Jul 24, 2017

The patch from @grucin doesn't work for me.

@rslota

This comment has been minimized.

rslota commented Jul 24, 2017

Also, running:

from gevent import monkey
monkey.patch_all(thread=True)

does fix the issue. Why Locust doesn't patch threads? What are the implications of overriding this?

@cscetbon

This comment has been minimized.

cscetbon commented Aug 14, 2017

@rslota when I use it tests are executed but statistics are not updated. Is it better for you ?

@rslota

This comment has been minimized.

rslota commented Aug 14, 2017

@cscetbon ,

Yeah, when I patch at least threads (but currently in my code I just patch everything - monkey.patch_all()) everything works great. No stability issues and all statistics seem to be just fine. But... I don't load test HTTP, I have my own "drivers" to load several DBMS, so I fire success and failure events in locust manually (with my own time measurements), so maybe that's the reason I don't have issue you are seeing.

@heyman

This comment has been minimized.

Member

heyman commented Aug 16, 2017

I really don't remember if there was a good reason for not patching threads (I should have left a comment). It was made in the very early days of the development, and it's very possible that we just simply didn't use any threads at the time. Therefore I suggest that we change it to patch everything since people have reported that it seems to work fine.

@heyman heyman closed this in 97c5b5f Sep 16, 2017

@heyman

This comment has been minimized.

Member

heyman commented Sep 16, 2017

I've now changed so that we also patch threading. It's released in the latest version on PyPI (0.8a4).

rizalfr pushed a commit to erlanggakrisnamukti/locust that referenced this issue May 28, 2018

Prima Update locust base version to 0.8.1 (#70)
* find locustfile in the root directory

* Add ability to write out csv file data as part of command line run

* Update minimum version of gevent to fix locustio#598

* CSV flag, review comments

* Modify gevent wsgi and libev dependencies

* Sort all python imports

* Sort setup.py imports

* Standardize utf8 file coding declarations

* Bump version to 0.8a3 for another pre-release candidate

* Changed from deprecated method name

* Added test_suite argument to setup.py’s setup() method call

Not specifying the test_suite argument (using the default one) resulted in all tests being run multiple times (on Python 3.5). I haven’t looked in to why, since specifying “locust.test” should be correct, and fixes it.

* Suppress a warning that appears when running the tests on Python 3, that can cause tests (that depends on output to stderr) to fail.

* Apply gevent monkey patching before we import requests, to avoid infinite recursion error when doing SSL requests. Should fix locustio#655.

* Also patch threading when applying gevent monkey patch. Should fix locustio#569.

* Bumped version to 0.8a4

* Improved changelog

* Update main.py

Code comment and slightly more descriptive variable name

* More info about 0.8 to the Changelog

* Added some random examples that maybe could be useful to someone

* Bumped version to 0.8

* Upgrade pyzmq to latest stable version, and changed so that we do not pin the version (will make it easier to install Locust on Windows)

* Better installation instructions for Windows

* Bumped version to 0.8.1

pancaprima added a commit to erlanggakrisnamukti/locust that referenced this issue Aug 21, 2018

Merge develop to master (#89)
* Prima Update locust base version to 0.8.1 (#70)

* find locustfile in the root directory

* Add ability to write out csv file data as part of command line run

* Update minimum version of gevent to fix locustio#598

* CSV flag, review comments

* Modify gevent wsgi and libev dependencies

* Sort all python imports

* Sort setup.py imports

* Standardize utf8 file coding declarations

* Bump version to 0.8a3 for another pre-release candidate

* Changed from deprecated method name

* Added test_suite argument to setup.py’s setup() method call

Not specifying the test_suite argument (using the default one) resulted in all tests being run multiple times (on Python 3.5). I haven’t looked in to why, since specifying “locust.test” should be correct, and fixes it.

* Suppress a warning that appears when running the tests on Python 3, that can cause tests (that depends on output to stderr) to fail.

* Apply gevent monkey patching before we import requests, to avoid infinite recursion error when doing SSL requests. Should fix locustio#655.

* Also patch threading when applying gevent monkey patch. Should fix locustio#569.

* Bumped version to 0.8a4

* Improved changelog

* Update main.py

Code comment and slightly more descriptive variable name

* More info about 0.8 to the Changelog

* Added some random examples that maybe could be useful to someone

* Bumped version to 0.8

* Upgrade pyzmq to latest stable version, and changed so that we do not pin the version (will make it easier to install Locust on Windows)

* Better installation instructions for Windows

* Bumped version to 0.8.1

* Erlangga Add team_configuration property (#72)

* add get team configuration

* change read json path

* Rizal upload json file for team configuration (#68)

* enabling json uploading

* fix modal ui in upload json

* enabling upload on json also

* modify ui a bit and avoid duplicated message

* unify the function

* bug fix team configuration function (#88)

* change team_configuration function

* update configuration works
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment