gc refcount related crash when loading web UI under Python 2.6.6 #49

Closed
daubman opened this Issue Dec 18, 2012 · 7 comments

Comments

Projects
None yet
5 participants
@daubman
Contributor

daubman commented Dec 18, 2012

Running under Python 2.6.6 (Centos 6.3) I get the following crash as soon as I try and load the web UI. This seems to work fine using Python 2.7.3 as an altinstall on the same host. I have seen this on two different Centos 6.3 hosts...

--snip--
$ /usr/bin/locust -f some_locust_script.py
/usr/lib/python2.6/site-packages/locust/rpc/init.py:7: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. This will not affect you if your not running locust in distributed mode, but if you are, we recommend you to install the python packages: pyzmq and gevent-zeromq
warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. This will not affect you if your not running locust in distributed mode, but if you are, we recommend you to install the python packages: pyzmq and gevent-zeromq")
[2012-12-18 15:14:52,705] INFO/locust.main: Starting web monitor on port 8089
[2012-12-18 15:14:52,705] INFO/locust.main: Starting Locust 0.6.1
Modules/gcmodule.c:348: visit_decref: Assertion "gc->gc.gc_refs != 0" failed.
refcount was too small
object : <weakref at 0x15b1db8; to 'gevent.core.http_request' at 0x15b1d60>
type : weakref
refcount: 1
address : 0x15b1db8
Aborted
--snip--

@cgbystrom

This comment has been minimized.

Show comment
Hide comment
@cgbystrom

cgbystrom Dec 18, 2012

Member

That sounds really strange. We've run Locust fine for countless hours under different 2.6.x versions.

  1. Are you using a vanilla CentOS 6.3 version?
  2. Did you use the default Python interpreter that CentOS was shipped with or did you build it yourself?

Since I haven't experienced this myself I did quick google and for some this issue is caused by too aggressive compilation options when building the Python interpreter.

Member

cgbystrom commented Dec 18, 2012

That sounds really strange. We've run Locust fine for countless hours under different 2.6.x versions.

  1. Are you using a vanilla CentOS 6.3 version?
  2. Did you use the default Python interpreter that CentOS was shipped with or did you build it yourself?

Since I haven't experienced this myself I did quick google and for some this issue is caused by too aggressive compilation options when building the Python interpreter.

@Jahaja

This comment has been minimized.

Show comment
Hide comment
@Jahaja

Jahaja Dec 18, 2012

Member

I was able to reproduce the problem using a minimal centos 6.3 installation on a VM. As well as using the python version that's shipped with it.

It seems to be related to gevent. Updating gevent to 1.0rc2 seems to solve the problem. So that should be a work-around for now.

Member

Jahaja commented Dec 18, 2012

I was able to reproduce the problem using a minimal centos 6.3 installation on a VM. As well as using the python version that's shipped with it.

It seems to be related to gevent. Updating gevent to 1.0rc2 seems to solve the problem. So that should be a work-around for now.

@cgbystrom

This comment has been minimized.

Show comment
Hide comment
@cgbystrom

cgbystrom Dec 19, 2012

Member

I was able reproduce it as well. @Jahaja's fix works!

Will close this issue as there is a short term workaround and it will be solved when gevent hits 1.0 (Locust will switch).
If you still have issues, please comment.

Member

cgbystrom commented Dec 19, 2012

I was able reproduce it as well. @Jahaja's fix works!

Will close this issue as there is a short term workaround and it will be solved when gevent hits 1.0 (Locust will switch).
If you still have issues, please comment.

@daubman

This comment has been minimized.

Show comment
Hide comment
@daubman

daubman Jan 8, 2013

Contributor

Would you please provide some more details on the fix, I have tried this and it does not appear to help - here's what I've done:

$ uname -a
Linux ip-10-46-247-69 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS release 6.3 (Final)

$ which python
/usr/bin/python
$ python --version
Python 2.6.6

$ yum install c-ares-devel
$ yum install libev-devel
$ yum install zeromq-devel
$ pip install https://github.com/downloads/SiteSupport/gevent/gevent-1.0rc2.tar.gz
$ pip search gevent
gevent - Python network library that uses greenlet and libevent for easy and scalable concurrency
INSTALLED: 1.0rc2
LATEST: 0.13.8
...
$ pip install https://github.com/locustio/locust/archive/master.zip
$ pip install gevent-zeromq
$ pip install pyzmq
$ locust -f my_locust_file.py --master
[2013-01-08 16:07:53,945] INFO/locust.main: Starting web monitor on port 8089
[2013-01-08 16:07:53,947] ERROR/stderr: Traceback (most recent call last):
[2013-01-08 16:07:53,947] ERROR/stderr: File "/usr/bin/locust", line 9, in
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,948] ERROR/stderr: load_entry_point('locustio==0.6.1', 'console_scripts', 'locust')()
[2013-01-08 16:07:53,948] ERROR/stderr: File "/usr/lib/python2.6/site-packages/locust/main.py", line 363, in main
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,948] ERROR/stderr: runners.locust_runner = MasterLocustRunner(locust_classes, options.hatch_rate, options.num_clients, num_requests=options.num_requests, host=options.host, master_host=options.master_host)
[2013-01-08 16:07:53,948] ERROR/stderr: File "/usr/lib/python2.6/site-packages/locust/runners.py", line 241, in init
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.greenlet.spawn(self.client_listener).link_exception()
[2013-01-08 16:07:53,949] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 370, in link_exception
[2013-01-08 16:07:53,949] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.link(receiver=receiver, SpawnedLink=SpawnedLink)
[2013-01-08 16:07:53,949] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 355, in link
[2013-01-08 16:07:53,949] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.rawlink(SpawnedLink(receiver))
[2013-01-08 16:07:53,950] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 23, in init
[2013-01-08 16:07:53,950] ERROR/stderr:
[2013-01-08 16:07:53,950] ERROR/stderr: raise TypeError("Expected callable: %r" % (callback, ))
[2013-01-08 16:07:53,950] ERROR/stderr: TypeError
[2013-01-08 16:07:53,950] ERROR/stderr: :
[2013-01-08 16:07:53,950] ERROR/stderr: Expected callable: None
[2013-01-08 16:07:53,950] ERROR/stderr:

Also, from this post, it appears locust's use of gevent is not compatible with 1.0rc2:
https://groups.google.com/forum/?fromgroups=#!topic/gevent/JHbtDU5HtL8

Contributor

daubman commented Jan 8, 2013

Would you please provide some more details on the fix, I have tried this and it does not appear to help - here's what I've done:

$ uname -a
Linux ip-10-46-247-69 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS release 6.3 (Final)

$ which python
/usr/bin/python
$ python --version
Python 2.6.6

$ yum install c-ares-devel
$ yum install libev-devel
$ yum install zeromq-devel
$ pip install https://github.com/downloads/SiteSupport/gevent/gevent-1.0rc2.tar.gz
$ pip search gevent
gevent - Python network library that uses greenlet and libevent for easy and scalable concurrency
INSTALLED: 1.0rc2
LATEST: 0.13.8
...
$ pip install https://github.com/locustio/locust/archive/master.zip
$ pip install gevent-zeromq
$ pip install pyzmq
$ locust -f my_locust_file.py --master
[2013-01-08 16:07:53,945] INFO/locust.main: Starting web monitor on port 8089
[2013-01-08 16:07:53,947] ERROR/stderr: Traceback (most recent call last):
[2013-01-08 16:07:53,947] ERROR/stderr: File "/usr/bin/locust", line 9, in
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,948] ERROR/stderr: load_entry_point('locustio==0.6.1', 'console_scripts', 'locust')()
[2013-01-08 16:07:53,948] ERROR/stderr: File "/usr/lib/python2.6/site-packages/locust/main.py", line 363, in main
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,948] ERROR/stderr: runners.locust_runner = MasterLocustRunner(locust_classes, options.hatch_rate, options.num_clients, num_requests=options.num_requests, host=options.host, master_host=options.master_host)
[2013-01-08 16:07:53,948] ERROR/stderr: File "/usr/lib/python2.6/site-packages/locust/runners.py", line 241, in init
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.greenlet.spawn(self.client_listener).link_exception()
[2013-01-08 16:07:53,949] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 370, in link_exception
[2013-01-08 16:07:53,949] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.link(receiver=receiver, SpawnedLink=SpawnedLink)
[2013-01-08 16:07:53,949] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 355, in link
[2013-01-08 16:07:53,949] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.rawlink(SpawnedLink(receiver))
[2013-01-08 16:07:53,950] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 23, in init
[2013-01-08 16:07:53,950] ERROR/stderr:
[2013-01-08 16:07:53,950] ERROR/stderr: raise TypeError("Expected callable: %r" % (callback, ))
[2013-01-08 16:07:53,950] ERROR/stderr: TypeError
[2013-01-08 16:07:53,950] ERROR/stderr: :
[2013-01-08 16:07:53,950] ERROR/stderr: Expected callable: None
[2013-01-08 16:07:53,950] ERROR/stderr:

Also, from this post, it appears locust's use of gevent is not compatible with 1.0rc2:
https://groups.google.com/forum/?fromgroups=#!topic/gevent/JHbtDU5HtL8

@heyman heyman reopened this Jan 9, 2013

heyman added a commit that referenced this issue Jan 9, 2013

@heyman

This comment has been minimized.

Show comment
Hide comment
@heyman

heyman Jan 9, 2013

Member

I've just pushed some changes which I think fixes the problems. Could you try to see if works and then I can push a new version to PyPI.

Thanks for reporting!

Member

heyman commented Jan 9, 2013

I've just pushed some changes which I think fixes the problems. Could you try to see if works and then I can push a new version to PyPI.

Thanks for reporting!

@jvenkataramanan

This comment has been minimized.

Show comment
Hide comment
@jvenkataramanan

jvenkataramanan Jan 9, 2013

Yes this change seems to fix all the issues. Thanks for getting to it so fast!

Yes this change seems to fix all the issues. Thanks for getting to it so fast!

@heyman

This comment has been minimized.

Show comment
Hide comment
@heyman

heyman Jan 10, 2013

Member

That's great to hear! The fix is now in a new 0.6.2 release on PyPI.

Member

heyman commented Jan 10, 2013

That's great to hear! The fix is now in a new 0.6.2 release on PyPI.

@heyman heyman closed this Jan 10, 2013

pancaprima added a commit to pancaprima/locust that referenced this issue May 14, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment