Skip to content

Error related to 1.4 "feature" validate_thread_sharing #293

Closed
wiz opened this Issue Jan 26, 2012 · 26 comments

6 participants

@wiz
wiz commented Jan 26, 2012

gevent and eventlet workers stopped working with a fix for issue #17258

Traceback (most recent call last):
File "/home/wiz/ops/venv/lib/python2.7/site-packages/gunicorn/workers/async.py", line 38, in handle
self.handle_request(req, client, addr)
File "/home/wiz/ops/venv/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 88, in handle_request
super(GeventWorker, self).handle_request(args)
File "/home/wiz/ops/venv/lib/python2.7/site-packages/gunicorn/workers/async.py", line 64, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/home/wiz/ops/venv/src/django/django/contrib/staticfiles/handlers.py", line 67, in call
return self.application(environ, start_response)
File "/home/wiz/ops/venv/src/django/django/core/handlers/wsgi.py", line 243, in call
signals.request_finished.send(sender=self.class)
File "/home/wiz/ops/venv/src/django/django/dispatch/dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, *
named)
File "/home/wiz/ops/venv/src/django/django/db/init.py", line 46, in close_connection
conn.close()
File "/home/wiz/ops/venv/src/django/django/db/backends/postgresql_psycopg2/base.py", line 133, in close
self.validate_thread_sharing()
File "/home/wiz/ops/venv/src/django/django/db/backends/init.py", line 136, in validate_thread_sharing
% (self.alias, self._thread_ident, thread.get_ident()))
DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with > alias 'default' was created in thread id 140373775284000 and this is thread id 53339144.

@bryanveloso

+1, just ran into this today.

@benoitc
Owner
benoitc commented Feb 20, 2012

When monkey patching with gevent, threading.local is patched to use the local greenlet. Since we are spawning a new greenlet per request , the local ident change which raises the issue you have.

I don't see any other solution other than not patching threading when starting gunicorn right now. Same problem should appear with eventlet. @tilgovi , @davisp what do you think?

Anyway, I guess that with this change, django with postgres probably stopped to work with any gevent/eventlet app that patch threading. Any feedback from upstream would be interesting.

@benoitc
Owner
benoitc commented Feb 20, 2012

bump

@benoitc
Owner
benoitc commented Feb 21, 2012

I couldn't reproduce on my machine. Can you eventually try with the last head?

@wiz
wiz commented Feb 21, 2012

Just did a full upgrade. It is still there with eventlet.

My versions are:

-e git+https://github.com/django/django.git@5cedcb412598018f9f58f7f1abac31c638caf1a9#egg=Django-dev

eventlet==0.9.16
greenlet==0.3.4
gunicorn==0.13.4

upd: same with gunicorn.git head...

@wiz
wiz commented Feb 21, 2012
diff --git a/gunicorn/workers/geventlet.py b/gunicorn/workers/geventlet.py
index e020ca4..140008d 100644
--- a/gunicorn/workers/geventlet.py
+++ b/gunicorn/workers/geventlet.py
@@ -23,7 +23,7 @@ class EventletWorker(AsyncWorker):
         import eventlet
         if eventlet.version_info < (0,9,7):
             raise RuntimeError("You need eventlet >= 0.9.7")
-        eventlet.monkey_patch(os=False)
+        eventlet.monkey_patch(os=False, thread=False)

     def init_process(self):
         hubs.use_hub()

Turning thread patching off does the job, but i'm unsure if any special effects will be happening, but it's should be okay for me since i don't use any threads in django sites.

@apollo13

@wiz: How exactly are you running gunicorn? (whole commandline please)

@wiz
wiz commented Feb 21, 2012

./manage.py run_gunicorn -n 'ercc' -b 127.0.0.1:4566 -p var/gunicorn.pid -w 2 --max-requests=1000 -k eventlet

@apollo13

does using gunicorn_django instead of the manage.py command fix the issue?

@bryanveloso

@apollo13 — I gave it a try and it does fix it for me.

@wiz
wiz commented Feb 21, 2012
venv/bin/gunicorn_django -b 0.0.0.0:8001 --settings=ercc.settings --pythonpath=. -k eventlet

Yes, it works this way.

@benoitc benoitc added a commit that closed this issue Feb 21, 2012
@benoitc don't validate models and activate translation in run_gunicorn command.
close #293.

We already do that in the worker application load. Not doing it here
should fix application reload and issue with threads.
.
a77a166
@benoitc benoitc closed this in a77a166 Feb 21, 2012
@benoitc
Owner
benoitc commented Feb 21, 2012

@wiz it should also now works with a77a166

@wiz
wiz commented Feb 21, 2012

@benoitc sadly, it doesn't. I've upgraded to a77a166 and it still 500-ing with run_gunicorn and OK with gunicorn_django.

@benoitc
Owner
benoitc commented Feb 21, 2012

odd. i should look if there is some magie to close the current connection thread. if you find anything let me know.

@benoitc benoitc reopened this Feb 21, 2012
@benoitc benoitc added a commit that closed this issue Feb 21, 2012
@benoitc monkey patch django. close #293 .
This patch make sure that we use real threads to get the ident which is
going to happen if we are using gevent or eventlet.
161c8c0
@benoitc benoitc closed this in 161c8c0 Feb 21, 2012
@benoitc
Owner
benoitc commented Feb 21, 2012

@wiz @bryanveloso it should be OK now. I don't reproduce it at all.

@wiz
wiz commented Feb 21, 2012

Yup. Works now. Thanks!

@apollo13

I would prefer if you monkeypatch only if manage.py run_gunicorn is used and not globally in gunicorn/workers/ggevent.py or is that not possible?

@benoitc
Owner
benoitc commented Feb 21, 2012

@apollo13 done. Thanks for the suggestion.

@ace-han
ace-han commented Sep 4, 2014

I came across this error today.
gunicorn 18.0 is working okay while 19.0, 19.1.0 $ 19.1.1 are not.

>$ uname -a
Linux ace 2.6.32-358.6.2.el6.x86_64 #1 SMP Thu May 16 20:59:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>$ python -V
Python 2.7.8

>$ pip freeze 
Django==1.6.5
MySQL-python==1.2.5
Pillow==2.4.0
South==0.8.4
Unidecode==0.04.16
argparse==1.2.1
cmsplugin-filer==0.9.9
dj-database-url==0.3.0
django-appconf==0.6
django-classy-tags==0.5.1
django-cms==3.0.2
django-email-extras==0.2
django-filer==0.9.5
django-forms-builder==0.11.1
django-mptt==0.5.5
django-polymorphic==0.5.5
django-reversion==1.8.0
django-sekizai==0.7
djangocms-admin-style==0.2.2
djangocms-column==1.3
djangocms-file==0.0.1
djangocms-flash==0.0.2
djangocms-googlemap==0.0.5
djangocms-inherit==0.0.1
djangocms-installer==0.5.1
djangocms-link==1.3.4
djangocms-picture==0.0.2
djangocms-snippet==1.0.2
djangocms-style==1.3
djangocms-teaser==0.0.1
djangocms-text-ckeditor==2.1.4
djangocms-video==0.0.1
easy-thumbnails==2.0.1
future==0.9.0
gevent==1.0.1
greenlet==0.4.2
gunicorn==19.1.1
html5lib==0.999
pysqlite==2.6.3
python-gnupg==0.3.6
pytz==2014.3
six==1.6.1
sphinx-me==0.2.1
wsgiref==0.1.2
xlwt==0.7.5
@romabysen

If you're using the gevent worker and postgresql you should also use psycogreen or a pure-python implementation like pg8000.

@ace-han
ace-han commented Sep 4, 2014

Hi, @romabysen , I'm using the gevent worker and mysql.

Could you be more specific on psycogreen for my case? where and how can I apply this psycogreen ? thx in advanced

@romabysen

Ah, I thought you used PostgreSQL. MySQL-python is not async-friendly, you should try using a pure-python implementation like pymysql instead.

@ace-han
ace-han commented Sep 4, 2014

As I'm using django, it got a dependency on MySQL-python, which makes me out of choice...
Any suggestion? Since 18.0 works fine, I will stick to 18.0 as a short term solution

@romabysen

It's probably only working in 18.0 by the grace of God and if Django can't use any other mysql driver than MySQL-python then you're SOL for async.

@ace-han
ace-han commented Sep 9, 2014

@romabysen , I followed the instruction based on this link to setup pymysql to work with django. And still got the same error. Any idea?

(venv1)[xxx@xxx dir1]$ python -V
Python 2.7.8
(venv1)[xxx@xxx dir1]$ pip freeze
Django==1.6.5
...
PyMySQL==0.6.2
...
future==0.9.0
gevent==1.0.1
greenlet==0.4.2
gunicorn==19.1.1
...

mysql> status
--------------
mysql  Ver 14.14 Distrib 5.6.10, for Linux (x86_64) using  EditLine wrapper

Connection id:          827181
Current database:
Current user:           vvcms@10.222.43.120
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server version:         5.6.16-log MySQL Community Server (GPL)
Protocol version:       10
Connection:             146.222.79.181 via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:               3306
Uptime:                 70 days 17 hours 43 min 36 sec

Threads: 39  Questions: 158162376  Slow queries: 13  Opens: 3252  Flush tables: 1  Open tables: 541  Queries per second avg: 25.878
--------------
@benoitc
Owner
benoitc commented Sep 9, 2014

mmm working on closed issue is hard. Can you open a new ticket with your issue? Also how do you start your application ? are you running fjango using the gunicorn command?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.