Seemingly random crashes when used with uWSGI, Django and Celery on MacOS #167
Comments
|
Most of #10 does not apply anymore as WeasyPrint moved away from PyGObject and PyGTK. What is relevant from #10 is that you can sometimes get Other than that, I’m sorry but I don’t know how to help. The second error is definitely in the realm of CFFI, though. Try asking on #pypy @ Freenode (which also covers CFFI) or on CFFI’s issue tracker what could cause this. |
|
Had the exact same errors today : WeasyPrint is simply imported in one of our Django app, and fails with these tracebacks. I believe it has something to do with threading : we use Gunicorn, and when simultaneous AJAX calls are processed, only 1 succeeds... |
|
@NotSqrt @ryankuczka Is this bug fixed using the current versions of the dependencies? |
|
@liZe if I remember correctly, upgrading a while back seemed to resolve the As for the other issue I had, I eventually removed all imports of weasyprint from the module level so that it was only imported when it was actually used which "resolved" the issue for me. I no longer use weasyprint in this project so I can't comment on if the issues still exist today. |
|
@ryankuczka Thank you. |
|
@liZe I haven't seen it recently, And was not able to reproduce it right now. |
|
We moved the |
|
@NotSqrt Thank you too, it's very useful. I bet it has something to do with uWSGI's "magic" forking, as explained in this StackOverflow post. Did you use uWSGI (last question, I promise :p)? |
|
At that time, it was gunicorn, in its regular config. |
|
Well, gunicorn doesn't preload the application before forking, as explained in the documentation. Sad :/. |
|
Has anyone learned anything more about this issue? Lazy imports (i.e. importing weasyprint only in the function that uses it) don't seem to help at this point. I'm also running into the issue using weasyprint under gunicorn, but gunicorn does now support preloading the application before forking. Loading application code pre or post fork both seem to fail when executing HTML() though (with no exception, I'm guessing its a segfault getting swallowed somewhere). Any guidance appreciated! |
|
(and I'm happy to provide any debug information that might be helpful) |
|
@wiltzius If you could find anything like a traceback or a segfault error (may be found in dmesg), that would be really useful. |
|
(Also: there's a lot of recent activity in https://github.com/fdemmer/django-weasyprint, may be useful to ask them too). |
|
Sure! Here's an example segfault: |
|
Thanks for the quick response, I appreciate it! |
|
I asked over there too, thanks for the tip. |
|
I've been using WeasyPrint in multithreaded Flask-based applications for a long time and I never hit this error… There must be something strange with gunicorn (I use uWSGI), but it's hard to find what. |
|
Perhaps significantly it's multi-process usage that's causing problems, not multi-threaded. Previously we ran it under waitress, which is multithreaded but single process. Does that give any clues? Are you using uWSGI with multithreading or multiprocessing (I believe it supports both)? I could switch us to uWSGI, but unfortunately Celery is going to be harder to ditch... |
|
Also, this is on MacOS. I'm not sure if the issue is present on Linux. From my first quick test it looks like it isn't an issue on Linux, but there are a lot of variables between my dev laptop and a linux server... |
It supports both and I use both.
I'd prefer find what's going on 😄.
Ah, OK. Does it work with waitress on MacOS? |
|
Figuring out what's going on sounds great :) Yes, it works with waitress (using ~20 threads and never had an issue) on MacOS. |
|
Further interesting news: it appears that it actually runs fine in Celery's forked worker processes. So the issue does appear specific to gunicorn! |
Good to know, thanks a lot for your help! I'll try to find what's happening as soon as I can get some time. |
|
Thanks, I appreciate the help today :) |
|
Confirmed that it works with uwsgi for me too. |
|
It seems I spoke too soon. This is utterly mysterious to me, but somehow between last night and this morning I started getting segfaults under uwsgi too. Here's an example, which has a bit more information than the gunicorn trace I posted above: |
|
(this is on weasyprint 0.36, MacOS, and the following brew package versions: libffi 3.0.13 |
liZe
changed the title
liZe
changed the title
liZe
changed the title
|
Hi, just wanted to add, that the same error occurs using django and django-q.
That's what @kgritesh wrote in #344 and it's exactly the same location where my script is failing. Unfortunately django-q doesn't nicely expose the resulting segfaults so I can't pitch in here. If you need more information please just tell me. I'll try to look into the issue some more over the next few days Regards, |
|
Could you try v0.41? f5367c8 may have fixed that. |
|
I don't think this has anything to do with uWSGI, this is present in Windows 7 with Python27
|
|
@sarathsunil Hello, Thanks for your comment, but please don't add comments long after issues are closed, you're turning issues into zombies! Your problem is explained in the documentation and in #589. |
I am using WeasyPrint with Django and Celery and seemingly random errors occur. I have a Django model which has a file field that is a pdf and a celery task which generates these pdfs using WeasyPrint.
The first error occurs on our web server and occurs on seemingly random requests and we receive the traceback below:
This exact error occurs with a relatively high frequency, although nowhere near all requests and it starts at seemingly random times. Restarting the uwsgi process which runs the server "solves" the problem.
This issue is also "solved" when I remove djcelery, since djcelery no longer tries to autoload all my tasks modules and WeasyPrint is no longer imported down the line at this point in time, only when it is required.
I originally thought this error would be related to issue #10 but I am not sure because none of the fixes in that issue seemed to work and I am also able to import and use WeasyPrint generally, it just seems to fail in this one specific case.
We also get an error on the machines which run celery workers. We have a Django management command which runs every 3 minutes, and about once a day, we receive the following traceback:
I have attempted to check the obvious issue of is the server out of memory which doesn't seem to be the case, and it also seemed odd that the error is the exact same every time. The management command also runs successfully 3 minutes before and 3 minutes after. Also, this occurs across 3 different machines, at the same time.
Both of these errors seem to occur when Django or Celery dynamically import something which eventually imports from WeasyPrint, and both seem to happen randomly as I have been unable to ever reproduce these errors myself.
The text was updated successfully, but these errors were encountered: