Incompatibility with django whitenoise and runmodwsgi

[jakobholmelund]

Hi i just stumbled upon a little problem, but it should be fairly easy to fix.

The problem is, that when you use manage.py runmodwsgi and have STATIC_ROOT configured Apache2 will take over static file serving from django whitenoise. A fix could be a flag that disables the automatic behavior from runmodwsgi.

evansd/whitenoise#38

[GrahamDumpleton]

What is the reason you would want to run whitenoise when hosting static files from Apache is going to be more efficient and avoid a reduction in potential capacity of the Python web application to handle requests truly intended for the Python web application?

Using whitenoise in a production system when better options are available wouldn't be the best idea.

[GrahamDumpleton]

BTW. If you want compressed responses, use the '--compress-responses' option to 'runmodwsgi'.

[jakobholmelund]

Well. I was under the impression that whitenoise was pretty widely used. If there is a better solution i'm all ears, after all i wan't the best solution. I'll try using --compress-responses instead. Is there any other settings i should look at ? It's quite hard finding any good articles on running in production. I've checked out your blog but it's really hard separating the parts that i can use.

--compress-responses gives me

[GrahamDumpleton]

Whitenoise became popular because Heroku makes it difficult to run proper web servers that handle static files well. So Whitenoise is a compromise for PaaS providers which provide such restrictions on what WSGI server you can run.

What requirements do you have beyond providing compressed responses? When using '--compress-responses', it will compress plain text, HTML, XML, CSS and Javascript. Other file types can be set up to be compressed if necessary with a bit of configuration magic.

I acknowledge that mod_wsgi documentation has got into a sad state these days. Hoping to start fixing that soon. I hope to also be continuing with some blog posts about performance tuning as well after I have got a talk about performance issues in WSGI servers out of the way at PyCon AU next week.

[jakobholmelund]

Makes sense :)

Hmm getting

AH00526: Syntax error on line 268 of /tmp/mod_wsgi-localhost:80:0/httpd.conf:
Invalid command 'AddOutputFilterByType', perhaps misspelled or defined by a module not included in the server configuration

Any ideas ? again google isn't any help

[GrahamDumpleton]

What version of Apache are you using?

[jakobholmelund]

I'm using https://github.com/GrahamDumpleton/mod_wsgi-docker it seems to state APACHE_VERSION=2.4.12

[GrahamDumpleton]

Seems it got moved to mod_filter in Apache 2.3.7. When I tested I was most likely using Apache 2.2, where it was part of core Apache or some other module which was being loaded already.

I will need to fix up mod_wsgi-express to add:

<IfVersion >= 2.4>
LoadModule LoadModule filter '${MOD_WSGI_MODULES_DIRECTORY}/mod_mfilter.so'
</IfVersion>

In the mean time you could use the --include-file option to include a file containing:

<IfModule !deflate_module>
LoadModule deflate_module '${MOD_WSGI_MODULES_DIRECTORY}/mod_deflate.so'
</IfModule>

<IfVersion >= 2.4>
LoadModule LoadModule filter '${MOD_WSGI_MODULES_DIRECTORY}/mod_mfilter.so'
</IfVersion>

AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/javascript

I will though try and get a quick update released.

[jakobholmelund]

Cool, nice to squash a bug. And very nice with --include-file, now i know how to add other apache modules myself :D

[jakobholmelund]

A couple of typos in your paste :) if anybody sees this, try this instead

<IfModule !deflate_module>
LoadModule deflate_module '/usr/lib/apache2/modules/mod_deflate.so'
</IfModule>

<IfVersion >= 2.4>
<IfModule !filter_module>
LoadModule filter_module '/usr/lib/apache2/modules/mod_filter.so'
</IfModule>
</IfVersion>

AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/javascript

[GrahamDumpleton]

If using a recent mod_wsgi-express version, then ${MOD_WSGI_MODULES_DIRECTORY} should be fine. It will use the environment variable of that name setup by the apachectl script.

[GrahamDumpleton]

The compression issue with Apache 2.4 is fixed in 'develop' branch for next release.

Can test by using:

pip install -U https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip

[evansd]

Unsurprisingly, I disagree with @GrahamDumpleton's assessment that WhiteNoise exists because Heroku won't let you run anything better ;)

The intention was always that, for anything serious, WhiteNoise would be used behind a CDN hence the far-future caching, and the allow-origin headers. Obviously it's possible to configure Apache/ngninx/etc to handle all this and the compression, but the point of WhiteNoise was to do the right thing out of the box with no extra configuration. Of course its performance is never going to match a "real" webserver, but if it's behind a CDN that doesn't matter anyway. And if you care about performance, why are you not using a CDN?

At any rate, that's my defence of WhiteNoise's existence!

[GrahamDumpleton]

Let me clarify what I meant. I wasn't implying that Whitenoise exists because of any issue with Heroku. In fact, others package came before Whitenoise, such as dj-static. What the various packages out there, in conjunction with restrictions on what you could easily run on Heroku caused, was a trend for many Python developers to get the idea that it was acceptable to host static files from the Python web application rather than use more performant solutions that already existed. This spread outwards to other hosting environments as well. It is all a part of a bigger problem in the Python web community that not a lot of focus is given to properly tuning the performance of Python web applications and the underlying WSGI servers. People too quickly look for quick solutions which they think will save them time, even though in the grander scheme of things if their site ever did become popular and needed to scale, could just waste them money as they would have to use more resources than if things had been setup in better ways. This would also affect Whitenoise, because although pairing it with a CDN could yield a more optimal solution, the likelihood of people doing that is probably low as they will see it as working anyway, so why bother changing.

[evansd]

Yep, all fair points.

I think because I'm always quite conscious of the performance tradeoffs I'm making I tend to overestimate the extent to which other people are even aware these things.

Thanks for all the work you're doing in this area.

[GrahamDumpleton]

The documented story out there about the performance implications of running certain workload types with different WSGI server configurations definitely needs to be better. I hope to start addressing that in a talk I will be presenting at PyCon AU next week and then some followup blog posts.

One of the things I will be talking about is server capacity and the implications of long running requests. This has some relevant to to the issue of serving static files out of the Python web application as serving up of large static files, especially over a slow network to a slow client, such as mobile devices, means that you are potentially keeping busy a request thread which could be serving up many requests in the same time for actual dynamic requests.

So the use of WSGI middleware to handle static files, would be a good illustration for my talk of the potential dangers of this problem.

[jakobholmelund]

Should i close this ?

[GrahamDumpleton]

No. I still need to look at providing some option to disable mod_wsgi-express integration taking over static file handling. No reason one shouldn't have that option.

[svancaster]

Graham, Thanks so much for your blogs and work on mod_wsgi-express. I am trying to use the directive --compress-responses. Is there anything else to this besides the LoadModule deflate_module /usr/app/apache/apache24/modules/mod_deflate.so? Is there a way to tell if the compression is really happening (browser header record or something)? Thanks

[svancaster]

How does mfilter play into compressions? Is this module needed, it did not load with my version of mod_wsgi-express 4.4.15. How can I get it if needed?

Thanks
Scott

[GrahamDumpleton]

Do you mean mod_filter? Not sure what you mean by mfilter.

Compression only works for certain file types by default, based on what response content type was set to:

<IfDefine MOD_WSGI_COMPRESS_RESPONSES>
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/javascript
</IfDefine>

What is your response content type?

Am busy travelling and a conference right now, but will when time try and verify that compression is still working and how you can check for yourself.

[svancaster]

Yes I do mean mod_filter, and I did find it installed on my server. Response content type is text/HTML. Thanks for your response.

[GrahamDumpleton]

Am going to change mod_wsgi-express to not automatically add the --url-alias option for static files if 'whitenoise.middleware.WhiteNoiseMiddleware' is listed in MIDDLEWARE_CLASSES.

If for some reason that middleware was included but really did want to use Apache to host static files, they can use --url-alias option themselves.

Sorry this has taken so long to come back to. Trying very hard to clear out backlog of issues.

[evansd]

Thanks Graham. That sounds like a good solution.

[GrahamDumpleton]

Closing. Change made in 37cb2e1 for version 4.5.25.

Also had to check for MIDDLEWARE in addition to MIDDLEWARE_CLASSES due to change in Django 1.10 to name of setting.