Serving static files with Daphne

[orokusaki]

I noticed during my quick testing of daphne / runworker that there is no automatic support for static files. I know that, in the ideal world, static files are served from a CDN, but over the past few years it's become very common practice to serve static files from your application's front-end server (e.g., using WhiteNoise - also see their IAQ).

With that said, is there a way to serve static in production with Daphne? Or, is that something you'd be interested in merging, if I was to find a reasonable way to add WhiteNoise-like functionality in a fork?

[andrewgodwin]

Serving large files over Channels is possible but not quite as good as doing it from a local filesystem - remember, every response is serialised onto the wire and back - but Channels does in fact ship with support for staticfiles, which it turns on for Daphne when you're using runserver.

I'm not averse to something like WhiteNoise, especially as I like the "you should be caching anyway" mantra, and you can implement it similarly with a wrapper around the Django http.request ASGI consumer (the equivalent of wrapping the WSGI application). I would definitely be interested in merging this or contributing support out to another project to give them ASGI support.

[jpadilla]

Here's one way to do so for anyone interested: https://github.com/jacobian/channels-example/blob/master/chat/routing.py#L5-L8

[jacobian]

@andrewgodwin it seems like perhaps Daphne should be the bit that can serve static files, thus preventing the need to shuttle the file over the wire. Does that sound right? Or is there a reason to implement static file serving as a channel consumer?

[andrewgodwin]

No, Daphne isn't necessarily co-located with the codebase; static files should either be served in front of it (by e.g. nginx) or behind it (via Django), IMO. Daphne's job is purely shutting data to and from HTTP/Channels, and I want to try and avoid adding any complexity if we can help it.

I did improve the core Handler a couple of days ago so any FileResponse is chunked up much better for transfer, which means the new WhiteNoise Django Middleware version should work well with that.

[exp0nge]

@andrewgodwin How are you setting up WhiteNoise to work with Channels in a production environment (without runserver)?

[andrewgodwin]

I am not yet running it, I need to get around to that.

[exp0nge]

I mean I tried WhiteNoise with Heroku, and JS/CSS files loaded fine but 404'd on images.

[bastianh]

I was testing WhiteNoise with dokku (http://dokku.viewdocs.io/dokku/) this weekend and it only worked for every second request for the same file.
With the first request it send the file, but when I reload in chrome or firefox and the browser asked for checking with the if-modified header the result was an empty 200 response instead of a not modified.
When trying with curl there were no errors.

[andrewgodwin]

Weird, probably an incompatability in WhiteNoise I should look into. What version were you using?

[bastianh]

I was using the current versions from pypi

whitenoise==3.2
asgi-redis==0.13.1
asgiref==0.13.3           # via asgi-redis, channels, daphne
daphne==0.12.1            # via channels
channels==0.14.1

might also have to do with the nginx setup? I will try to find out more later but would be nice if you have any ideas.

you can try by accessing this file: https://ptest.lambda.leebe.de/static/CACHE/css/cc0e6539359b.css

[helderco]

Has anyone got whitenoise to work? Anything else to do besides the django integration in their manual?

[helderco]

I thought it wasn't working but I'm actually having the same issue as @bastianh (I'm using docker).

I'm holding out on using django channels to keep it simple, or do you have another suggestion for static files?

[exp0nge]

I ended up using S3+CloudFront, I think that's your best bet if you want to get something working and something that scales well.

[helderco]

I use S3 on the media files, but I prefer having my static files in the docker container, next to the code that generated them (versions).

[bastianh]

I decided to put nginx into the container and then I decided to use the patched nginx from https://nchan.slact.net/ to directly handle websockets and channels for this project.

[andrewgodwin]

Found the issue - a bug with empty streaming responses - and it's out now in 0.15.1.

[exp0nge]

What is this fix exactly for?

[andrewgodwin]

It makes whitenoise work properly with Daphne, and this is what I'm suggesting for static file serving for now on - no need to reinvent the wheel.

[slact]

Hello there, I'm the author of Nchan. I'm just curious how you're using it -- looks like you're serving large-ish files over WS -- and if you're also using the javascript client or rolling your own.

[johnnyflute]

so the 'python manage.py runsever' can be used to handle static file or not

[exp0nge]

@johnnyflute It's been a while since I used this but maybe Andrew's answer can help:

It makes whitenoise work properly with Daphne, and this is what I'm suggesting for static file serving for now on - no need to reinvent the wheel.

So you need to do some extra leg work.

[andrewgodwin]

Runserver works properly with static files and has basically since launch. Daphne by itself does not.

[puterleat]

Joining this late, but the problem with Whitenoise is that it won't work for client uploaded media. Taking the case where these are stored locally (or on a mounted volume in dokku) dj_static works quite nicely with standard django (no channels) by wrapping the WSGI application. Could this also work for asgi/daphne?

[andrewgodwin]

If it's a WSGI wrapper/middleware it won't work. You'll need to either serve those files using a file-serving webserver (nginx, etc), or find a Django-level application that works.

[benwhalley]

Sorry - I wasn't being clear. I realise it won't work out of the box as a wsgi wrapper, but I suppose my question was, will this general approach still be feasible with Daphne.

My use case is a smallish app where I don't want to use S3 as the storage. I have a lot of user-uploaded assets which are used for processing. For staticfiles I use cloudfront as a CDN, so load on the server isn't an issue. For the mediafiles there's not much opportunity for caching anyway. I've previously used dj_static for both staticfiles and mediafiles which has worked well, but I now want to add channels to my app.

For staticfiles this is OK because, as you note, whitenoise now works with Daphne, so I can switch to that. However for mediafiles this won't work because it builds a manifest of all images at startuptime, not on the fly. I suppose what I'd really like is for something like dj_static to be built into Daphne. I would really rather not have to reconfigure nginx to serve the media files.

[andrewgodwin]

I'm pretty resistant to building static file serving into Daphne, mainly as I want to try and keep it as simple and maintainable as possible; doing static file serving right requires stream options, a mimetype library, and more, and that's out of scope with the maintenance team we have.

I think what's needed is something on the Django end that correctly serves user media if required, that plugs in well. I'm sorry nothing perfectly fills this gap right now, but most people I know already host static files on a subdomain or CDN URL and so they already have a separate server configured to handle them.

[TheBlusky]

I had a similar issue, WhiteNoise couldn't work with daphne as it was configure as a WSGI wrapper (with application = DjangoWhiteNoise(application)).

However, you can configure WhiteNoise as a django middleware, adding :

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    ....
]

to your settings.py file.

Worked for me with daphne.