Offload ZODB BLOB download to front end web server using XSendfile/HTTP-Accel Headers.
XSendFile is an enhancement over HTTP front end proxy protocol which allows offloading of file uploads and downloads to the front end web server.
collective.xsendfile package adds XSendFile support for Plone.
- Plone handles HTTP request publishing, permission checks, etc. still normally
- But instead of sending the file content over proxy connection Plone sends HTTP response with special header telling the front end web server to read the file from the disk and send the file for the user
Blob handling in ZODB is very effective already (async sockets, just like Apache or nginx would do). Right after the headers are written to the response, the file gets handed over to the medusa async loop and the Zope thread is freed. This add-on only removes the need to proxy the file data over socket connection. The overhead of this may depend on the use case, so you might want to run some benchmarks before conclusion.
XSendFile support is available as
collective.xsendfile add-on for Plone.
This work is still unfinished as ZODB lacks one crucial feature regarding permissions
- direct url to ATFile and ATImage objects
- direct url to
plone.app.contenttypesFile and Image objects
Other urls will use the normal zope download mechanism.
Currently image scales aren't handled as xsendfile even though they are stored as blobs.
There are two ways to configure collective.xsendfile, either site by site, or globally per zope instance
- Put collective.xsendfile to your buildout
- Install the add-on to your site(s) through Plone add-on control panel
- Enable XSendFile module on your front-end web server and virtual host configuration
- In XSendFile Plone control panel, set HTTP header according to your server (Apache/Nginx)
It is also possible to setup collective.xsendfile globablly for all your plone sites in a plone instance by using environment variables. Note configuration this way will disable the ability to configure per site. There is no need to activate the plugin in your Plone instance for this to work.
Put collective.xsendfile to your buildout
configure you zope instance (probably via buildout) to include set the following environment variables
will activate global configuration. Likely values are either
True means if
HTTP_X_FORWARDED_FORisn't found in the request prevent xsendfile processing from occuring.
If you need modify the full path of a blob you can extract parts of it here. Defaults to
If you need to modify the full path of a blob you can use this replace parts of the path here. Defaults to
\1. If you are using nginx is will likely be something like
Include it in the buildout.cfg:
eggs = collective.xsendfile
Install Apache module (Debian/Ubuntu):
# alternatively -thread-dev, depends on your apache configuration sudo apt-get install apt-get install apache2-prefork-dev wget --no-check-certificate https://tn123.org/mod_xsendfile/mod_xsendfile.c sudo apxs2 -cia mod_xsendfile.c
Enable Apache module:
sudo a2enmod xsendfile
Related virtual host configuration file:
Listen 8082 LoadModule xsendfile_module modules/mod_xsendfile.so <VirtualHost *:8082> ServerName test XSendFile on XSendFilePath / RewriteEngine On RewriteRule (.*) http://127.0.0.1:8080/VirtualHostBase/http/test:8082/VirtualHostRoot/$1 [L,P] </VirtualHost>
Here's a nginx.conf, take a closer look at the server locations, that's where the magic happens.
- Apache XSendFile installation instructions (Debian/Ubuntu)
If you get HTTP response like:
OK The requested URL /site-images/xxx/cairo.jpg was not found on this server.
It is probably a file permission issue.
- Peter Holzer email@example.com
- Georg Gogo. BERNHARD firstname.lastname@example.org
- Mikko Ohtamaa email@example.com
- Jens W. Klein firstname.lastname@example.org
- Dylan Jay email@example.com
Special thanks to Kapil Thangavelu, we extensively borrowed from his code ;-)