Skip to content

Commit

Permalink
[#519] Update deployment docs for CKAN 2.0
Browse files Browse the repository at this point in the history 
This is a fairly big rewrite of the deployment docs that also necessitates
changes to other docs (like the source install docs, etc.)

The idea is to document the deployment process that OKFN is currently using on
its servers, so that the deployment docs can become the docs for the OKFN dev
team and we can dogfood our own docs, instead of having a bunch of wiki pages,
READMEs, etc. that we follow and ignoring the "official" deployment docs.

Secondly, this aims to merge source install and deployment into one process.
Deploying CKAN is just doing a source install, and then doing a couple of
extra things to create the WSGI and Apache config files. But because the
source install docs used `~/pyenv` and the deployment docs used
`/usr/local/demo.ckan.net`, it was much harder to do a deployment than it
should be. You would first have to follow the source install docs but replace
`~/pyenv` with `/usr/local/demo.ckan.net` everywhere and deal with all the
consequences. Then you would follow the deployment docs.

So both the Source Install and Deployment docs now describe the layout we use
on our servers, where there's space for multiple CKAN sites on the same
server, with virtualenvs in `/lib/ckan/` and config files in `/etc/ckan/`.

Since the multi-site layout requires a site name (for the name of the site's
virtualenv, etc.) I've invented a fictional CKAN site called `masaq` as an
example.

- Update the deployment docs to use a virtualenv at `/lib/ckan/masaq`, config
  files at `/etc/ckan/masaq`, etc.
- The deployment docs now use the WSGI and Apache config files that we're
  actually using on our servers, not the old versions that were in the docs
- Deleted stuff about enabling CORS from deployment, apparently this is
  builtin since CKAN 1.5.

- Update the install-from-source docs to use a virtualenv at `/lib/ckan/masaq`
  and config files at `/etc/ckan/masaq`, etc.

  The install-from-source and deployment now use the same ``masaq`` example
  with the same directory layout, database names, etc. So it's possible to
  step through the source install and the deployment docs exactly as written
  (copy-pasting everything without modification) and end up with a working
  deployed site.

- Also added info to the install-from-source docs about how the filesystem
  layout etc is done and why

- Update the DataStore Setup docs to use the `masaq` database user, and to
  call the datastore database `masaq-datastore`.

- Update the solr-setup docs, this is just tweaking some formatting issues
  and updating some commands to follow the `masaq` example.

- Update the post-install docs to fit with the `masaq` example as well

- Update the testing docs to fit with the `masaq` example

- Update `test-core.ini` to use the `masaq` postgres user name

- Disable the log file in `deployment.ini_tmpl`, as we do on our servers, and
  rely only on the Apache logs
  • Loading branch information
Sean Hammond committed Apr 24, 2013
1 parent 425133e commit 808391f
Show file tree
Hide file tree
Showing 8 changed files with 311 additions and 265 deletions.
25 changes: 16 additions & 9 deletions ckan/config/deployment.ini_tmpl
View file
Expand Up @@ -46,7 +46,14 @@ use = egg:ckan
full_stack = true
cache_dir = %(here)s/data
beaker.session.key = ckan

# This is the secret token that the beaker library uses to hash the cookie sent
# to the client. `paster make-config` generates a unique value for this each
# time it generates a config file.
beaker.session.secret = ${app_instance_secret}

# `paster make-config` generates a unique value for this each time it generates
# a config file.
app_instance_uuid = ${app_instance_uuid}

# List the names of CKAN extensions to activate.
Expand Down Expand Up @@ -302,24 +309,24 @@ ckan.debug_supress_header = false
keys = root, ckan, ckanext

[handlers]
keys = console, file
keys = console

[formatters]
keys = generic

[logger_root]
level = WARNING
handlers = console, file
handlers = console

[logger_ckan]
level = INFO
handlers = console, file
handlers = console
qualname = ckan
propagate = 0

[logger_ckanext]
level = DEBUG
handlers = console, file
handlers = console
qualname = ckanext
propagate = 0

Expand All @@ -329,11 +336,11 @@ args = (sys.stderr,)
level = NOTSET
formatter = generic

[handler_file]
class = logging.handlers.RotatingFileHandler
formatter = generic
level = NOTSET
args = ("ckan.log", "a", 20000000, 9)
#[handler_file]
#class = logging.handlers.RotatingFileHandler
#formatter = generic
#level = NOTSET
#args = ("ckan.log", "a", 20000000, 9)

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
32 changes: 16 additions & 16 deletions doc/datastore-setup.rst
View file
Expand Up @@ -32,37 +32,37 @@ Create users and databases

.. tip::

As is done in the example commands below, we recommend reusing your existing
CKAN database user (``ckanuser`` in :doc:`install-from-source`) as the
readwrite user for your datastore database.
If your CKAN database and DataStore databases are on different servers, then
you need to create a new database user on the server where the DataStore
database will be created. As in :doc:`install-from-source` we'll name the
database user ``masaq``, after the site we're creating::

However, this might not be possible if the CKAN database and the DataStore
database are on different servers. In this case, you should create a new
database user on the server with the DataStore database::

sudo -u postgres createuser -S -D -R -P -l writeuser

Then in the commands below, replace ``ckanuser`` with ``writeuser``.
sudo -u postgres createuser -S -D -R -P -l masaq

Create a database user called ``readonlyuser``. This user will be given
read-only access to your DataStore database in the `Set Permissions`_ step
below::

sudo -u postgres createuser -S -D -R -P -l readonlyuser

Create the database (owned by ``ckanuser``), which we'll call ``datastore``::
Create the database (owned by ``masaq``), which we'll call
``masaq-datastore``::

sudo -u postgres createdb -O ckanuser datastore -E utf-8
sudo -u postgres createdb -O masaq masaq-datastore -E utf-8

Set URLs
--------

Now, uncomment the ``ckan.datastore.write_url`` and ``ckan.datastore.read_url`` lines in your CKAN config file and edit them if necessary::
Now, uncomment the ``ckan.datastore.write_url`` and ``ckan.datastore.read_url``
lines in your CKAN config file and edit them, for example::

# Datastore
# Uncommment to set the datastore urls
ckan.datastore.write_url = postgresql://ckanuser:pass@localhost/datastore
ckan.datastore.read_url = postgresql://readonlyuser:pass@localhost/datastore
ckan.datastore.write_url = postgresql://masaq:pass@localhost/masaq-datastore
ckan.datastore.read_url = postgresql://readonlyuser:pass@localhost/masaq-datastore

Replace ``pass`` with the passwords you created for your ``masaq`` and
``readonlyuser`` database users.

Set Permissions
---------------
Expand All @@ -78,7 +78,7 @@ This option is preferred if CKAN and PostgreSQL are on the same server.

To set the permissions, use this paster command after you've set the database URLs (make sure to have your virtualenv activated)::

paster datastore set-permissions postgres
paster datastore set-permissions postgres -c /etc/ckan/masaq/development.ini

The ``postgres`` at the end of this command should be the name of a postgres
user with permission to create new tables and users, grant permissions, etc.
Expand Down
147 changes: 56 additions & 91 deletions doc/deployment.rst
View file
Expand Up @@ -19,18 +19,17 @@ deployment configurations including:
* Nginx_ with paster and reverse proxy
* Nginx_ with uwsgi

.. note:: below, we will only be able to give a few example of setups and many
other ones are possible.

.. _Apache: http://httpd.apache.org/
.. _Nginx: http://wiki.nginx.org/Main

Deploying CKAN on an Ubuntu Server using Apache and modwsgi
===========================================================

These instructions have been tested on Ubuntu 10.04 with CKAN 1.7.
Deploying CKAN on Ubuntu using Apache and modwsgi
=================================================

Once you've installed CKAN on your Ubuntu server by following the instructions
in :doc:`install-from-source`, you can follow these instructions to deploy your
site using Apache and modwsgi.

This is the standard way to deploy CKAN.

Install Apache and modwsgi
--------------------------
Expand All @@ -42,29 +41,35 @@ support to Apache)::

.. _modwsgi: https://code.google.com/p/modwsgi/

Install CKAN
------------

The following assumes you have installed to ``/usr/local/demo.ckan.net`` with your virtualenv at ``/usr/local/demo.ckan.net/pyenv``.
Create the ``production.ini`` File
----------------------------------

We'll continue with the ``http://masaq.ckanhosted.com`` example from
:doc:`install-from-source`. Create your site's ``production.ini`` file, by
copying the ``development.ini`` file you created in :doc:`install-from-source`
earlier::

cp /etc/ckan/masaq/development.ini /etc/ckan/masaq/production.ini


Create the WSGI Script File
---------------------------

Create the WSGI script file for your CKAN instance,
``/usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py``::
Create your site's WSGI script file
``/etc/ckan/masaq/apache.wsgi`` with the following contents::

import os
instance_dir = '/usr/local/demo.ckan.net'
config_file = '/usr/local/demo.ckan.net/pyenv/src/ckan/development.ini'
pyenv_bin_dir = os.path.join(instance_dir, 'pyenv', 'bin')
activate_this = os.path.join(pyenv_bin_dir, 'activate_this.py')
activate_this = os.path.join('/usr/lib/ckan/masaq/bin/activate_this.py')
execfile(activate_this, dict(__file__=activate_this))

from paste.deploy import loadapp
config_filepath = os.path.join(instance_dir, config_file)
config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'production.ini')
from paste.script.util.logging_config import fileConfig
fileConfig(config_filepath)
application = loadapp('config:%s' % config_filepath)


The modwsgi Apache module will redirect requests to your web server to this
WSGI script file. The script file then handles those requests by directing them
on to your CKAN instance (after first configuring the Python environment for
Expand All @@ -73,67 +78,54 @@ CKAN to run in).
Create the Apache Config File
-----------------------------

Create the Apache config file for your CKAN instance by copying the default
Apache config file:

cd /etc/apache2/sites-available
sudo cp default demo.ckan.net

Edit ``/etc/apache2/sites-available/demo.ckan.net``, before the last line
(``</VirtualHost>``) add these lines::

ServerName demo.ckan.net
ServerAlias demo.ckan.net
WSGIScriptAlias / /usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py

# pass authorization info on (needed for rest api)
WSGIPassAuthorization On
ErrorLog /var/log/apache2/demo.ckan.net.error.log
CustomLog /var/log/apache2/demo.ckan.net.custom.log combined

This tells the Apache modwsgi module to redirect any requests to the web server
to the CKAN WSGI script that you created above (``demo.ckan.net.py``). Your
WSGI script in turn directs the requests to your CKAN instance.
Create your site's Apache config file at
``/etc/apache2/sites-available/masaq``, with the following
contents::

<VirtualHost 0.0.0.0:8080>
ServerName masaq.ckanhosted.com
ServerAlias www.masaq.ckanhosted.com
WSGIScriptAlias / /etc/ckan/masaq/apache.wsgi

Create Directories for CKAN's Temporary Files
---------------------------------------------
# Pass authorization info on (needed for rest api).
WSGIPassAuthorization On

Make the data and sstore directories and give them the right permissions::
# Deploy as a daemon (avoids conflicts between CKAN instances).
WSGIDaemonProcess masaq display-name=masaq processes=2 threads=15

cd /usr/local/demo.ckan.net/pyenv/src/ckan/
mkdir data sstore
chmod g+w -R data sstore
sudo chgrp -R www-data data sstore
WSGIProcessGroup masaq

CKAN Log File
-------------
ErrorLog /var/log/apache2/masaq.error.log
CustomLog /var/log/apache2/masaq.custom.log combined
</VirtualHost>

Edit your CKAN config file (e.g.
``/usr/local/demo.ckan.net/pyenv/src/ckan/development.ini``), find this line::
This tells the Apache modwsgi module to redirect any requests to the web server
to the CKAN WSGI script that you created above (``masaq.py``).
Your WSGI script in turn directs the requests to your CKAN instance.

args = ("ckan.log", "a", 20000000, 9)

and change it to set the ckan.log file location to somewhere that CKAN can write to, e.g.::
Set the ``data`` and ``sstore`` Directory Permissions
-----------------------------------------------------

args = ("/var/log/ckan/demo.ckan.net/ckan.log", "a", 20000000, 9)
Make sure that Apache's user (``www-data`` on Ubuntu) has permission to write to
the site's ``data`` and ``sstore`` directories::

Then create that directory and give it the right permissions::
chmod g+w -R /etc/ckan/masaq/data /etc/ckan/masaq/sstore
sudo chgrp -R www-data /etc/ckan/masaq/data /etc/ckan/masaq/sstore

sudo mkdir -p /var/log/ckan/demo.ckan.net
sudo chown www-data /var/log/ckan/demo.ckan.net

Enable Your CKAN Site
---------------------

Finally, enable your CKAN site in Apache::

sudo a2ensite demo.ckan.net
sudo /etc/init.d/apache2 restart
sudo a2ensite masaq
sudo /etc/init.d/apache2 reload

You should now be able to visit your server in a web browser and see your new
CKAN instance.


Troubleshooting
---------------

Expand All @@ -144,19 +136,18 @@ If you see a default Apache welcome page where your CKAN front page should be,
it may be because the default Apache config file is overriding your CKAN config
file (both use port 80), so disable it and restart Apache::

$ sudo a2dissite default
$ sudo /etc/init.d/apache2 restart
sudo a2dissite default
sudo /etc/init.d/apache2 reload

403 Forbidden and 500 Internal Server Error
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you see a 403 Forbidden or 500 Internal Server Error page where your CKAN
front page should be, you may have a problem with your unix file permissions.
The Apache web server needs to have permission to access your WSGI script file
(e.g. ``/usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py``) ''and all of its
parent directories''. The permissions of the file should look like
``-rw-r--r--`` and the permissions of each of its parent directories should
look like ``drwxr-xr-x``.
and all of its parent directories''. The permissions of the file should look
like ``-rw-r--r--`` and the permissions of each of its parent directories
should look like ``drwxr-xr-x``.

IOError: sys.stdout access restricted by mod_wsgi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -184,8 +175,7 @@ Log Files
~~~~~~~~~

In general, if it's not working look in the log files in ``/var/log/apache2``
for error messages. ``demo.ckan.net.error.log`` should be particularly
interesting.
for error messages. Your ``*.error.log`` should be particularly interesting.

modwsgi wiki
~~~~~~~~~~~~
Expand All @@ -207,29 +197,4 @@ CKAN (since version 1.6) can run mounted at a 'sub-directory' URL, such as
http://mysite.com/data/. This is achieved by changing the WSGIScriptAlias first
parameter (in the Apache site config). e.g.::

WSGIScriptAlias /data /home/dread/etc/ckan-pylons.py

CORS
====

**As of CKAN v1.5 CORS is built in to CKAN so for CKAN >= 1.5 no modifications
to your webserver config are needed.**

CORS = Cross Origin Resource Sharing. It is away to allow browsers (and hence
javascript in browsers) make requests to domains other than the one the browser
is currently on.

In Apache you can enable CORS for you CKAN site by setting the following in
your config::

Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

# Respond to all OPTIONS requests with 200 OK
# This could be done in the webapp
# This is need for pre-flighted requests (POSTs/PUTs)
RewriteEngine On
RewriteCond %{REQUEST_METHOD} OPTIONS
RewriteRule ^(.*)$ $1 [R=200,L]

WSGIScriptAlias /data /etc/ckan/masaq/apache.wsgi

0 comments on commit 808391f

Please sign in to comment.