Skip to content
This repository has been archived by the owner on Jun 19, 2023. It is now read-only.


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Build Status Coverage Status

ckanext-dgu - extension

This is an extension to CKAN that provides customisations specifically for the project.

The official version is available at:

Contributions from 1st March 2012 are by the Cabinet Office. It is Crown Copyright and opened up under both the Open Government Licence (OGL) (which is compatible with Creative Commons Attibution License) and the GNU Affero General Public License (AGPL) v3.0. Before 1st March 2012, contributions are Copyright (c) 2010-2012 Open Knowledge Foundation. This material is open and licensed under the GNU Affero General Public License (AGPL) v3.0.

OGL terms: AGPL terms:

Related extensions that use in addition are: ckanext-archiver, ckanext-harvest, ckanext-spatial, ckanext-os, ckanext-qa, ckanext-social


This extension contains a number of elements, principally:

  • DGU-specific templates
    • Custom dataset edit form - includes a number of custom fields such as temporal_coverage and geographic_coverage.
    • Harvest Object inserted into the CKAN package view page.
  • DGU-customized search - different facets etc
  • Reports (integrated from ckanext-report)
  • Drupal integration - Drupal login, user details copied into CKAN, comments below datasets, provide APIs to Drupal for displaying recent datasets
  • Authorization customizations - display of users is tightened up
  • Feedback form (below datasets)
  • gov_daily - a script (for running daily) that save the database dumps for end-users (JSON/CSV) and backups (SQL).
  • various other command-line utilities



Based on PackageController but at /dataset/* rather than /package/* and adds the delete function. Proxy for getting Drupal comments (only for running in paster). NO FORM HERE



Middleware to log-in the user based on Drupal cookies and requests to Drupal.



Changes permissions:

  • hierarchy structure - edit package/group if is an editor for the group or an admin for the group or its parents.
  • All package creations/edits need an API key - no anonymous ones
  • UKLP packages can't be edited through the form or API - only by harvesting (unless sysadmin)
  • ONS packages can't be edited through the form or API - only by ONS loader (unless sysadmin)
  • Packages can only be deleted by sysadmin or UKLP packages by their editor/admin.
  • Users can only be viewed by the user and sysadmin
  • User list can only be viewed by sysadmin, editors, admins.



Sets 'ckan.auth.profile' to 'publisher' (and same for harvesting: 'ckan.harvest.auth.profile' = 'publisher'). Publisher controller, based on Group:

  • Publisher browse page includes hierarchy, alpha-browse and search.
  • Apply to be a publisher editor or admin.
  • Edit admins/editors for a publisher
  • Publisher read shows publisher hierarchy, search pane with results and pager
  • Display publisher's admins/editors for appropriate users.
  • Report pages - users not assigned to groups, groups without admins, publishers vs users, users When user is created, flashes "You can now apply for publisher access"


from ckanext.dgu.lib import helpers

Data, Tag and Reports Controllers. Templates, helper functions. Random extras:

  • Viewing user names (e.g. in History) change them for dept if not editor/admin.
  • Add 'Vary: Cookie' header to all responses.
from ckanext.dgu.search_indexing import SearchIndexing

Add fields to search index. Default sort-by. Escape SOLR characters. Search field weighting adjusted.



New group form, type 'publisher' with schema. Added fields: contact, foi contacts, category, abbreviation.



New dataset form. Lots of schema and validation customisation.



Util API for Drupal - latest-datasets (front page), dataset-count (front page), revisions (unused). Reports, starting with organisation_resources.


ckanext.dgu.plugin:ResourceModificationPlugin ckanext.dgu.plugin:ResourceURLModificationPlugin

When a resource is created/deleted/URL-changed, this updates the last_major_modification date of its package.

Non-plugin code

ckanext/dgu/ - mostly not used now ckanext/dgu/ - for getting user info from Drupal dgu/ckanext/dgu/bin/ - scripts used at one time or another dgu/ckanext/dgu/commands/ - scripts used at one time or another


This is how to install ckanext-dgu, ckan and their dependencies into a python virtual environment::

virtualenv pyenv
pip -E pyenv install -e git+
pip -E pyenv install -e git+
pip -E pyenv install -r pyenv/src/ckan/pip-requirements.txt
pip -E pyenv install -r pyenv/src/ckanext-dgu/pip-requirements.txt

There are plenty of other little details about getting CKAN running under Apache, SOLR config etc in the CKAN README.

When configuring Apache, DGU enables some modules SSL and for ckanext-os. This is the complete list:: sudo a2enmod proxy wsgi headers proxy_http rewrite ssl


Different parts of the DGU extension require options to be set in the CKAN configuration file (.ini) in the [app:main] section

To use the DGU package form specify::

ckan.plugins = dgu_form dgu_package_form

For the Drupal RPC connection (for user data etc.) supply the hostname, and credentials for HTTP Basic Authentication (if necessary)::

dgu.xmlrpc_domain =
dgu.xmlrpc_username = ckan
dgu.xmlrpc_password = letmein

The DGU-version of the SOLR schema is required instead of the CKAN SOLR schema. Whether you use a single or mult-core SOLR setup, you'll need a link to the DGU SOLR schema like this::

sudo ln -s /home/okfn/pyenv/src/ckanext-dgu/config/solr/schema-1.4-dgu.xml /etc/solr/conf/schema.xml

Initialise database

Creating the database is as usual for CKAN::

sudo -u postgres createuser -S -D -R -P $db_user
sudo -u postgres createdb -O $db_user $db

And because we usually use ckanext-dgu also with ckanext-spatial, then PostGIS needs setting up too::

sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql -v ON_ERROR_ROLLBACK=on
sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql
sudo -u postgres psql -d $db -c "ALTER TABLE spatial_ref_sys OWNER TO $db_user;"
sudo -u postgres psql -d $db -c "ALTER TABLE geometry_columns OWNER TO $db_user;"

(For more info on that, see the README for ckanext-spatial.)


There is a front-page added to CKAN which describes the Catalogue APIs. The usual CKAN front-page has been moved to /ckan/ .



There are a number of command-line scripts for processing data. Over the years they have been created in three different ways, but more recently we've been focussing on just creating stand-alone python scripts.

To run a python script (in the ckanext/dgu/bin directory) you activate the environment and then call it with python::

. pyenv/bin/activate
python ckanext/dgu/bin/ --help

To run a console script (listed in you should activate the environment first. For example::

. pyenv/bin/activate
bulk_delete --help

To run a paster command (listed in you activate the environment and if your CWD is not ckanext-dgu, specify it as the plugin::

. pyenv/bin/activate
paster --plugin=ckanext-dgu prod --help


The DGU theme uses assets (images, javascript, css) from this repo and the shared assets repo::

Both repos should be cloned next to each other on developer and server machines. If this is not possible then you need to set the dgu.shared_assets_timestamp_path config option to tell CKAN where the shared assets timestamp file is. e.g.::

dgu.shared_assets_timestamp_path = /vagrant/src/shared_dguk_assets/assets/timestamp

Assets are stored in the repo in 'source' form - the form easiest for developers to edit them in. Before they can be served, Grunt must be run on both repos to create the 'public' versions of these files. This does concatenation, minification, compilation of the less, and recording a timestamp (see Gruntfile.js for details).


images:     ckanext/dgu/theme/src/images
javascript: ckanext/dgu/theme/src/scripts
css (less): ckanext/dgu/theme/src/css

And when you run grunt, you get:

images:     ckanext/dgu/theme/public/images
javascript: ckanext/dgu/theme/public/scripts
css:        ckanext/dgu/theme/public/css
timestamp:  ckanext/dgu/theme/

Read more about Grunt installation and running it:

The shared assets need to be served at /assets. On a deployment server, setup nginx or apache to do this. A developer running under paster will find that the shared assets are served autoimatically, as long as the repo is cloned alongside ckanext-dgu and that this config option is not set: dgu.shared_assets_timestamp_path.


#####Unit and functional tests

To test the DGU extension you need the setup with CKAN (see above) and creation of a configured pyenv/src/ckan/development.ini (see ).

To run the tests::

cd {pyenv}/ckanext-dgu
nosetests --ckan ckanext/dgu/tests/

or run them from another directory by specifying the test.ini::

nosetests --ckan --with-pylons={pyenv}/src/ckanext-dgu/test.ini {pyenv}/src/ckanext-dgu/ckanext/dgu/tests/

You can either run the 'quick and dirty' tests with SQLite or more comprehensively with PostgreSQL. Set --with-pylons to point to the relevant configuration - either test.ini or test-core.ini (both from the ckanext-dgu repo, not the ckan one). For more information, see .

#####Browser tests

Selenium is used to test a site is operating to a basic minimum standard, and specific checks on javascript elements.

To run the Selenium tests, TODO


#####Address and Connection errors

  • socket.error: [Errno 98] Address already in use
  • error: [Errno 111] Connection refused

These errors usually means a previous run of the tests has not cleaned up the Mock Drupal process. You can verify that::

$ ps a | grep mock_drupal
4748 pts/8    S      0:00 /home/dread/hgroot/pyenv-dgu/bin/python /home/dread/hgroot/pyenv-dgu/bin/paster --plugin=ckanext-dgu mock_drupal run -q

Now kill it before running the tests again::

$ kill 4748

#####Config errors

  • DrupalXmlRpcSetupError: Drupal XMLRPC not configured.

The missing settings that result in this error are to be found in {pyenv}/src/ckanext-dgu/test-core.ini which is also imported into {pyenv}/src/ckanext-dgu/test.ini, so make sure you are specifying either of these config files in your nosetests --with-pylons parameter.

  • HTTP code=400, reason=ERROR: [f752f33380e3eec1379cfb89e0fdded8] multiple values encountered for non multiValued field parent_publishers: [london-borough-of-barnet, local-authorities]

This is due to SOLR using the CKAN SOLR schema, rather than the specific DGU one. Change it using the ln -s command above, followed by stopping and starting SOLR.


DGU is an extension for CKAN:

This README file is part of the DGU Developer Documentation, stored in the ckanext-dgu repo at ckanext-dgu/doc.

The Developer Docs can be built using Sphinx <>_::

  python build_sphinx