From c21020907dfbff4b5fce6e65f2a3aa5e41ee2026 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Mon, 19 Mar 2012 17:49:43 +0000 Subject: [PATCH 1/9] [#2226,doc][xs]: correct some links etc that were causing warnings. --- doc/authorization.rst | 2 +- doc/configuration.rst | 14 -------------- doc/domain-model.rst | 2 +- 3 files changed, 2 insertions(+), 16 deletions(-) diff --git a/doc/authorization.rst b/doc/authorization.rst index 66aba3fe84e..20b8d99ce29 100644 --- a/doc/authorization.rst +++ b/doc/authorization.rst @@ -25,7 +25,7 @@ In more detail, these concepts are as follows: * To simplify mapping users to actions and objects, actions are aggregated into a set of **roles**. For example, an editor role would automatically have edit and read actions. * Finally, CKAN has registered **users**. -Recent support for authorization profiles has been implemented using a publisher/group based profile that is described in :doc:`publisher_auth_profile` +Recent support for authorization profiles has been implemented using a publisher/group based profile that is described in :doc:`publisher-profile`. Objects +++++++ diff --git a/doc/configuration.rst b/doc/configuration.rst index f0c533c3973..bc62dab5668 100644 --- a/doc/configuration.rst +++ b/doc/configuration.rst @@ -596,20 +596,6 @@ With this example setting, visitors and logged-in users can only read datasets t Defaults: see in ``ckan/model/authz.py`` for: ``default_default_user_roles`` - -auth_profile -^^^^^^^^^^^^ - -This allows you to specify the auth profile to use for this installation. By default this is empty and uses the default authorisation code, if set to publisher it will use the publisher profile in ckan/logic/auth/publisher. See :doc:`publisher_auth_profile` for more information. - -Example:: - ckan.auth.profile = publisher - -With this example setting the publisher auth profile will be used. - -Defaults: The default authorisation from ``ckan/logic/auth/*`` will be used - - Plugin Settings --------------- diff --git a/doc/domain-model.rst b/doc/domain-model.rst index a834f2c492f..e99ed605622 100644 --- a/doc/domain-model.rst +++ b/doc/domain-model.rst @@ -40,5 +40,5 @@ Entity List Part of the domain model but not central: * Revision - changes to the domain model -* :doc:`task-status` - key/value information stored by CKAN Tasks +* :doc:`domain-model-task-status` - key/value information stored by CKAN Tasks From 4bf547e95a914452e12b8eff1e5b96adc9ed1653 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Mon, 19 Mar 2012 19:27:17 +0000 Subject: [PATCH 2/9] [#2226,doc/background-tasks][m]: add in documentation on background tasks. * Incorporates http://wiki.ckan.org/Writing_asynchronous_tasks (for dev section) * Moves over admin/setup section from extensions.rst --- doc/background-tasks.rst | 173 +++++++++++++++++++++++++++++++++++++++ doc/extensions.rst | 28 +------ doc/index.rst | 1 + 3 files changed, 176 insertions(+), 26 deletions(-) create mode 100644 doc/background-tasks.rst diff --git a/doc/background-tasks.rst b/doc/background-tasks.rst new file mode 100644 index 00000000000..9f626b0ec84 --- /dev/null +++ b/doc/background-tasks.rst @@ -0,0 +1,173 @@ +================ +Background Tasks +================ + +.. version-added: 1.5.1 + +CKAN allows you to create tasks that run in the 'background', that is +asynchronously and without blocking the main application (these tasks can also +be automatically retried in the case of transient failures). Such tasks can be +created in :doc:`Extensions ` or in core CKAN. + +Background tasks can be essential to providing certain kinds of functionality, +for example: + +* Creating webhooks that notify other services when certain changes occur (for + example a dataset is updated) +* Performing processing or validation or on data (as done by the Archiver and + DataStorer Extensions) + + +Enabling Background Tasks +========================= + +To manage and run background tasks requires a job queue and CKAN uses celery_ +(plus the CKAN database) for this purpose. Thus, to use background tasks you +need to install and run celery_. + +Installation of celery_ will normally be taken care of by whichever component +or extension utilizes it so we skip that here. + +To run the celery daemon you have two options: + +1. In development setup you can just use paster. This can be done as simply + as:: + + paster celeryd + + This only works if you have a ``development.ini`` file in ckan root. + +2. In production, the daemon should be run with a different ini file and be run + as an init script. The simplest way to do this is to install supervisor:: + + apt-get install supervisor + + Using this file as a template and copy to ``/etc/supservisor/conf.d``:: + + https://github.com/okfn/ckan/blob/master/ckan/config/celery-supervisor.conf + + Alternatively, you can run:: + + paster celeryd --config=/path/to/file.ini + + +Writing Background Tasks +========================== + +These instructions should show you how to write an background task and how to +call it from inside CKAN or another extension using celery. + +Examples +-------- + +Here are some existing real examples of writing CKAN tasks: + +* https://github.com/okfn/ckanext-archiver +* https://github.com/okfn/ckanext-qa +* https://github.com/okfn/ckanext-datastorer + +Setup +----- + +An entry point is required inside the ``setup.py`` for your extension, and so +you should add something resembling the following that points to a function in +a module. In this case the function is called task_imports in the +``ckanext.NAME.celery_import`` module:: + + entry_points = """ + [ckan.celery_task] + tasks = ckanext.NAME.celery_import:task_imports + """ + +The function, in this case ``task_imports`` should be a function that returns +fully qualified module paths to modules that contain the defined task (see the +next section). In this case we will put all of our tasks in a file called +``tasks.py`` and so ``task_imports`` should be in a file called +``ckanext/NAME/celery_import.py``:: + + def task_imports(): + return ['ckanext.NAME.tasks'] + +This returns an iterable of all of the places to look to find tasks, in this +example we are only putting them in one place. + + +Implementing the tasks +---------------------- + +The most straightforward way of defining tasks in our ``tasks.py`` module, is +to use the decorators provided by celery. These decorators make it easy to just +define a function and then give it a name and make it accessible to celery. +Make sure you import celery from ckan.lib.celery_app:: + + from ckan.lib.celery_app import celery + +Implement your function, specifying the arguments you wish it to take. For our +sample we will use a simple echo task that will print out its argument to the +console:: + + def echo( message ): + print message + +Next it is important to decorate your function with the celery task decorator. +You should give the task a name, which is used later on when calling the task:: + + @celery.task(name = "NAME.echofunction") + def echo( message ): + print message + +That's it, your function is ready to be run asynchronously outside of the main +execution of the CKAN app. Next you should make sure you run ``python setup.py +develop`` in your extensions folder and then go to your CKAN installation +folder (normally pyenv/src/ckan/) to run the following command:: + + paster celeryd + +Once you have done this your task name ``NAME.echofunction`` should appear in +the list of tasks loaded. If it is there then you are all set and ready to go. +If not then you should try the following to try and resolve the problem: + +1. Make sure the entry point is defined correctly in your ``setup.py`` and that + you have executed ``python setup.py develop`` +2. Check that your task_imports function returns an iterable with valid module + names in +3. Ensure that the decorator marks the functions (if there is more than one + decorator, make sure the celery.task is the first one - which means it will + execute last). +4. If none of the above helps, go into #ckan on irc.freenode.net where there + should be people who can help you resolve your issue. + +Calling the task +---------------- + +Now that the task is defined, and has been loaded by celery it is ready to be +called. To call a background task you need to know only the name of the task, +and the arguments that it expects as well as providing it a task id.:: + + import uuid + from ckan.lib.celery_app import celery + celery.send_task("NAME.echofunction", args=["Hello World"], task_id=str(uuid.uuid4())) + +After executing this code you should see the message printed in the console +where you ran ``paster celeryd``. + + +Retrying on errors +------------------ + +Should your task fail to complete because of a transient error, it is possible +to ask celery to retry the task, after some period of time. The default wait +before retrying is three minutes, but you can optionally specify this in the +call to retry via the countdown parameter, and you can also specify the +exception that triggered the failure. For our example the call to retry would +look like the following - note that it calls the function name, not the task +name given in the decorator:: + + try: + ... some work that may fail, http request? + except Exception, e: + # Retry again in 2 minutes + echo.retry(args=(message), exc=e, countdown=120, max_retries=10) + +If you don't want to wait a period of time you can use the eta datetime +parameter to specify an explicit time to run the task (i.e. 9AM tomorrow) diff --git a/doc/extensions.rst b/doc/extensions.rst index 3dd1f84b08a..c786121cf0d 100644 --- a/doc/extensions.rst +++ b/doc/extensions.rst @@ -84,33 +84,9 @@ Your extension should now be enabled. You can disable it at any time by removing it from the list of ckan.plugins in the config file. - Enabling an Extension with Background Tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Some extensions need to run tasks in the background. In order to do this we use celery as a job queue. -Examples of these types of extensions are: - -* `ckanext-webstorer `_: Put resources that are tabular into the webstore i.e give tabular data a restful api. -* `ckanext-archiver `_: Archives resources so that ckan holds a copy of them i.e caches them. - -The above steps needs to be followed for these, but also: - -3. The celery daemon needs to be started. This can be done as simply as:: - - paster celeryd - - This only works if you have a ``development.ini`` file in ckan root. - - In production the daemon should be run with a different ini file and be run as an init script. - The simplest way to do this is to install supervisor:: - - apt-get install supervisor - - Using this file as a template and copy to ``/etc/supservisor/conf.d``:: - - https://github.com/okfn/ckan/blob/master/ckan/config/celery-supervisor.conf - - Also you can run:: +Some extensions need to run tasks in the background. See +:doc:`background-tasks` for how to enable background tasks. - paster celeryd --config=/path/to/file.ini diff --git a/doc/index.rst b/doc/index.rst index 9fca3db10fb..2433466eac1 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -35,6 +35,7 @@ Customizing and Extending linked-data-and-rdf filestore datastore + background-tasks Publishing Datasets =================== From f75c5b72a33a217d922b0ea76e3b1585177f88ef Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Mon, 19 Mar 2012 19:50:56 +0000 Subject: [PATCH 3/9] [#2226,doc/deployment][s]: incorporate http://wiki.ckan.org/Deployment. --- doc/deployment.rst | 235 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 doc/deployment.rst diff --git a/doc/deployment.rst b/doc/deployment.rst new file mode 100644 index 00000000000..2ee41dd3eb9 --- /dev/null +++ b/doc/deployment.rst @@ -0,0 +1,235 @@ +=============== +CKAN Deployment +=============== + +.. note:: If you use the package installation method your site will already + have been deployed using the Apache and modwsgi route described + below. + +This document covers how to deploy CKAN in a production setup where it is +available on the Internet. This will usually involve connecting the CKAN web +application to a web server such as Apache_ or NGinx_. + +As CKAN uses WSGI, a standard interface between web servers and Python web +applications, CKAN can be used with a number of different web server and +deployment configurations including: + +* Apache_ with the modwsgi Apache module +* Apache_ with paster and reverse proxy +* Nginx_ with paster and reverse proxy +* Nginx_ with uwsgi + +.. note:: below, we will only be able to give a few example of setups and many + other ones are possible. + +.. _Apache: http://httpd.apache.org/ +.. _Nginx: http://wiki.nginx.org/Main + +Deploying CKAN on an Ubuntu Server using Apache and modwsgi +=========================================================== + +These instructions have been tested on Ubuntu 10.04 with CKAN 1.6.1. + +This is the standard way to deploy CKAN. + +Install Apache and modwsgi +-------------------------- + +Install Apache_ (a web server) and modwsgi_ (an Apache module that adds WSGI +support to Apache):: + + sudo aptitude install apache2 libapache2-mod-wsgi + +.. _modwsgi: https://code.google.com/p/modwsgi/ + +Install CKAN +------------ + +The following assumes you have installed to ``/usr/local/demo.ckan.net`` with your virtualenv at ``/usr/local/demo.ckan.net/pyenv``. + +Create the WSGI Script File +--------------------------- + +Create the WSGI script file for your CKAN instance, +``/usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py``:: + + import os + instance_dir = '/usr/local/demo.ckan.net' + config_file = '/usr/local/demo.ckan.net/pyenv/src/ckan/development.ini' + pyenv_bin_dir = os.path.join(instance_dir, 'pyenv', 'bin') + activate_this = os.path.join(pyenv_bin_dir, 'activate_this.py') + execfile(activate_this, dict(__file__=activate_this)) + from paste.deploy import loadapp + config_filepath = os.path.join(instance_dir, config_file) + from paste.script.util.logging_config import fileConfig + fileConfig(config_filepath) + application = loadapp('config:%s' % config_filepath) + +The modwsgi Apache module will redirect requests to your web server to this +WSGI script file. The script file then handles those requests by directing them +on to your CKAN instance (after first configuring the Python environment for +CKAN to run in). + +Create the Apache Config File +----------------------------- + +Create the Apache config file for your CKAN instance by copying the default +Apache config file: + + cd /etc/apache2/sites-available + sudo cp default demo.ckan.net + +Edit ``/etc/apache2/sites-available/demo.ckan.net``, before the last line +(````) add these lines:: + + ServerName demo.ckan.net + ServerAlias demo.ckan.net + WSGIScriptAlias / /usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py + + # pass authorization info on (needed for rest api) + WSGIPassAuthorization On + ErrorLog /var/log/apache2/demo.ckan.net.error.log + CustomLog /var/log/apache2/demo.ckan.net.custom.log combined + +This tells the Apache modwsgi module to redirect any requests to the web server +to the CKAN WSGI script that you created above (``demo.ckan.net.py``). Your +WSGI script in turn directs the requests to your CKAN instance. + + +Create Directories for CKAN's Temporary Files +--------------------------------------------- + +Make the data and sstore directories and give them the right permissions:: + + cd /usr/local/demo.ckan.net/pyenv/src/ckan/ + mkdir data sstore + chmod g+w -R data sstore + sudo chgrp -R www-data data sstore + +CKAN Log File +------------- + +Edit your CKAN config file (e.g. +``/usr/local/demo.ckan.net/pyenv/src/ckan/development.ini``), find this line:: + + args = ("ckan.log", "a", 20000000, 9) + +and change it to set the ckan.log file location to somewhere that CKAN can write to, e.g.:: + + args = ("/var/log/ckan/demo.ckan.net/ckan.log", "a", 20000000, 9) + +Then create that directory and give it the right permissions:: + + sudo mkdir -p /var/log/ckan/demo.ckan.net + sudo chown www-data /var/log/ckan/demo.ckan.net + +Enable Your CKAN Site +--------------------- + +Finally, enable your CKAN site in Apache:: + + sudo a2ensite demo.ckan.net + sudo /etc/init.d/apache2 restart + +You should now be able to visit your server in a web browser and see your new +CKAN instance. + +Troubleshooting +--------------- + +Default Apache Welcome Page +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you see a default Apache welcome page where your CKAN front page should be, +it may be because the default Apache config file is overriding your CKAN config +file (both use port 80), so disable it and restart Apache:: + + $ sudo a2dissite default + $ sudo /etc/init.d/apache2 restart + +403 Forbidden and 500 Internal Server Error +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you see a 403 Forbidden or 500 Internal Server Error page where your CKAN +front page should be, you may have a problem with your unix file permissions. +The Apache web server needs to have permission to access your WSGI script file +(e.g. ``/usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py``) ''and all of its +parent directories''. The permissions of the file should look like +``-rw-r--r--`` and the permissions of each of its parent directories should +look like ``drwxr-xr-x``. + +IOError: sys.stdout access restricted by mod_wsgi +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're getting 500 Internal Server Error pages and you see ``IOError: +sys.stdout access restricted by mod_wsgi`` in your log files, it means that +something in your WSGI application (e.g. your WSGI script file, your CKAN +instance, or one of your CKAN extensions) is trying to print to stdout, for +example by using standard Python ``print`` statements. WSGI applications are +not allowed to write to stdout. Possible solutions include: + +1. Remove the offending print statements. One option is to replace print + statements with statements like ``print >> sys.stderr, "..."`` + +2. Redirect all print statements to stderr:: + + import sys + sys.stdout = sys.stderr + +3. Allow your application to print to stdout by putting ``WSGIRestrictStdout Off`` in your Apache config file (not recommended). + +Also see https://code.google.com/p/modwsgi/wiki/ApplicationIssues + +Log Files +~~~~~~~~~ + +In general, if it's not working look in the log files in ``/var/log/apache2`` +for error messages. ``demo.ckan.net.error.log`` should be particularly +interesting. + +modwsgi wiki +~~~~~~~~~~~~ + +Some pages on the modwsgi wiki have some useful information for troubleshooting modwsgi problems: + +* https://code.google.com/p/modwsgi/wiki/ApplicationIssues +* http://code.google.com/p/modwsgi/wiki/DebuggingTechniques +* http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide +* http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines +* http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions +* http://code.google.com/p/modwsgi/wiki/ConfigurationIssues + + +Mounting CKAN at a non-root URL +=============================== + +CKAN (since version 1.6) can run mounted at a 'sub-directory' URL, such as +http://mysite.com/data/. This is achieved by changing the WSGIScriptAlias first +parameter (in the Apache site config). e.g.:: + + WSGIScriptAlias /data /home/dread/etc/ckan-pylons.py + +CORS +==== + +**As of CKAN v1.5 CORS is built in to CKAN so for CKAN >= 1.5 no modifications +to your webserver config are needed.** + +CORS = Cross Origin Resource Sharing. It is away to allow browsers (and hence +javascript in browsers) make requests to domains other than the one the browser +is currently on. + +In Apache you can enable CORS for you CKAN site by setting the following in +your config:: + + Header always set Access-Control-Allow-Origin "*" + Header always set Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" + Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type" + + # Respond to all OPTIONS requests with 200 OK + # This could be done in the webapp + # This is need for pre-flighted requests (POSTs/PUTs) + RewriteEngine On + RewriteCond %{REQUEST_METHOD} OPTIONS + RewriteRule ^(.*)$ $1 [R=200,L] + From 0d83cd877a1db85d05f5683d185ca374b7ae5c1d Mon Sep 17 00:00:00 2001 From: kindly Date: Mon, 19 Mar 2012 23:41:54 +0000 Subject: [PATCH 4/9] [2237] speed up tests and clean broken ones due to oredering --- ckan/lib/dictization/model_dictize.py | 2 +- .../functional/api/model/test_vocabulary.py | 67 +++++++++++++++---- ckan/tests/functional/test_admin.py | 4 +- ckan/tests/functional/test_home.py | 5 -- ckan/tests/functional/test_user.py | 10 ++- ckan/tests/lib/test_mailer.py | 6 ++ ckan/tests/misc/test_mock_mail_server.py | 6 ++ ckan/tests/mock_mail_server.py | 2 + ckan/tests/models/test_package.py | 3 + test-core.ini | 1 - 10 files changed, 82 insertions(+), 24 deletions(-) diff --git a/ckan/lib/dictization/model_dictize.py b/ckan/lib/dictization/model_dictize.py index ca8df610c03..1c766bd44d8 100644 --- a/ckan/lib/dictization/model_dictize.py +++ b/ckan/lib/dictization/model_dictize.py @@ -294,7 +294,7 @@ def user_list_dictize(obj_list, context, return sorted(result_list, key=sort_key, reverse=reverse) def member_dictize(member, context): - return table_dictize(member, context) + return d.table_dictize(member, context) def user_dictize(user, context): diff --git a/ckan/tests/functional/api/model/test_vocabulary.py b/ckan/tests/functional/api/model/test_vocabulary.py index 456fe17151b..0548811c216 100644 --- a/ckan/tests/functional/api/model/test_vocabulary.py +++ b/ckan/tests/functional/api/model/test_vocabulary.py @@ -2,27 +2,55 @@ from pylons.test import pylonsapp import paste.fixture from ckan.lib.helpers import json +import ckan.lib.dictization.model_dictize as model_dictize import sqlalchemy from nose.tools import raises, assert_raises class TestVocabulary(object): - def setup(self): + @classmethod + def setup_class(self): self.app = paste.fixture.TestApp(pylonsapp) - ckan.tests.CreateTestData.create() - self.sysadmin_user = ckan.model.User.get('testsysadmin') - self.normal_user = ckan.model.User.get('annafan') - # Make a couple of test vocabularies needed later. - self.genre_vocab = self._create_vocabulary(vocab_name="Genre", - user=self.sysadmin_user) - self.timeperiod_vocab = self._create_vocabulary( - vocab_name="Time Period", user=self.sysadmin_user) - self.composers_vocab = self._create_vocabulary(vocab_name="Composers", - user=self.sysadmin_user) - def teardown(self): + @classmethod + def teardown_class(self): ckan.model.repo.rebuild_db() + def setup(self): + self.clean_vocab() + model = ckan.model + context = {'model': model} + + genre = model.Vocabulary("Genre") + time_period = ckan.model.Vocabulary("Time Period") + composers = ckan.model.Vocabulary("Composers") + model.Session.add_all([genre, time_period, composers]) + + self.genre_vocab = model_dictize.vocabulary_dictize(genre, context) + self.timeperiod_vocab = model_dictize.vocabulary_dictize(time_period, context) + self.composers_vocab = model_dictize.vocabulary_dictize(composers, context) + ckan.model.Session.commit() + + self.sysadmin_user = ckan.model.User.get('admin') + self.normal_user = ckan.model.User.get('normal') + if not self.sysadmin_user: + normal_user = ckan.model.User(name=u'normal', password=u'annafan') + sysadmin_user = ckan.model.User(name=u'admin', password=u'testsysadmin') + ckan.model.Session.add(normal_user) + ckan.model.Session.add(sysadmin_user) + ckan.model.add_user_to_role(sysadmin_user, ckan.model.Role.ADMIN, ckan.model.System()) + ckan.model.Session.commit() + self.sysadmin_user = ckan.model.User.get('admin') + self.normal_user = ckan.model.User.get('normal') + + def clean_vocab(self): + ckan.model.Session.execute('delete from package_tag_revision') + ckan.model.Session.execute('delete from package_tag') + ckan.model.Session.execute('delete from tag') + ckan.model.Session.execute('delete from vocabulary') + ckan.model.Session.commit() + + @classmethod def _post(self, url, params=None, extra_environ=None): if params is None: params = {} @@ -32,6 +60,7 @@ def _post(self, url, params=None, extra_environ=None): assert not response.errors return response.json + @classmethod def _create_vocabulary(self, vocab_name=None, user=None): # Create a new vocabulary. params = {'name': vocab_name} @@ -506,6 +535,8 @@ def test_vocabulary_update_bad_tags(self): '''Test updating vocabularies with invalid tags. ''' + apikey = str(self.sysadmin_user.apikey) + for tags in ( [{'id': 'xxx'}, {'name': 'foo'}], [{'name': 'foo'}, {'name': None}], @@ -517,8 +548,7 @@ def test_vocabulary_update_bad_tags(self): params = {'id': self.genre_vocab['name'], 'tags': tags} response = self.app.post('/api/action/vocabulary_update', params=json.dumps(params), - extra_environ = {'Authorization': - str(self.sysadmin_user.apikey)}, + extra_environ = {'Authorization': apikey}, status=409) assert response.json['success'] == False assert response.json['error']['tags'] @@ -773,6 +803,9 @@ def test_add_vocab_tag_to_dataset(self): '''Test that a tag belonging to a vocab can be added to a dataset, retrieved from the dataset, and then removed from the dataset.''' + ckan.model.repo.rebuild_db() + self.setup() + ckan.tests.CreateTestData.create() # First add a tag to the vocab. vocab = self.genre_vocab tag = self._create_tag(self.sysadmin_user, 'noise', vocab) @@ -815,6 +848,9 @@ def test_add_vocab_tag_to_dataset(self): def test_delete_tag_from_vocab(self): '''Test that a tag can be deleted from a vocab.''' + ckan.model.repo.rebuild_db() + self.setup() + ckan.tests.CreateTestData.create() vocab = self.genre_vocab # First add some tags to the vocab. @@ -877,6 +913,9 @@ def test_delete_free_tag(self): automatically removed from datasets. ''' + ckan.model.repo.rebuild_db() + self.setup() + ckan.tests.CreateTestData.create() # Get a package from the API. package = (self._post('/api/action/package_show', {'id': self._post('/api/action/package_list')['result'][0]}) diff --git a/ckan/tests/functional/test_admin.py b/ckan/tests/functional/test_admin.py index 593d77ec47a..e15772f69c9 100644 --- a/ckan/tests/functional/test_admin.py +++ b/ckan/tests/functional/test_admin.py @@ -9,7 +9,7 @@ def setup_class(cls): @classmethod def teardown_class(self): - CreateTestData.delete() + model.repo.rebuild_db() #test that only sysadmins can access the /ckan-admin page def test_index(self): @@ -46,7 +46,7 @@ def setup_class(cls): @classmethod def teardown_class(self): - CreateTestData.delete() + model.repo.rebuild_db() def test_role_table(self): diff --git a/ckan/tests/functional/test_home.py b/ckan/tests/functional/test_home.py index 67d430f7fe7..f47893dbb48 100644 --- a/ckan/tests/functional/test_home.py +++ b/ckan/tests/functional/test_home.py @@ -41,11 +41,6 @@ def test_template_head_end(self): res = self.app.get(offset) assert 'ckan.template_head_end = ' - def test_template_head_end(self): - offset = url_for('home') - res = self.app.get(offset) - assert 'ckan.template_head_end = ' - def test_template_footer_end(self): offset = url_for('home') res = self.app.get(offset) diff --git a/ckan/tests/functional/test_user.py b/ckan/tests/functional/test_user.py index f7339c075bd..5ab1efcb599 100644 --- a/ckan/tests/functional/test_user.py +++ b/ckan/tests/functional/test_user.py @@ -1,5 +1,7 @@ from routes import url_for from nose.tools import assert_equal +from pylons import config +import hashlib from pprint import pprint from ckan.tests import search_related, CreateTestData @@ -12,7 +14,13 @@ class TestUserController(FunctionalTestCase, HtmlCheckMethods, PylonsTestCase, SmtpServerHarness): @classmethod - def setup_class(self): + def setup_class(cls): + smtp_server = config.get('test_smtp_server') + if smtp_server: + host, port = smtp_server.split(':') + port = int(port) + int(str(hashlib.md5(cls.__name__).hexdigest())[0], 16) + config['test_smtp_server'] = '%s:%s' % (host, port) + PylonsTestCase.setup_class() SmtpServerHarness.setup_class() CreateTestData.create() diff --git a/ckan/tests/lib/test_mailer.py b/ckan/tests/lib/test_mailer.py index bddef78d207..a8da49d6ba2 100644 --- a/ckan/tests/lib/test_mailer.py +++ b/ckan/tests/lib/test_mailer.py @@ -2,6 +2,7 @@ from nose.tools import assert_equal, assert_raises from pylons import config from email.mime.text import MIMEText +import hashlib from ckan import model from ckan.tests.pylons_controller import PylonsTestCase @@ -13,6 +14,11 @@ class TestMailer(SmtpServerHarness, PylonsTestCase): @classmethod def setup_class(cls): + smtp_server = config.get('test_smtp_server') + if smtp_server: + host, port = smtp_server.split(':') + port = int(port) + int(str(hashlib.md5(cls.__name__).hexdigest())[0], 16) + config['test_smtp_server'] = '%s:%s' % (host, port) CreateTestData.create_user(name='bob', email='bob@bob.net') CreateTestData.create_user(name='mary') #NB No email addr provided SmtpServerHarness.setup_class() diff --git a/ckan/tests/misc/test_mock_mail_server.py b/ckan/tests/misc/test_mock_mail_server.py index 6d254e0cf45..b410d81f928 100644 --- a/ckan/tests/misc/test_mock_mail_server.py +++ b/ckan/tests/misc/test_mock_mail_server.py @@ -2,6 +2,7 @@ from nose.tools import assert_equal from pylons import config from email.mime.text import MIMEText +import hashlib from ckan.tests.pylons_controller import PylonsTestCase from ckan.tests.mock_mail_server import SmtpServerHarness @@ -10,6 +11,11 @@ class TestMockMailServer(SmtpServerHarness, PylonsTestCase): @classmethod def setup_class(cls): + smtp_server = config.get('test_smtp_server') + if smtp_server: + host, port = smtp_server.split(':') + port = int(port) + int(str(hashlib.md5(cls.__name__).hexdigest())[0], 16) + config['test_smtp_server'] = '%s:%s' % (host, port) SmtpServerHarness.setup_class() PylonsTestCase.setup_class() diff --git a/ckan/tests/mock_mail_server.py b/ckan/tests/mock_mail_server.py index bf8350e76eb..9020cb41a9d 100644 --- a/ckan/tests/mock_mail_server.py +++ b/ckan/tests/mock_mail_server.py @@ -2,6 +2,7 @@ import asyncore import socket from smtpd import SMTPServer +import hashlib from pylons import config @@ -67,6 +68,7 @@ def setup_class(cls): host, port = smtp_server.split(':') else: host, port = smtp_server, 25 + cls.port = port cls.smtp_thread = MockSmtpServerThread(host, int(port)) cls.smtp_thread.start() diff --git a/ckan/tests/models/test_package.py b/ckan/tests/models/test_package.py index 61ee8674ebe..ffc49cb4709 100644 --- a/ckan/tests/models/test_package.py +++ b/ckan/tests/models/test_package.py @@ -479,6 +479,9 @@ class TestPackagePurge: @classmethod def setup_class(self): CreateTestData.create() + @classmethod + def teardown_class(self): + model.repo.rebuild_db() def test_purge(self): pkgs = model.Session.query(model.Package).all() for p in pkgs: diff --git a/test-core.ini b/test-core.ini index 48862cfd602..3e7fde4831f 100644 --- a/test-core.ini +++ b/test-core.ini @@ -24,7 +24,6 @@ ckan.site_title = CKAN ckan.site_logo = /images/ckan_logo_fullname_long.png ckan.site_description = package_form = standard -ckan.plugins = synchronous_search licenses_group_url = # pyamqplib or queue carrot_messaging_library = queue From c594ef2a3f791e24bb5a6976ddf0f6458410f3b0 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Tue, 20 Mar 2012 08:44:02 +0000 Subject: [PATCH 5/9] [#2226,doc][xs]: items missing from prenultimate commit f75c5b72a33a217d922b0ea76e3b1585177f88ef. --- doc/index.rst | 1 + doc/post-installation.rst | 5 +---- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/index.rst b/doc/index.rst index 2433466eac1..a5cc0befe02 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -19,6 +19,7 @@ Installation install-from-package install-from-source post-installation + deployment solr-setup Customizing and Extending diff --git a/doc/post-installation.rst b/doc/post-installation.rst index 63d2ad1b04e..5e391249cb2 100644 --- a/doc/post-installation.rst +++ b/doc/post-installation.rst @@ -56,8 +56,5 @@ You may want to deploy your CKAN instance at this point, to share with others. If you have installed CKAN from packages, then Apache and WSGI deployment scripts are already configured for you in standard locations. -If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see http://wiki.ckan.net/Deployment +If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see :doc:`deployment`. -CKAN has been successfully deployed by a variety of other methods including Apache reverse proxy + paster, nginx reverse proxy + paster, and nginx + uwsgi. - -You can now proceed to :doc:`theming`. From 858e23008855e900cfc0c96269880a8a4931b06b Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Tue, 20 Mar 2012 08:49:54 +0000 Subject: [PATCH 6/9] [master,controller/template][xs]: support urls which do not have .html in them (but template does) and only abort 404 on template not found as otherwise we can hide errors. --- ckan/controllers/template.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/ckan/controllers/template.py b/ckan/controllers/template.py index e789857ccfa..04468f7d68b 100644 --- a/ckan/controllers/template.py +++ b/ckan/controllers/template.py @@ -1,4 +1,5 @@ from ckan.lib.base import * +from genshi.template.loader import TemplateNotFound class TemplateController(BaseController): @@ -26,5 +27,12 @@ def view(self, url): """ try: return render(url) - except: - abort(404) + except TemplateNotFound: + if url.endswith('.html'): + abort(404) + url += '.html' + try: + return render(url) + except TemplateNotFound: + abort(404) + From 9965818e20f7532b37eb97a3b76c078263eb0dd1 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Tue, 20 Mar 2012 11:07:25 +0000 Subject: [PATCH 7/9] [#1559,master][xs]: add comment hooks into dataset and resource templates and recent comments hook into ckan-admin index page to utilize new ckanext-disqus setup (fixes #1559). --- ckan/templates/admin/index.html | 2 ++ ckan/templates/package/read.html | 2 ++ ckan/templates/package/resource_read.html | 2 ++ 3 files changed, 6 insertions(+) diff --git a/ckan/templates/admin/index.html b/ckan/templates/admin/index.html index 9f71864ddf1..c7f4b350b6b 100644 --- a/ckan/templates/admin/index.html +++ b/ckan/templates/admin/index.html @@ -16,6 +16,8 @@

Current Sysadmins

${h.linked_user(user)} + + diff --git a/ckan/templates/package/read.html b/ckan/templates/package/read.html index 3fcc1844f71..a9da991cb44 100644 --- a/ckan/templates/package/read.html +++ b/ckan/templates/package/read.html @@ -89,6 +89,8 @@

Related Datasets

+ + diff --git a/ckan/templates/package/resource_read.html b/ckan/templates/package/resource_read.html index 9cdfbb94705..9f1aa1aa440 100644 --- a/ckan/templates/package/resource_read.html +++ b/ckan/templates/package/resource_read.html @@ -164,6 +164,8 @@

Additional Information

+ + From 7835604b1afb96cefa7d68cc7d9eec337519ca58 Mon Sep 17 00:00:00 2001 From: John Glover Date: Tue, 20 Mar 2012 11:21:11 +0000 Subject: [PATCH 8/9] [2245][docs] updating 'writing extensions' doc, removing out-of-date text --- doc/writing-extensions.rst | 788 ++++--------------------------------- 1 file changed, 83 insertions(+), 705 deletions(-) diff --git a/doc/writing-extensions.rst b/doc/writing-extensions.rst index 88f89998e8f..5e3aeee6b46 100644 --- a/doc/writing-extensions.rst +++ b/doc/writing-extensions.rst @@ -6,13 +6,12 @@ If you want to extend CKAN core functionality, the best way to do so is by writi Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system. -To meet the need to customize CKAN efficiently, we have introduced the concepts of CKAN extensions, plugin -interfaces and workers. These work together to provide a simple mechanism to -extend core CKAN functionality. +To meet the need to customize CKAN efficiently, we have introduced the concepts of CKAN extensions and plugin +interfaces. These work together to provide a simple mechanism to extend core CKAN functionality. .. warning:: This is an advanced topic. We are working to make the most popular extensions more easily available as Debian packages. -.. note:: The terms **extension**, **plugin interface** and **worker** have very precise meanings: the use of the generic word **plugin** to describe any way in which CKAN might be extended is deprecated. +.. note:: The terms **extension** and **plugin interface** have very precise meanings: the use of the generic word **plugin** to describe any way in which CKAN might be extended is deprecated. .. contents :: @@ -25,20 +24,20 @@ package which means that they can be imported like this: :: $ python - >>> import ckanext.queue + >>> import ckanext.example -Individual CKAN *extensions* may implement one or more *plugin interfaces* or -*workers* to provide their functionality. You'll learn about these later on. +Individual CKAN *extensions* may implement one or more *plugin interfaces* +to provide their functionality. -Create Your Own Extension -~~~~~~~~~~~~~~~~~~~~~~~~~ +Creating CKAN Extensions +~~~~~~~~~~~~~~~~~~~~~~~~ All CKAN extensions must start with the name ``ckanext-``. You can create your own CKAN extension like this: :: - (pyenv)$ paster create -t ckanext ckanext-myname + (pyenv)$ paster create -t ckanext ckanext-myextension You'll get prompted to complete a number of variables which will be used in your dataset. You change these later by editing the generated ``setup.py`` file. Here's some example output: @@ -48,9 +47,9 @@ You'll get prompted to complete a number of variables which will be used in your ckan#ckanext CKAN extension project template Variables: - egg: ckanext_myname - package: ckanextmyname - project: ckanext-myname + egg: ckanext_myextension + package: ckanextmyextension + project: ckanext-myextension Enter version (Version (like 0.1)) ['']: 0.4 Enter description (One-line description of the package) ['']: Great extension package Enter author (Author name) ['']: James Gardner @@ -58,76 +57,57 @@ You'll get prompted to complete a number of variables which will be used in your Enter url (URL of homepage) ['']: http://jimmyg.org Enter license_name (License name) ['']: GPL Creating template ckanext - Creating directory ./ckanext-myname - Directory ./ckanext-myname exists + Creating directory ./ckanext-myextension + Directory ./ckanext-myextension exists Skipping hidden file pyenv/src/ckan/ckan/pastertemplates/template/.setup.py_tmpl.swp Recursing into ckanext - Creating ./ckanext-myname/ckanext/ + Creating ./ckanext-myextension/ckanext/ .svn/ does not exist; cannot add directory Recursing into +project+ - Creating ./ckanext-myname/ckanext/myname/ + Creating ./ckanext-myextension/ckanext/myextension/ .svn/ does not exist; cannot add directory - Copying __init__.py to ./ckanext-myname/ckanext/myname/__init__.py + Copying __init__.py to ./ckanext-myextension/ckanext/myextension/__init__.py .svn/ does not exist; cannot add file - Copying __init__.py to ./ckanext-myname/ckanext/__init__.py + Copying __init__.py to ./ckanext-myextension/ckanext/__init__.py .svn/ does not exist; cannot add file - Copying setup.py_tmpl to ./ckanext-myname/setup.py + Copying setup.py_tmpl to ./ckanext-myextension/setup.py .svn/ does not exist; cannot add file Running pyenv/bin/python setup.py egg_info -Once you've run this you should find your extension is already set up in your -virtual environment so you can import it: +Once you've run this, you should now install the extension in your virtual environment: :: + (pyenv)$ cd ckanext-myextension + (pyenv)$ python setup.py develop (pyenv)$ python Python 2.6.6 (r266:84292, Oct 6 2010, 16:19:55) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information. - >>> import ckanext.myname + >>> import ckanext.myextension >>> +.. note:: + Running ``python setup.py develop`` will add a ``.egg-link`` file to your python + site-packages directory (which is on your python path). + This allows your extension to be imported and used, with + any changes made to the extension source code showing up immediately without needing + to be reinstalled, which is very useful during development. + + To instead install a python package by copying all of the files to the site-packages directory + run ``python setup.py install``. + To build useful extensions you need to be able to "hook into" different parts of CKAN in order to extend its functionality. You do this using CKAN's plugin architecture. We'll look at this in the next section. -Testing Extensions -`````````````````` - -CKAN extensions ordinarily have their own ``test.ini`` that refers to the CKAN ``test.ini``, so you can run them in exactly the same way. For example:: - - cd ckanext-dgu - nosetests ckanext/dgu/tests --ckan - nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini - -To test your changes you'll need to use the ``paster serve`` command from the ``ckan`` directory: - -:: - - cd /home/ubuntu/pyenv/src/ckan - . ../../bin/activate - paster make-config ckan development.ini - -Then make any changes to the ``development.ini`` file that you need before continuing: - -:: - - paster db upgrade - paster serve --reload - -You should also make sure that your CKAN installation passes the developer tests, as described in :doc:`test`. -Finally, if you write a CKAN, extension you may well want to publish it so others can use it too. See the `Publishing your extension`_ section below for details. - -Plugins -------- +Plugins: An Overview +-------------------- Plugin interfaces provide a specification which extensions can implement in order to "hook into" core CKAN functionality. -Summary -~~~~~~~ - The CKAN plugin implementation is based on the PyUtilib_ component architecture (PCA). Here's a quick summary, we'll go through all this in much more detail in a minute: @@ -150,263 +130,52 @@ a minute: Here's a list of some of the more commonly used plugin interfaces: -``IMapper`` +``IDatasetForm`` + Provide a custom dataset form and schema. - Listens and react to every database change +``IMapper`` + Listens and react to every database change. ``IRoutes`` and ``IController`` - - Provide an implementation to handle a particular URL + Provide an implementation to handle a particular URL. ``IGenshiStreamFilter`` - - Intercept template rendering to modify the output - -``ISession`` + Intercept template rendering to modify the output. ``IDomainObjectModification`` + Listens for changes to CKAN domain objects. ``IGroupController`` - Plugins for in the groups controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete. ``IConfigurable`` - Pass configuration to plugins and extensions + Pass configuration to plugins and extensions. ``IAuthorizer`` - Allows customisation of the default Authorization behaviour + Allows customisation of the default Authorization behaviour. If you look in `ckan/plugins/interfaces.py `_ you -will see the latest plugin interfaces. Alternativlly see the `Plugin API -documentation`_ below. - -.. note :: - - The existing 'IRoutesExtension', 'IMapperExtension' and 'ISessionExtension' - should be renamed in the code to not have the word 'Extension' in their names. - -An Example -~~~~~~~~~~ - -Plugin interfaces are basically just Python classes where each method is a hook -which allows a plugin that uses the interface to be notified when it is called. - -As an example, let's look at a plugin which gets configuration options from a -config file and is called each time a template is rendered in order to add some -HTML to the page. - -.. tip :: - - This example is based on real code used to implement the ``ckanext-disqus`` plugin - to add commenting to datasets. You can see the latest version of this code at - https://github.com/okfn/ckanext-disqus/blob/master/ckanext/plugins/disqus/__init__.py. - -First we set up logging and some helpers we'll need from Genshi to transfer the stream: - -:: - - import logging - log = logging.getLogger(__name__) - - import html - from genshi.core import TEXT - from genshi.input import HTML - from genshi.filters import Transformer - -Then we import the CKAN plugin code: - -:: - - from ckan.plugins.core import SingletonPlugin, implements - from ckan.plugins.interfaces import IConfigurable, IGenshiStreamFilter - -In this case we are implementing both the ``IConfigurable`` and -``IGenshiStreamFilter`` plugin interfaces in our plugin class. The -``IConfigurable`` plugin interface defines a ``configure()`` method which will -be is called on out plugin to let it know about configuration options. The -``IGenshiStreamFilter`` plugin interface defines a ``filter()`` method which -will be called on the plugin to give it the opportunity to change the template -before the HTML is returned to the browser. - -Let's have a look at the code: - -:: - - class Disqus(SingletonPlugin): - """ - Insert javascript fragments into dataset pages and the home page to - allow users to view and create comments on any dataset. - """ - - implements(IConfigurable) - implements(IGenshiStreamFilter) - - def configure(self, config): - """ - Called upon CKAN setup, will pass current configuration dict - to the plugin to read custom options. - """ - self.disqus_name = config.get('disqus.name', None) - if self.disqus_name is None: - log.warn("No disqus forum name is set. Please set \ - 'disqus.name' in your .ini!") - self.disqus_name = 'ckan' - - def filter(self, stream): - """ - Required to implement IGenshiStreamFilter; will apply some HTML - transformations to the page currently rendered. Depends on Pylons - global objects, how can this be fixed without obscuring the - inteface? - """ - - from pylons import request, tmpl_context as c - routes = request.environ.get('pylons.routes_dict') - - if routes.get('controller') == 'package' and \ - routes.get('action') == 'read' and c.pkg.id: - data = {'name': self.disqus_name, - 'identifier': 'pkg-' + c.pkg.id} - stream = stream | Transformer('body')\ - .append(HTML(html.BOTTOM_CODE % data)) - stream = stream | Transformer('body//div[@id="comments"]')\ - .append(HTML(html.COMMENT_CODE % data)) - - if routes.get('controller') == 'home' and \ - routes.get('action') == 'index': - data = {'name': self.disqus_name} - stream = stream | Transformer('body//\ - div[@id="main"]//ul[@class="xoxo"]')\ - .append(HTML(html.LATEST_CODE % data)) - - return stream - -Notice that the ``Disqus`` class explicitly states that it implements ``IConfigurable`` -and ``IGenshiStreamFilter`` with these two lines: - -:: - - implements(IConfigurable) - implements(IGenshiStreamFilter) - -Also notice that ``Disqus`` inherits from ``SingletonPlugin``. This means that -only one instance of the plugin is needed to provide the service. There is also -a ``Plugin`` class for occasions where you need multiple instances. - -.. autoclass:: ckan.plugins.core.Plugin - -.. autoclass:: ckan.plugins.core.SingletonPlugin - -By carefully choosing the plugin interfaces your plugin uses you can hook into -lots of parts of CKAN. Later on you'll see how to write your own plugin -interfaces to define your own "hooks". Before we can use the ``Disqus`` plugin -there is one more thing to do: add it to the extension and set an *entry point*. - -Setting the Entry Point -~~~~~~~~~~~~~~~~~~~~~~~ - -Imagine the code above was saved into a file named ``disqus.py`` in the -``ckanext-myname/ckanext/myname`` directory of the extension that was created earlier by the -``paster create -t ckanext ckanext-myname`` command. - -At this point CKAN still doesn't know where to find your plugin, even though -the module is installed. To find the plugin it looks up an *entry point*. An -entry point is just a feature of setuptools that links a string in the form -``package_name.entry_point_name`` to a particular object in Python code. - -.. tip :: - - If you are interested in reading a tutorial about entry points see: - - * http://reinout.vanrees.org/weblog/2010/01/06/zest-releaser-entry-points.html - * http://jimmyg.org/blog/2010/python-setuptools-egg-plugins.html - - -CKAN finds plugins by searching for entry points in the group ``ckan.plugin``. - -Entry points are defined in a package's ``setup.py`` file. If you look in the -``setup.py`` file for the ``ckanext-myname`` extension you'll see these -lines commented out towards the end: - -:: - - entry_points=\ - """ - [ckan.plugins] - # Add plugins here, eg - # myplugin=ckanext.myname:PluginClass - """, - -The entry point will be called without any parameters and must return an -instance of ``ckan.plugins.Plugin``. - -To enable the ``Disqus`` plugin uncomment the bottom line and change it to this: - -:: - - disqus_example=ckanext.myname:Disqus - -Any time you change the ``setup.py`` file you will need to run one of these two -commands again before the change will take effect: - -:: - - python setup.py develop - python setup.py egg_info - -With your entry point in place and installed you can now add the extension to -your CKAN config as described earlier. To add our example Disqus plugin you -would change your ``~/var/srvc/ckan.net/ckan.net.ini`` config file like this: - -:: - - [app:main] - ckan.plugins = disqus_example - -Note that the name of the plugin implementation that you give to -``ckan.plugins`` is always the same as the name of the entry point you've -defined in the ``setup.py`` file. It is therefore important that you don't -choose an entry point name that is being used by in any existing extension to -refer to its plugins. - -The same extension can have multiple different plugins, all implementing -different interfaces to provide useful functionality. For each new plugin your -extension implements you need to add another entry point (remembering to re-run -``python setup.py develop`` if needed). Your users will then need to add the -new entry point name to their ``ckan.plugins`` config option too. +will see the latest plugin interfaces. +Alternatively see the `Plugin API documentation`_ below. -Writing a Database Plugin -~~~~~~~~~~~~~~~~~~~~~~~~~ +.. .. note:: +.. The existing 'IRoutesExtension', 'IMapperExtension' and 'ISessionExtension' +.. should be renamed in the code to not have the word 'Extension' in their names. -You've seen how to use ``IConfigurable`` and ``IGenshiStreamFilter``. Here's -another example which implements ``IMapperExtension`` to log messages after any -record is inserted into the database. -:: - - from logging import getLogger - from ckan.plugins import implements, SingletonPlugin - from ckan.plugins import IMapperExtension - - log = getLogger(__name__) - - class InsertLoggerPlugin(SingletonPlugin): - """ - Emit a log line when objects are inserted into the database - """ - - implements(IMapperExtension, inherit=True) +Example CKAN Extension +---------------------- - def after_insert(mapper, connection, instance): - log.info('Object %r was inserted', instance) +A example CKAN extension can be found at http://github.com/okfn/ckanext-example. +Have a look at the README file for installation instructions. -Publishing Your Extension -~~~~~~~~~~~~~~~~~~~~~~~~~ +Publishing Extensions +--------------------- At this point you might want to share your extension with the public. @@ -427,12 +196,12 @@ the Python Package Index: You'll then see your extension at http://pypi.python.org/pypi. Others will be able to install your plugin with ``pip``. -Finally, please also add a summary of your extension and its entry points to the Extensions page on -http://wiki.ckan.net. +Finally, please also add a summary of your extension and its entry points to the Extensions page at +http://wiki.ckan.org/Extensions. Writing a Plugin Interface -~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------------------------- This describes how to add a plugin interface to make core CKAN code pluggable. @@ -475,6 +244,29 @@ plugins. However, be aware that pyutilib uses slightly different terminology. It calls ``PluginImplementations`` ``ExtensionPoint`` and it calls instances of a plugin object a *service*. + +Testing +------- + + +Testing CKAN Extensions +~~~~~~~~~~~~~~~~~~~~~~~ + +CKAN extensions ordinarily have their own ``test.ini`` that refers to the CKAN ``test.ini``, so you can run them in exactly the same way. For example:: + + cd ckanext-dgu + nosetests ckanext/dgu/tests --ckan + nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini + +To test your changes you'll need to use the ``paster serve`` command from the ``ckan`` directory: + +:: + + paster serve --reload -c + +You should also make sure that your CKAN installation passes the developer tests, as described in :doc:`test`. + + Testing Plugins ~~~~~~~~~~~~~~~ @@ -516,8 +308,9 @@ Here is an example test set-up:: At this point you should be able to write your own plugins and extensions together with their tests. + Ordering of Extensions -~~~~~~~~~~~~~~~~~~~~~~ +---------------------- .. caution :: @@ -546,418 +339,3 @@ Plugin API Documentation .. automodule:: ckan.plugins.interfaces :members: - -.. _disqus: http://disqus.com/ -.. _pyutilib: https://software.sandia.gov/trac/pyutilib -.. _deliverance: http://pypi.python.org/pypi/Deliverance -.. _RabbitMQ: http://www.rabbitmq.com/ - - -The Queue Extension -------------------- - -.. warning:: The queue extension is currently under development. These docs may not work for you. - - -Certain tasks that CKAN performs lend themselves to the use of a queue system. -Queue systems can be very simple. At their heart is the idea that you have two -separate processes, a *publisher* and a *consumer*. The publisher publishes a -message of some description to the queue and at another time, the consumer -takes that message off the queue and processes it. - -By writing code that puts things on the queue and then writing workers to take -things off, you can build lots of useful functionality. At the moment we are -writing facilities to check for broken links and harvest documents from geodata -servers. - -To use the queue in CKAN you need the ``ckanext-queue`` package. To install the -latest version of ``ckanext-queue`` in editable more so that you can look at -the source code run: - -:: - - pip install -e git+http://github.com/okfn/ckanext-queue#egg=ckanext-queue - -You will then see the source code in ``/pyenv/src/ckanext-queue/ckanext/queue`` -and ``README`` file in ``/pyenv/src/ckanext-queue/README.md``. - -Installing ``ckanext-queue`` will also install a ``worker`` command you will -use in a minute to run workers against the queue. - -Internally the queue extension uses the ``carrot`` library so that we could -potentially use different queue backends at some point in the future. For the -moment only the AMQP backend is supported so let's install an AMQP server -called RabbitMQ. - -Installing RabbitMQ -~~~~~~~~~~~~~~~~~~~ - -CentOS -`````` - -First you need to install Erlang. To do that first install its dependencies: - -:: - - yum install ncurses-devel flex.x86_64 m4.x86_64 openssl-devel.x86_64 unixODBC-devel.x86_64 - - -Install Erlang like this: - -:: - - - wget http://www.erlang.org/download/otp_src_R14B.tar.gz - tar zxfv otp_src_R14B.tar.gz - cd otp_src_R14B - LANG=C; export LANG - ./configure --prefix=/opt/erlang_R14B - make - make install - -Next download and install RabbitMQ: - -:: - - wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.2.0/rabbitmq-server-2.2.0-1.noarch.rpm - rpm -Uhv --no-deps rabbitmq-server-2.2.0-1.noarch.rpm - -Finally edit the ``/etc/init.d/rabbitmq-server`` script so that it uses the correct path for your Erlang install. Change this line - -:: - - PATH=/sbin:/usr/sbin:/bin:/usr/bin - -to: - -:: - - PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/erlang_R14B/bin:/opt/erlang_R14B/lib - -You can start it like this: - -:: - - /etc/init.d/rabbitmq-server start - -Ubuntu -`````` - -Just run: - -:: - - sudo apt-get install rabbitmq-server - -Working Directly with Carrot -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -As you learned earlier, CKAN uses carrot with the ``pyamqplib`` backend. Carrot is well-documented at this URL: - -http://ask.github.com/carrot/introduction.html - -Before you learn how to use the tools that ``ckanext-queue`` uses to work with -the queue, it is instructive to see a simple example that uses carrot directly. - -Save this as ``publisher.py``: - -:: - - from carrot.connection import BrokerConnection - conn = BrokerConnection( - hostname="localhost", - port=5672, - userid="guest", - password="guest", - virtual_host="/", - ) - - from carrot.messaging import Publisher - publisher = Publisher( - connection=conn, - exchange='local', - routing_key='*', - ) - publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}) - publisher.close() - -Now save this as ``consumer.py``: - -:: - - from carrot.connection import BrokerConnection - conn = BrokerConnection( - hostname="localhost", - port=5672, - userid="guest", - password="guest", - virtual_host="/", - ) - - from carrot.messaging import Consumer - consumer = Consumer( - connection=conn, - queue='local.workerchain', - exchange='local', - routing_key='*', - ) - def import_feed_callback(message_data, message): - feed_url = message_data["import_feed"] - print("Got message for: %s" % feed_url) - # something importing this feed url - # import_feed(feed_url) - message.ack() - consumer.register_callback(import_feed_callback) - # Go into the consumer loop. - consumer.wait() - -You'll notice that both examples set up a connection to the same AMQP server -with the same settings, in this case running on localhost. These also happen to -be the settings that (at the time of writing) ``ckanext-queue`` uses by default -if you don't specify other configuration settings. - -Make sure you have ``ckanext-queue`` installed (so that carrot and its -dependencies are installed too) then start the consumer: - -:: - - python consumer.py - -In a different console run the publisher: - -:: - - python publisher.py - -The publisher will quickly exit but if you now switch back to the consumer -you'll see the message was sent to the queue and the consumer recieved it -printing this message: - -:: - - Got message for: http://cnn.com/rss/edition.rss - - -Working with CKANext Queue -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Rather than working with carrot publishers and consumers directly, -``ckanext-queue`` provides two useful Python objects to help you: - -``ckanext.queue.connection.get_publisher(config)`` - - This returns a ``Publisher`` instance which has a ``send()`` method for adding an item to the queue. - - The ``config`` object is the same as ``pylons.config``. If you are writing - a standalone script, you can obtain a config object from a config file with - code similar to this, adjusting the ``relative_to`` option as necessary: - - :: - - from paste.deploy import appconfig - config = appconfig('config:development.ini', relative_to='pyenv/src/ckan') - -``ckanext.queue.worker.Worker`` - - This is a base class which you can inherit from. You can override its - ``consume()`` method to asynchronously pull items from the queue to do useful - things - - -.. note :: - - To use the queue extension you don't need to implement any new plugin - interfaces, you just need to use the ``get_publisher(config).send()`` method and the - ``Worker`` class. Of course your own extension might use plugins to hook into - other parts of CKAN to get information to put or retrieve from the queue. - -The worker implementation runs outside the CKAN server process, interacting -directly with both the AMQP queue and the CKAN API (to get CKAN data). The -``Worker`` class therefore subclasses both the ``carrot`` ``Consumer`` class -and the ``ckanclient`` ``CkanClient`` class so that your workers can make calls -to the running CKAN server via its API. - -Writing a Publisher -``````````````````` - -Here's a simple publisher. Save it as ``publish_on_queue.py``: - -:: - - from ckanext.queue import connection - from paste.deploy import appconfig - - import logging - logging.basicConfig(level=logging.DEBUG) - - config = appconfig('config:ckan.ini', relative_to='.') - publisher = connection.get_publisher(config) - publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}) - publisher.close() - print "Sent!" - -Note that this requires a ``ckan.ini`` file relative to the current working -directory to run. Here's what a sample file will look like: - -:: - - [app:main] - use = egg:ckan - ckan.site_id = local - queue.port = 5672 - queue.user_id = guest - queue.password = guest - queue.hostnane = localhost - queue.virtual_host = / - -The idea here is that publishers you write will be able to use the same -settings as CKAN itself would. In the next section you'll see how these same -options and more to a standard CKAN install config file to enable CKAN to use -the RabbitMQ queue. - -With the ``consumer.py`` script still running, execute the new script: - -:: - - python publish_on_queue.py - -You'll see that once again the consumer picks up the message. - -Writing a Worker -```````````````` - -Now let's replace the consumer with a worker. - -Each worker should have its own config file. There is an example you can use -named ``worker.cfg`` in the ``ckanext-queue`` source code. If you don't specify -a config file, the defaults will be used. - -.. tip :: - - Since the ``worker.cfg`` and CKAN configuration file are both in INI file - format you can also set these variables directly in your CKAN config file and - point the worker directly to your CKAN config instead. It will just ignore the - CKAN options. - -In particular it is worth setting ``queue.name`` which will be used internally -by RabbitMQ. - -Here's a suitable configuration for a worker: - -:: - - [worker] - ckan.site_id = local_test - ckan.site_url = http://localhost:5000 - ckan.api_key = XXX - - # queue.name = - # queue.routing_key = * - - # queue.port = - # queue.user_id = - # queue.password = - # queue.hostname = - # queue.virtual_host = - - -You can run it like this: - -:: - - worker echo -d -c worker.cfg - -The echo example comes from ``ckanext.queue.echo:EchoWorker``. It looks like this: - -:: - - from worker import Worker - from pprint import pprint - - class EchoWorker(Worker): - - def consume(self, routing_key, operation, payload): - print "Route %s, op %s" % (routing_key, operation) - pprint(payload) - -The ``EchoWorker`` has an entry point registered in ``ckanext-queue``'s ``setup.py`` so that the ``worker`` -script in ``pyenv/bin/worker`` can find it:: - - [ckan.workers] - echo = ckanext.queue.echo:EchoWorker - -When you run the ``worker`` command with the ``echo`` worker it looks up this -entry point to run the correct code. - -With the worker still running, try to run the publisher again: - -:: - - python publish_on_queue.py - -Once again the message will be output on the command line, this time via the worker. - -Internally the ``worker`` script you just used uses the -``ckanext.queue.worker.WorkerChain`` class to run all workers you specify on -the command line. You can get a list of all available workers by executing -``worker`` with no arguments. - -:: - - # worker - WARNING:root:No config file specified, using worker.cfg - ERROR:ckanext.queue.worker:No workers. Aborting. - -Configuring CKAN to use the queue -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once you have installed RabbitMQ and have it running you need to enable the -CKAN queue functionality within CKAN by adding this to your CKAN config file - -:: - - ckan.plugins = queue - -You don't need to specify configuration options to connect to RabbitMQ because -the defaults are fine. - -At this point if you edit a dataset it should be using the queue. If you have -the echo worker running you'll see the message added to the queue. - -Logging -``````` - -When using the queue with CKAN it is also useful to have logging set up. - -To get logging working you need to modify your CKAN config file to also include -the ``queue`` logger. Here's an example: - -:: - - [loggers] - keys = root, queue - - [handlers] - keys = console - - [formatters] - keys = generic - - [logger_root] - level = INFO - handlers = console - - [logger_queue] - level = DEBUG - handlers = console - qualname = ckanext - -You will also need to set this in your CKAN configuration and ensure any -workers and producers also set their ``ckan.site_id`` to the same value. - -:: - - ckan.site_id = local_test - -Now that you know about extensions, plugins and workers you should be able to -extend CKAN in lots of new and interesting ways. From 4dae5b4250246a722357cf48a5c8d642560f9c4a Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Tue, 20 Mar 2012 11:21:17 +0000 Subject: [PATCH 9/9] [#2226,doc/commenting][s]: add documentation re comments and commenting functionality in CKAN. --- doc/commenting.rst | 23 +++++++++++++++++++++++ doc/index.rst | 1 + 2 files changed, 24 insertions(+) create mode 100644 doc/commenting.rst diff --git a/doc/commenting.rst b/doc/commenting.rst new file mode 100644 index 00000000000..5db444f19a9 --- /dev/null +++ b/doc/commenting.rst @@ -0,0 +1,23 @@ +======================= +Comments and Commenting +======================= + +In a default CKAN install commenting is disabled. To enable it you have to +install and enable the disqus (commenting) extension: + +https://github.com/okfn/ckanext-disqus/ + +Please read and follow the instructions there (as well as the standard +:doc:`extensions` documentation). + +Important Note +-------------- + +Once installed and enabled, the presence of comments on a given page is +configured by your theme (see the documentation in the disqus extension for +details). + +In the default CKAN theme (as of v1.6.1), comments will be shown only on dataset +and resource pages (and recent comments only on the +:doc:`administrative-dashboard`). + diff --git a/doc/index.rst b/doc/index.rst index a5cc0befe02..408000f2a3b 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -29,6 +29,7 @@ Customizing and Extending :maxdepth: 2 theming + commenting extensions writing-extensions forms