From c21020907dfbff4b5fce6e65f2a3aa5e41ee2026 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Mon, 19 Mar 2012 17:49:43 +0000 Subject: [PATCH 1/6] [#2226,doc][xs]: correct some links etc that were causing warnings. --- doc/authorization.rst | 2 +- doc/configuration.rst | 14 -------------- doc/domain-model.rst | 2 +- 3 files changed, 2 insertions(+), 16 deletions(-) diff --git a/doc/authorization.rst b/doc/authorization.rst index 66aba3fe84e..20b8d99ce29 100644 --- a/doc/authorization.rst +++ b/doc/authorization.rst @@ -25,7 +25,7 @@ In more detail, these concepts are as follows: * To simplify mapping users to actions and objects, actions are aggregated into a set of **roles**. For example, an editor role would automatically have edit and read actions. * Finally, CKAN has registered **users**. -Recent support for authorization profiles has been implemented using a publisher/group based profile that is described in :doc:`publisher_auth_profile` +Recent support for authorization profiles has been implemented using a publisher/group based profile that is described in :doc:`publisher-profile`. Objects +++++++ diff --git a/doc/configuration.rst b/doc/configuration.rst index f0c533c3973..bc62dab5668 100644 --- a/doc/configuration.rst +++ b/doc/configuration.rst @@ -596,20 +596,6 @@ With this example setting, visitors and logged-in users can only read datasets t Defaults: see in ``ckan/model/authz.py`` for: ``default_default_user_roles`` - -auth_profile -^^^^^^^^^^^^ - -This allows you to specify the auth profile to use for this installation. By default this is empty and uses the default authorisation code, if set to publisher it will use the publisher profile in ckan/logic/auth/publisher. See :doc:`publisher_auth_profile` for more information. - -Example:: - ckan.auth.profile = publisher - -With this example setting the publisher auth profile will be used. - -Defaults: The default authorisation from ``ckan/logic/auth/*`` will be used - - Plugin Settings --------------- diff --git a/doc/domain-model.rst b/doc/domain-model.rst index a834f2c492f..e99ed605622 100644 --- a/doc/domain-model.rst +++ b/doc/domain-model.rst @@ -40,5 +40,5 @@ Entity List Part of the domain model but not central: * Revision - changes to the domain model -* :doc:`task-status` - key/value information stored by CKAN Tasks +* :doc:`domain-model-task-status` - key/value information stored by CKAN Tasks From 4bf547e95a914452e12b8eff1e5b96adc9ed1653 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Mon, 19 Mar 2012 19:27:17 +0000 Subject: [PATCH 2/6] [#2226,doc/background-tasks][m]: add in documentation on background tasks. * Incorporates http://wiki.ckan.org/Writing_asynchronous_tasks (for dev section) * Moves over admin/setup section from extensions.rst --- doc/background-tasks.rst | 173 +++++++++++++++++++++++++++++++++++++++ doc/extensions.rst | 28 +------ doc/index.rst | 1 + 3 files changed, 176 insertions(+), 26 deletions(-) create mode 100644 doc/background-tasks.rst diff --git a/doc/background-tasks.rst b/doc/background-tasks.rst new file mode 100644 index 00000000000..9f626b0ec84 --- /dev/null +++ b/doc/background-tasks.rst @@ -0,0 +1,173 @@ +================ +Background Tasks +================ + +.. version-added: 1.5.1 + +CKAN allows you to create tasks that run in the 'background', that is +asynchronously and without blocking the main application (these tasks can also +be automatically retried in the case of transient failures). Such tasks can be +created in :doc:`Extensions ` or in core CKAN. + +Background tasks can be essential to providing certain kinds of functionality, +for example: + +* Creating webhooks that notify other services when certain changes occur (for + example a dataset is updated) +* Performing processing or validation or on data (as done by the Archiver and + DataStorer Extensions) + + +Enabling Background Tasks +========================= + +To manage and run background tasks requires a job queue and CKAN uses celery_ +(plus the CKAN database) for this purpose. Thus, to use background tasks you +need to install and run celery_. + +Installation of celery_ will normally be taken care of by whichever component +or extension utilizes it so we skip that here. + +To run the celery daemon you have two options: + +1. In development setup you can just use paster. This can be done as simply + as:: + + paster celeryd + + This only works if you have a ``development.ini`` file in ckan root. + +2. In production, the daemon should be run with a different ini file and be run + as an init script. The simplest way to do this is to install supervisor:: + + apt-get install supervisor + + Using this file as a template and copy to ``/etc/supservisor/conf.d``:: + + https://github.com/okfn/ckan/blob/master/ckan/config/celery-supervisor.conf + + Alternatively, you can run:: + + paster celeryd --config=/path/to/file.ini + + +Writing Background Tasks +========================== + +These instructions should show you how to write an background task and how to +call it from inside CKAN or another extension using celery. + +Examples +-------- + +Here are some existing real examples of writing CKAN tasks: + +* https://github.com/okfn/ckanext-archiver +* https://github.com/okfn/ckanext-qa +* https://github.com/okfn/ckanext-datastorer + +Setup +----- + +An entry point is required inside the ``setup.py`` for your extension, and so +you should add something resembling the following that points to a function in +a module. In this case the function is called task_imports in the +``ckanext.NAME.celery_import`` module:: + + entry_points = """ + [ckan.celery_task] + tasks = ckanext.NAME.celery_import:task_imports + """ + +The function, in this case ``task_imports`` should be a function that returns +fully qualified module paths to modules that contain the defined task (see the +next section). In this case we will put all of our tasks in a file called +``tasks.py`` and so ``task_imports`` should be in a file called +``ckanext/NAME/celery_import.py``:: + + def task_imports(): + return ['ckanext.NAME.tasks'] + +This returns an iterable of all of the places to look to find tasks, in this +example we are only putting them in one place. + + +Implementing the tasks +---------------------- + +The most straightforward way of defining tasks in our ``tasks.py`` module, is +to use the decorators provided by celery. These decorators make it easy to just +define a function and then give it a name and make it accessible to celery. +Make sure you import celery from ckan.lib.celery_app:: + + from ckan.lib.celery_app import celery + +Implement your function, specifying the arguments you wish it to take. For our +sample we will use a simple echo task that will print out its argument to the +console:: + + def echo( message ): + print message + +Next it is important to decorate your function with the celery task decorator. +You should give the task a name, which is used later on when calling the task:: + + @celery.task(name = "NAME.echofunction") + def echo( message ): + print message + +That's it, your function is ready to be run asynchronously outside of the main +execution of the CKAN app. Next you should make sure you run ``python setup.py +develop`` in your extensions folder and then go to your CKAN installation +folder (normally pyenv/src/ckan/) to run the following command:: + + paster celeryd + +Once you have done this your task name ``NAME.echofunction`` should appear in +the list of tasks loaded. If it is there then you are all set and ready to go. +If not then you should try the following to try and resolve the problem: + +1. Make sure the entry point is defined correctly in your ``setup.py`` and that + you have executed ``python setup.py develop`` +2. Check that your task_imports function returns an iterable with valid module + names in +3. Ensure that the decorator marks the functions (if there is more than one + decorator, make sure the celery.task is the first one - which means it will + execute last). +4. If none of the above helps, go into #ckan on irc.freenode.net where there + should be people who can help you resolve your issue. + +Calling the task +---------------- + +Now that the task is defined, and has been loaded by celery it is ready to be +called. To call a background task you need to know only the name of the task, +and the arguments that it expects as well as providing it a task id.:: + + import uuid + from ckan.lib.celery_app import celery + celery.send_task("NAME.echofunction", args=["Hello World"], task_id=str(uuid.uuid4())) + +After executing this code you should see the message printed in the console +where you ran ``paster celeryd``. + + +Retrying on errors +------------------ + +Should your task fail to complete because of a transient error, it is possible +to ask celery to retry the task, after some period of time. The default wait +before retrying is three minutes, but you can optionally specify this in the +call to retry via the countdown parameter, and you can also specify the +exception that triggered the failure. For our example the call to retry would +look like the following - note that it calls the function name, not the task +name given in the decorator:: + + try: + ... some work that may fail, http request? + except Exception, e: + # Retry again in 2 minutes + echo.retry(args=(message), exc=e, countdown=120, max_retries=10) + +If you don't want to wait a period of time you can use the eta datetime +parameter to specify an explicit time to run the task (i.e. 9AM tomorrow) diff --git a/doc/extensions.rst b/doc/extensions.rst index 3dd1f84b08a..c786121cf0d 100644 --- a/doc/extensions.rst +++ b/doc/extensions.rst @@ -84,33 +84,9 @@ Your extension should now be enabled. You can disable it at any time by removing it from the list of ckan.plugins in the config file. - Enabling an Extension with Background Tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Some extensions need to run tasks in the background. In order to do this we use celery as a job queue. -Examples of these types of extensions are: - -* `ckanext-webstorer `_: Put resources that are tabular into the webstore i.e give tabular data a restful api. -* `ckanext-archiver `_: Archives resources so that ckan holds a copy of them i.e caches them. - -The above steps needs to be followed for these, but also: - -3. The celery daemon needs to be started. This can be done as simply as:: - - paster celeryd - - This only works if you have a ``development.ini`` file in ckan root. - - In production the daemon should be run with a different ini file and be run as an init script. - The simplest way to do this is to install supervisor:: - - apt-get install supervisor - - Using this file as a template and copy to ``/etc/supservisor/conf.d``:: - - https://github.com/okfn/ckan/blob/master/ckan/config/celery-supervisor.conf - - Also you can run:: +Some extensions need to run tasks in the background. See +:doc:`background-tasks` for how to enable background tasks. - paster celeryd --config=/path/to/file.ini diff --git a/doc/index.rst b/doc/index.rst index 9fca3db10fb..2433466eac1 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -35,6 +35,7 @@ Customizing and Extending linked-data-and-rdf filestore datastore + background-tasks Publishing Datasets =================== From f75c5b72a33a217d922b0ea76e3b1585177f88ef Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Mon, 19 Mar 2012 19:50:56 +0000 Subject: [PATCH 3/6] [#2226,doc/deployment][s]: incorporate http://wiki.ckan.org/Deployment. --- doc/deployment.rst | 235 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 doc/deployment.rst diff --git a/doc/deployment.rst b/doc/deployment.rst new file mode 100644 index 00000000000..2ee41dd3eb9 --- /dev/null +++ b/doc/deployment.rst @@ -0,0 +1,235 @@ +=============== +CKAN Deployment +=============== + +.. note:: If you use the package installation method your site will already + have been deployed using the Apache and modwsgi route described + below. + +This document covers how to deploy CKAN in a production setup where it is +available on the Internet. This will usually involve connecting the CKAN web +application to a web server such as Apache_ or NGinx_. + +As CKAN uses WSGI, a standard interface between web servers and Python web +applications, CKAN can be used with a number of different web server and +deployment configurations including: + +* Apache_ with the modwsgi Apache module +* Apache_ with paster and reverse proxy +* Nginx_ with paster and reverse proxy +* Nginx_ with uwsgi + +.. note:: below, we will only be able to give a few example of setups and many + other ones are possible. + +.. _Apache: http://httpd.apache.org/ +.. _Nginx: http://wiki.nginx.org/Main + +Deploying CKAN on an Ubuntu Server using Apache and modwsgi +=========================================================== + +These instructions have been tested on Ubuntu 10.04 with CKAN 1.6.1. + +This is the standard way to deploy CKAN. + +Install Apache and modwsgi +-------------------------- + +Install Apache_ (a web server) and modwsgi_ (an Apache module that adds WSGI +support to Apache):: + + sudo aptitude install apache2 libapache2-mod-wsgi + +.. _modwsgi: https://code.google.com/p/modwsgi/ + +Install CKAN +------------ + +The following assumes you have installed to ``/usr/local/demo.ckan.net`` with your virtualenv at ``/usr/local/demo.ckan.net/pyenv``. + +Create the WSGI Script File +--------------------------- + +Create the WSGI script file for your CKAN instance, +``/usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py``:: + + import os + instance_dir = '/usr/local/demo.ckan.net' + config_file = '/usr/local/demo.ckan.net/pyenv/src/ckan/development.ini' + pyenv_bin_dir = os.path.join(instance_dir, 'pyenv', 'bin') + activate_this = os.path.join(pyenv_bin_dir, 'activate_this.py') + execfile(activate_this, dict(__file__=activate_this)) + from paste.deploy import loadapp + config_filepath = os.path.join(instance_dir, config_file) + from paste.script.util.logging_config import fileConfig + fileConfig(config_filepath) + application = loadapp('config:%s' % config_filepath) + +The modwsgi Apache module will redirect requests to your web server to this +WSGI script file. The script file then handles those requests by directing them +on to your CKAN instance (after first configuring the Python environment for +CKAN to run in). + +Create the Apache Config File +----------------------------- + +Create the Apache config file for your CKAN instance by copying the default +Apache config file: + + cd /etc/apache2/sites-available + sudo cp default demo.ckan.net + +Edit ``/etc/apache2/sites-available/demo.ckan.net``, before the last line +(````) add these lines:: + + ServerName demo.ckan.net + ServerAlias demo.ckan.net + WSGIScriptAlias / /usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py + + # pass authorization info on (needed for rest api) + WSGIPassAuthorization On + ErrorLog /var/log/apache2/demo.ckan.net.error.log + CustomLog /var/log/apache2/demo.ckan.net.custom.log combined + +This tells the Apache modwsgi module to redirect any requests to the web server +to the CKAN WSGI script that you created above (``demo.ckan.net.py``). Your +WSGI script in turn directs the requests to your CKAN instance. + + +Create Directories for CKAN's Temporary Files +--------------------------------------------- + +Make the data and sstore directories and give them the right permissions:: + + cd /usr/local/demo.ckan.net/pyenv/src/ckan/ + mkdir data sstore + chmod g+w -R data sstore + sudo chgrp -R www-data data sstore + +CKAN Log File +------------- + +Edit your CKAN config file (e.g. +``/usr/local/demo.ckan.net/pyenv/src/ckan/development.ini``), find this line:: + + args = ("ckan.log", "a", 20000000, 9) + +and change it to set the ckan.log file location to somewhere that CKAN can write to, e.g.:: + + args = ("/var/log/ckan/demo.ckan.net/ckan.log", "a", 20000000, 9) + +Then create that directory and give it the right permissions:: + + sudo mkdir -p /var/log/ckan/demo.ckan.net + sudo chown www-data /var/log/ckan/demo.ckan.net + +Enable Your CKAN Site +--------------------- + +Finally, enable your CKAN site in Apache:: + + sudo a2ensite demo.ckan.net + sudo /etc/init.d/apache2 restart + +You should now be able to visit your server in a web browser and see your new +CKAN instance. + +Troubleshooting +--------------- + +Default Apache Welcome Page +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you see a default Apache welcome page where your CKAN front page should be, +it may be because the default Apache config file is overriding your CKAN config +file (both use port 80), so disable it and restart Apache:: + + $ sudo a2dissite default + $ sudo /etc/init.d/apache2 restart + +403 Forbidden and 500 Internal Server Error +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you see a 403 Forbidden or 500 Internal Server Error page where your CKAN +front page should be, you may have a problem with your unix file permissions. +The Apache web server needs to have permission to access your WSGI script file +(e.g. ``/usr/local/demo.ckan.net/pyenv/bin/demo.ckan.net.py``) ''and all of its +parent directories''. The permissions of the file should look like +``-rw-r--r--`` and the permissions of each of its parent directories should +look like ``drwxr-xr-x``. + +IOError: sys.stdout access restricted by mod_wsgi +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're getting 500 Internal Server Error pages and you see ``IOError: +sys.stdout access restricted by mod_wsgi`` in your log files, it means that +something in your WSGI application (e.g. your WSGI script file, your CKAN +instance, or one of your CKAN extensions) is trying to print to stdout, for +example by using standard Python ``print`` statements. WSGI applications are +not allowed to write to stdout. Possible solutions include: + +1. Remove the offending print statements. One option is to replace print + statements with statements like ``print >> sys.stderr, "..."`` + +2. Redirect all print statements to stderr:: + + import sys + sys.stdout = sys.stderr + +3. Allow your application to print to stdout by putting ``WSGIRestrictStdout Off`` in your Apache config file (not recommended). + +Also see https://code.google.com/p/modwsgi/wiki/ApplicationIssues + +Log Files +~~~~~~~~~ + +In general, if it's not working look in the log files in ``/var/log/apache2`` +for error messages. ``demo.ckan.net.error.log`` should be particularly +interesting. + +modwsgi wiki +~~~~~~~~~~~~ + +Some pages on the modwsgi wiki have some useful information for troubleshooting modwsgi problems: + +* https://code.google.com/p/modwsgi/wiki/ApplicationIssues +* http://code.google.com/p/modwsgi/wiki/DebuggingTechniques +* http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide +* http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines +* http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions +* http://code.google.com/p/modwsgi/wiki/ConfigurationIssues + + +Mounting CKAN at a non-root URL +=============================== + +CKAN (since version 1.6) can run mounted at a 'sub-directory' URL, such as +http://mysite.com/data/. This is achieved by changing the WSGIScriptAlias first +parameter (in the Apache site config). e.g.:: + + WSGIScriptAlias /data /home/dread/etc/ckan-pylons.py + +CORS +==== + +**As of CKAN v1.5 CORS is built in to CKAN so for CKAN >= 1.5 no modifications +to your webserver config are needed.** + +CORS = Cross Origin Resource Sharing. It is away to allow browsers (and hence +javascript in browsers) make requests to domains other than the one the browser +is currently on. + +In Apache you can enable CORS for you CKAN site by setting the following in +your config:: + + Header always set Access-Control-Allow-Origin "*" + Header always set Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" + Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type" + + # Respond to all OPTIONS requests with 200 OK + # This could be done in the webapp + # This is need for pre-flighted requests (POSTs/PUTs) + RewriteEngine On + RewriteCond %{REQUEST_METHOD} OPTIONS + RewriteRule ^(.*)$ $1 [R=200,L] + From 0d83cd877a1db85d05f5683d185ca374b7ae5c1d Mon Sep 17 00:00:00 2001 From: kindly Date: Mon, 19 Mar 2012 23:41:54 +0000 Subject: [PATCH 4/6] [2237] speed up tests and clean broken ones due to oredering --- ckan/lib/dictization/model_dictize.py | 2 +- .../functional/api/model/test_vocabulary.py | 67 +++++++++++++++---- ckan/tests/functional/test_admin.py | 4 +- ckan/tests/functional/test_home.py | 5 -- ckan/tests/functional/test_user.py | 10 ++- ckan/tests/lib/test_mailer.py | 6 ++ ckan/tests/misc/test_mock_mail_server.py | 6 ++ ckan/tests/mock_mail_server.py | 2 + ckan/tests/models/test_package.py | 3 + test-core.ini | 1 - 10 files changed, 82 insertions(+), 24 deletions(-) diff --git a/ckan/lib/dictization/model_dictize.py b/ckan/lib/dictization/model_dictize.py index ca8df610c03..1c766bd44d8 100644 --- a/ckan/lib/dictization/model_dictize.py +++ b/ckan/lib/dictization/model_dictize.py @@ -294,7 +294,7 @@ def user_list_dictize(obj_list, context, return sorted(result_list, key=sort_key, reverse=reverse) def member_dictize(member, context): - return table_dictize(member, context) + return d.table_dictize(member, context) def user_dictize(user, context): diff --git a/ckan/tests/functional/api/model/test_vocabulary.py b/ckan/tests/functional/api/model/test_vocabulary.py index 456fe17151b..0548811c216 100644 --- a/ckan/tests/functional/api/model/test_vocabulary.py +++ b/ckan/tests/functional/api/model/test_vocabulary.py @@ -2,27 +2,55 @@ from pylons.test import pylonsapp import paste.fixture from ckan.lib.helpers import json +import ckan.lib.dictization.model_dictize as model_dictize import sqlalchemy from nose.tools import raises, assert_raises class TestVocabulary(object): - def setup(self): + @classmethod + def setup_class(self): self.app = paste.fixture.TestApp(pylonsapp) - ckan.tests.CreateTestData.create() - self.sysadmin_user = ckan.model.User.get('testsysadmin') - self.normal_user = ckan.model.User.get('annafan') - # Make a couple of test vocabularies needed later. - self.genre_vocab = self._create_vocabulary(vocab_name="Genre", - user=self.sysadmin_user) - self.timeperiod_vocab = self._create_vocabulary( - vocab_name="Time Period", user=self.sysadmin_user) - self.composers_vocab = self._create_vocabulary(vocab_name="Composers", - user=self.sysadmin_user) - def teardown(self): + @classmethod + def teardown_class(self): ckan.model.repo.rebuild_db() + def setup(self): + self.clean_vocab() + model = ckan.model + context = {'model': model} + + genre = model.Vocabulary("Genre") + time_period = ckan.model.Vocabulary("Time Period") + composers = ckan.model.Vocabulary("Composers") + model.Session.add_all([genre, time_period, composers]) + + self.genre_vocab = model_dictize.vocabulary_dictize(genre, context) + self.timeperiod_vocab = model_dictize.vocabulary_dictize(time_period, context) + self.composers_vocab = model_dictize.vocabulary_dictize(composers, context) + ckan.model.Session.commit() + + self.sysadmin_user = ckan.model.User.get('admin') + self.normal_user = ckan.model.User.get('normal') + if not self.sysadmin_user: + normal_user = ckan.model.User(name=u'normal', password=u'annafan') + sysadmin_user = ckan.model.User(name=u'admin', password=u'testsysadmin') + ckan.model.Session.add(normal_user) + ckan.model.Session.add(sysadmin_user) + ckan.model.add_user_to_role(sysadmin_user, ckan.model.Role.ADMIN, ckan.model.System()) + ckan.model.Session.commit() + self.sysadmin_user = ckan.model.User.get('admin') + self.normal_user = ckan.model.User.get('normal') + + def clean_vocab(self): + ckan.model.Session.execute('delete from package_tag_revision') + ckan.model.Session.execute('delete from package_tag') + ckan.model.Session.execute('delete from tag') + ckan.model.Session.execute('delete from vocabulary') + ckan.model.Session.commit() + + @classmethod def _post(self, url, params=None, extra_environ=None): if params is None: params = {} @@ -32,6 +60,7 @@ def _post(self, url, params=None, extra_environ=None): assert not response.errors return response.json + @classmethod def _create_vocabulary(self, vocab_name=None, user=None): # Create a new vocabulary. params = {'name': vocab_name} @@ -506,6 +535,8 @@ def test_vocabulary_update_bad_tags(self): '''Test updating vocabularies with invalid tags. ''' + apikey = str(self.sysadmin_user.apikey) + for tags in ( [{'id': 'xxx'}, {'name': 'foo'}], [{'name': 'foo'}, {'name': None}], @@ -517,8 +548,7 @@ def test_vocabulary_update_bad_tags(self): params = {'id': self.genre_vocab['name'], 'tags': tags} response = self.app.post('/api/action/vocabulary_update', params=json.dumps(params), - extra_environ = {'Authorization': - str(self.sysadmin_user.apikey)}, + extra_environ = {'Authorization': apikey}, status=409) assert response.json['success'] == False assert response.json['error']['tags'] @@ -773,6 +803,9 @@ def test_add_vocab_tag_to_dataset(self): '''Test that a tag belonging to a vocab can be added to a dataset, retrieved from the dataset, and then removed from the dataset.''' + ckan.model.repo.rebuild_db() + self.setup() + ckan.tests.CreateTestData.create() # First add a tag to the vocab. vocab = self.genre_vocab tag = self._create_tag(self.sysadmin_user, 'noise', vocab) @@ -815,6 +848,9 @@ def test_add_vocab_tag_to_dataset(self): def test_delete_tag_from_vocab(self): '''Test that a tag can be deleted from a vocab.''' + ckan.model.repo.rebuild_db() + self.setup() + ckan.tests.CreateTestData.create() vocab = self.genre_vocab # First add some tags to the vocab. @@ -877,6 +913,9 @@ def test_delete_free_tag(self): automatically removed from datasets. ''' + ckan.model.repo.rebuild_db() + self.setup() + ckan.tests.CreateTestData.create() # Get a package from the API. package = (self._post('/api/action/package_show', {'id': self._post('/api/action/package_list')['result'][0]}) diff --git a/ckan/tests/functional/test_admin.py b/ckan/tests/functional/test_admin.py index 593d77ec47a..e15772f69c9 100644 --- a/ckan/tests/functional/test_admin.py +++ b/ckan/tests/functional/test_admin.py @@ -9,7 +9,7 @@ def setup_class(cls): @classmethod def teardown_class(self): - CreateTestData.delete() + model.repo.rebuild_db() #test that only sysadmins can access the /ckan-admin page def test_index(self): @@ -46,7 +46,7 @@ def setup_class(cls): @classmethod def teardown_class(self): - CreateTestData.delete() + model.repo.rebuild_db() def test_role_table(self): diff --git a/ckan/tests/functional/test_home.py b/ckan/tests/functional/test_home.py index 67d430f7fe7..f47893dbb48 100644 --- a/ckan/tests/functional/test_home.py +++ b/ckan/tests/functional/test_home.py @@ -41,11 +41,6 @@ def test_template_head_end(self): res = self.app.get(offset) assert 'ckan.template_head_end = ' - def test_template_head_end(self): - offset = url_for('home') - res = self.app.get(offset) - assert 'ckan.template_head_end = ' - def test_template_footer_end(self): offset = url_for('home') res = self.app.get(offset) diff --git a/ckan/tests/functional/test_user.py b/ckan/tests/functional/test_user.py index f7339c075bd..5ab1efcb599 100644 --- a/ckan/tests/functional/test_user.py +++ b/ckan/tests/functional/test_user.py @@ -1,5 +1,7 @@ from routes import url_for from nose.tools import assert_equal +from pylons import config +import hashlib from pprint import pprint from ckan.tests import search_related, CreateTestData @@ -12,7 +14,13 @@ class TestUserController(FunctionalTestCase, HtmlCheckMethods, PylonsTestCase, SmtpServerHarness): @classmethod - def setup_class(self): + def setup_class(cls): + smtp_server = config.get('test_smtp_server') + if smtp_server: + host, port = smtp_server.split(':') + port = int(port) + int(str(hashlib.md5(cls.__name__).hexdigest())[0], 16) + config['test_smtp_server'] = '%s:%s' % (host, port) + PylonsTestCase.setup_class() SmtpServerHarness.setup_class() CreateTestData.create() diff --git a/ckan/tests/lib/test_mailer.py b/ckan/tests/lib/test_mailer.py index bddef78d207..a8da49d6ba2 100644 --- a/ckan/tests/lib/test_mailer.py +++ b/ckan/tests/lib/test_mailer.py @@ -2,6 +2,7 @@ from nose.tools import assert_equal, assert_raises from pylons import config from email.mime.text import MIMEText +import hashlib from ckan import model from ckan.tests.pylons_controller import PylonsTestCase @@ -13,6 +14,11 @@ class TestMailer(SmtpServerHarness, PylonsTestCase): @classmethod def setup_class(cls): + smtp_server = config.get('test_smtp_server') + if smtp_server: + host, port = smtp_server.split(':') + port = int(port) + int(str(hashlib.md5(cls.__name__).hexdigest())[0], 16) + config['test_smtp_server'] = '%s:%s' % (host, port) CreateTestData.create_user(name='bob', email='bob@bob.net') CreateTestData.create_user(name='mary') #NB No email addr provided SmtpServerHarness.setup_class() diff --git a/ckan/tests/misc/test_mock_mail_server.py b/ckan/tests/misc/test_mock_mail_server.py index 6d254e0cf45..b410d81f928 100644 --- a/ckan/tests/misc/test_mock_mail_server.py +++ b/ckan/tests/misc/test_mock_mail_server.py @@ -2,6 +2,7 @@ from nose.tools import assert_equal from pylons import config from email.mime.text import MIMEText +import hashlib from ckan.tests.pylons_controller import PylonsTestCase from ckan.tests.mock_mail_server import SmtpServerHarness @@ -10,6 +11,11 @@ class TestMockMailServer(SmtpServerHarness, PylonsTestCase): @classmethod def setup_class(cls): + smtp_server = config.get('test_smtp_server') + if smtp_server: + host, port = smtp_server.split(':') + port = int(port) + int(str(hashlib.md5(cls.__name__).hexdigest())[0], 16) + config['test_smtp_server'] = '%s:%s' % (host, port) SmtpServerHarness.setup_class() PylonsTestCase.setup_class() diff --git a/ckan/tests/mock_mail_server.py b/ckan/tests/mock_mail_server.py index bf8350e76eb..9020cb41a9d 100644 --- a/ckan/tests/mock_mail_server.py +++ b/ckan/tests/mock_mail_server.py @@ -2,6 +2,7 @@ import asyncore import socket from smtpd import SMTPServer +import hashlib from pylons import config @@ -67,6 +68,7 @@ def setup_class(cls): host, port = smtp_server.split(':') else: host, port = smtp_server, 25 + cls.port = port cls.smtp_thread = MockSmtpServerThread(host, int(port)) cls.smtp_thread.start() diff --git a/ckan/tests/models/test_package.py b/ckan/tests/models/test_package.py index 61ee8674ebe..ffc49cb4709 100644 --- a/ckan/tests/models/test_package.py +++ b/ckan/tests/models/test_package.py @@ -479,6 +479,9 @@ class TestPackagePurge: @classmethod def setup_class(self): CreateTestData.create() + @classmethod + def teardown_class(self): + model.repo.rebuild_db() def test_purge(self): pkgs = model.Session.query(model.Package).all() for p in pkgs: diff --git a/test-core.ini b/test-core.ini index 48862cfd602..3e7fde4831f 100644 --- a/test-core.ini +++ b/test-core.ini @@ -24,7 +24,6 @@ ckan.site_title = CKAN ckan.site_logo = /images/ckan_logo_fullname_long.png ckan.site_description = package_form = standard -ckan.plugins = synchronous_search licenses_group_url = # pyamqplib or queue carrot_messaging_library = queue From c594ef2a3f791e24bb5a6976ddf0f6458410f3b0 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Tue, 20 Mar 2012 08:44:02 +0000 Subject: [PATCH 5/6] [#2226,doc][xs]: items missing from prenultimate commit f75c5b72a33a217d922b0ea76e3b1585177f88ef. --- doc/index.rst | 1 + doc/post-installation.rst | 5 +---- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/index.rst b/doc/index.rst index 2433466eac1..a5cc0befe02 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -19,6 +19,7 @@ Installation install-from-package install-from-source post-installation + deployment solr-setup Customizing and Extending diff --git a/doc/post-installation.rst b/doc/post-installation.rst index 63d2ad1b04e..5e391249cb2 100644 --- a/doc/post-installation.rst +++ b/doc/post-installation.rst @@ -56,8 +56,5 @@ You may want to deploy your CKAN instance at this point, to share with others. If you have installed CKAN from packages, then Apache and WSGI deployment scripts are already configured for you in standard locations. -If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see http://wiki.ckan.net/Deployment +If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see :doc:`deployment`. -CKAN has been successfully deployed by a variety of other methods including Apache reverse proxy + paster, nginx reverse proxy + paster, and nginx + uwsgi. - -You can now proceed to :doc:`theming`. From 858e23008855e900cfc0c96269880a8a4931b06b Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Tue, 20 Mar 2012 08:49:54 +0000 Subject: [PATCH 6/6] [master,controller/template][xs]: support urls which do not have .html in them (but template does) and only abort 404 on template not found as otherwise we can hide errors. --- ckan/controllers/template.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/ckan/controllers/template.py b/ckan/controllers/template.py index e789857ccfa..04468f7d68b 100644 --- a/ckan/controllers/template.py +++ b/ckan/controllers/template.py @@ -1,4 +1,5 @@ from ckan.lib.base import * +from genshi.template.loader import TemplateNotFound class TemplateController(BaseController): @@ -26,5 +27,12 @@ def view(self, url): """ try: return render(url) - except: - abort(404) + except TemplateNotFound: + if url.endswith('.html'): + abort(404) + url += '.html' + try: + return render(url) + except TemplateNotFound: + abort(404) +