Merge branch 'master' into 356-auth-create-user-option

Conflicts:
	ckan/templates/header.html
	doc/organizations_and_groups.rst

.tx/config

@@ -1,7 +1,7 @@
 [main]
 host = http://www.transifex.net
 
-[ckan.1-8]
+[ckan.2-0]
 file_filter = ckan/i18n/<lang>/LC_MESSAGES/ckan.po
 source_file = ckan/i18n/ckan.pot
 source_lang = en

CHANGELOG.txt

@@ -1,21 +1,180 @@
 CKAN CHANGELOG
 ++++++++++++++
 
-v2.0
-====
+v2.0 2013-05-10
+===============
+
+.. note:: Starting on v2.0, issue numbers with four digits refer to the old
+ ticketing system at http://trac.ckan.org and the ones with three digits refer
+ to GitHub issues. For example:
+
+ * #3020 is http://trac.ckan.org/ticket/3020
+ * #271 is https://github.com/okfn/ckan/issues/271
+
+ Some GitHub issues URLs will redirect to GitHub pull request pages.
+
+.. note:: v2.0 is a huge release so the changes listed here are just the
+ highlights. Bug fixes are not listed.
+
+Note: This version requires a requirements upgrade on source installations
+
+Note: This version requires a database upgrade
+
+Note: This version requires a Solr schema upgrade
+
+Organizations based authorization (see :doc:`authorization`):
+ CKAN's new "organizations" feature replaces the old authorization system
+ with a new one based on publisher organizations. It replaces the "Publisher
+ Profile and Workflow" feature from CKAN 1.X, any instances relying on it will
+ need to be updated.
+
+ * New organization-based authorization and organization of datasets
+ * Supports private datasets
+ * Publisher workflow
+ * New authorization ini file options
+
+
+New frontend (see :doc:`theming`):
+ CKAN's frontend has been completely redesigned, inside and out. There is
+ a new default theme and the template engine has moved from Genshi to
+ Jinja2. Any custom templates using Genshi will need to be updated, although
+ there is a :ref:`ckan.legacy_templates` setting to aid in the migration.
+
+ * Block-based template inheritance
+ * Custom jinja tags: {% ckan_extends %}, {% snippet %} and {% url_for %} (#2502, #2503)
+ * CSS "primer" page for theme developers
+ * We're now using LESS for CSS
+ * Scalable font icons (#2563)
+ * Social sharing buttons (google plus, facebook, twitter)
+   (this replaces the ckanext-social extension)
+ * Three-stage dataset creation form (#2501)
+ * New `paster front-end-build` command does everything needed to build the
+   frontend for a production CKAN site (runs `paster less` to compile the css
+   files, `paster minify` to minify the css and js files, etc.)
+
+Plugins & Extensions:
+ * New plugins toolkit provides a stable set of utility and helper functions
+   for CKAN plugins to depend on.
+ * The IDatasetForm plugin interface has been redesigned (note: this breaks
+   backwards-compatibility with existing IDatasetForm plugins) (#649)
+ * Many IDatasetForm bugs were fixed
+ * New example extensions in core, and better documentation for the relevant
+   plugin interfaces: example_itemplatehelpers (#447),
+   example_idatasetform (#2750), hopefully more to come in 2.1!
+ * New IFacets interface that allows to modify the facets shown on various
+   pages. (#400)
+ * The get_action() function now automatically adds 'model' and 'session' to
+   the context dict (this saves on boiler-plate code, and means plugins don't
+   have to import ckan.model in order to call get_action()) (#172)
+
+Activity Streams, Following & User Dashboard:
+ * New visual design for activity streams (#2941)
+ * Group activity streams now include activities for changes to any of the
+   group's datasets (#1664)
+ * Group activity streams now appear on group pages (previously they could
+   only be retrieved via the api)
+ * Dataset activity streams now appear on dataset pages (previously they could
+   only be retrieved via the api) (#3024)
+ * Users can now follow groups (previously you could only follow users or
+   datasets) (#3005)
+ * Activity streams and following are also supported for organizations (#505)
+ * When you're logged into CKAN, you now get a notifications count in the
+   top-right corner of the site, telling you how many new notifications you
+   have on your dashboard. Clicking on the count takes you to your dashboard
+   page to view your notifications. (#3009)
+ * Optionally, you can also receive notifications by email when you have new
+   activities on your dashboard (#1635)
+ * Infinite scrolling of activity streams (if you scroll to the bottom of a
+   an activity stream, CKAN will automatically load more activities) (#3018)
+ * Redesigned user dashboard (#3028):
+
+   - New dropdown-menu enables you to filter you dashboard activity stream to
+     show only activities from a particular user, dataset, group or
+     organization that you're following
+   - New sidebar shows previews and unfollow buttons (when the activity stream
+     is filtered)
+ * New :ref:`ckan.activity_streams_enabled` config file setting allows you to
+   disable the generation of activity streams (#654)
+
+Data Preview:
+ * PDF files preview (#2203)
+ * JSON files preview
+ * HTML pages preview (in an iframe) (#2888)
+ * New plugin extension point that allows plugins to add custom data previews
+   for different data types (#2961)
+ * Improved Recline Data Explorer previews (CSV files, Excel files..)
+ * Plain text files preview
+
+
+API:
+ * The Action API is now CKAN's default API, and the API documentation has
+   been rewritten (#357)
+
+Other highlights:
+ * CKAN now has continuous integration testing at
+   https://travis-ci.org/okfn/ckan/
+ * Dataset pages now have <link rel="alternate" type="application/rdf+xml"
+   links in the HTML headers, allows linked-data tools to find CKAN's RDF
+   rendering of a dataset's metadata (#413)
+ * CKAN's DataStore and Data API have been rewritten, and now use PostgreSQL
+   instead of elasticsearch, so there's no need to install elasticsearch
+   anymore (this feature was also back-ported to CKAN 1.8) (#2733)
+ * New Config page for sysadmins (/ckan-admin/config) enables sysadmins to set
+   the site title, tag line, logo, the intro text shown on the front page,
+   the about text shown on the /about page, select a theme, and add custom
+   CSS (#2302, #2781)
+ * New `paster color` command for creating color schemes
+ * Fanstatic integration (#2371):
+
+   - CKAN now uses Fanstatic to specify required static resource files
+     (js, css..) for web pages
+   - Enables each page to only include the static files that it needs,
+     reducing page loads
+   - Enables CKAN to use bundled and minified static files, further reducing
+     page loads
+   - CKAN's new `paster minify` command is used to create minified js and
+     css files (#2950) (also see `paster front-end-build`)
+ * CKAN will now recognise common file format strings such as
+   "application/json", "JSON", ".json" and "json" as a single file type "json"
+   (#2416)
+ * CKAN now supports internalization of strings in javascript files, the new
+   `paster trans` command is used to pull translatable strings out of
+   javascript files (#2774, #2750)
+ * convert_to/from_extras have been fixed to not add quotes around strings (#2930)
+ * Updated CKAN coding standards (#3020) and CONTRIBUTING.rst file
+ * Built-in page view counting and 'popular' badges on datasets and resources
+   There's also a paster command to export the tracking data to a csv file (#195)
+ * Updated CKAN Coding Standards and new CONTRIBUTING.rst file
+ * You can now change the sort ordering of datasets on the dataset search page
+
+Deprecated and removed:
+ * The IGenshiStreamFilter plugin interface is deprecated (#271), use the
+   ITemplateHelpers plugin interface instead
+ * The Model, Search and Util APIs are deprecated, use the Action API instead
+ * Removed restrict_template_vars config setting (#2257)
+ * Removed deprecated facet_title() template helper function, use
+   get_facet_title() instead (#2257)
+ * Removed deprecated am_authorized() template helper function, use
+   check_access() instead (#2257)
+ * Removed deprecated datetime_to_datestr() template helper function (#2257)
+
+
+v1.8.1 2013-05-10
+=================
+
+Bug fixes:
+ * Fixed possible XSS vulnerability on html input (#703)
+ * Fix unicode template 500 error (#808)
+ * Fix error on related controller
 
-* [#2257] Removed restrict_template_vars config setting.
-* [#2257] Removed deprecated facet_title() template helper function, use
-  get_facet_title() instead.
-* [#2257] Removed deprecated am_authorized() template helper function, use
-  check_access() instead.
-* [#2257] Removed deprecated datetime_to_datestr() template helper function.
 
 v1.8 2012-10-19
 ===============
 
 Note: This version requires a requirements upgrade on source installations
+
 Note: This version requires a database upgrade
+
 Note: This version does not require a Solr schema upgrade
 
 Major
@@ -65,6 +224,13 @@ API changes and deprecation:
    but is deprecated, and will be removed in future versions. (#2313)
 
 
+v1.7.3 2013-05-10
+=================
+
+Bug fixes:
+ * Fixed possible XSS vulnerability on html input (#703)
+
+
 v1.7.2 2012-10-19
 =================
 

CONTRIBUTING.rst

@@ -5,17 +5,23 @@ Contributing to CKAN
 .. _CKAN repo on GitHub: https://github.com/okfn/ckan
 .. _CKAN issue tracker: https://github.com/okfn/ckan/issues
 .. _docs.ckan.org: http://docs.ckan.org
-.. _Contributing to CKAN's Documentation: https://github.com/okfn/ckan/blob/master/CONTRIBUTING.rst#contributing-to-ckans-documentation
 
 (This section is about contributing code, if you want to contribute
-documentation see `Contributing to CKAN's Documentation`_.)
+documentation see `Contributing to the CKAN Documentation`_.)
 
 CKAN is a free software project and code contributions are welcome. To
 contribute code to CKAN you should fork CKAN to your own GitHub account, push
 your code to a feature branch on your fork, then make a pull request for your
 branch on the central CKAN repo. We'll go through each step in detail below...
 
 
+Coding Standards
+----------------
+
+When writing code for CKAN, try to follow our
+`coding standards <http://docs.ckan.org/en/latest/#for-ckan-developers>`_.
+
+
 Fork CKAN on GitHub
 -------------------
 
@@ -114,11 +120,8 @@ When submitting a pull request:
   see `Feature Branches`_.
 - Your branch should contain new or changed tests for any new or changed
   code.
-- Your branch should contain updates to the
-  `CHANGELOG file <https://github.com/okfn/ckan/blob/master/CHANGELOG.txt>`_
-  briefly summarising your code changes.
 - Your branch should contain new or updated documentation for any new or
-  updated code, see `Contributing to CKAN's Documentation`_.
+  updated code, see `Contributing to the CKAN Documentation`_.
 - Your branch should be up to date with the master branch of the central
   CKAN repo, see `Keeping Up with master`_.
 - All the CKAN tests should pass on your branch, see
@@ -133,16 +136,9 @@ When merging a feature or bug branch into master:
 - Use the ``--no-ff`` option in the ``git merge`` command,
 
 
-Coding Standards
-----------------
-
-When writing code for CKAN, try to follow our
-`coding standards <http://docs.ckan.org/en/latest/#for-ckan-developers>`_.
-
-
-====================================
-Contributing to CKAN's Documentation
-====================================
+======================================
+Contributing to the CKAN Documentation
+======================================
 
 Note: getting started with contributing to `docs.ckan.org`_ is a little
 complicated.  An easier way to contribute documentation to CKAN is to

README.rst

@@ -18,14 +18,21 @@ Installation
 See the `CKAN Documentation <http://docs.ckan.org>`_ for installation instructions.
 
 
-Community
----------
+Support
+-------
 
-* Developer mailing list: `ckan-dev@lists.okfn.org <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_
-* Developer IRC channel: `#ckan on irc.freenode.net <http://webchat.freenode.net/?channels=ckan>`_
-* `Issue tracker <https://github.com/okfn/ckan/issues>`_
-* `CKAN tag on StackOverflow <http://stackoverflow.com/questions/tagged/ckan>`_
-* `Wiki <https://github.com/okfn/ckan/wiki>`_
+If you need help with CKAN or want to ask a question about CKAN, use either the
+`ckan-discuss`_ mailing list or the `CKAN tag on Stack Overflow`_ (try
+searching the Stack Overflow and ckan-discuss archives for an answer to your
+question first).
+
+If you've found a bug in CKAN, open a new issue on CKAN's `GitHub Issues`_ (try
+searching first to see if there's already an issue for your bug).
+
+
+.. _CKAN tag on Stack Overflow: http://stackoverflow.com/questions/tagged/ckan
+.. _ckan-discuss: http://lists.okfn.org/mailman/listinfo/ckan-discuss
+.. _GitHub Issues: https://github.com/okfn/ckan/issues
 
 
 Contributing to CKAN
@@ -34,11 +41,22 @@ Contributing to CKAN
 For contributing to CKAN or its documentation, see
 `CONTRIBUTING <https://github.com/okfn/ckan/blob/master/CONTRIBUTING.rst>`_.
 
+If you want to talk about CKAN development say hi to the CKAN developers on the
+`ckan-dev`_ mailing list or in the `#ckan`_ IRC channel on irc.freenode.net.
+
+If you've figured out how to do something with CKAN and want to document it for
+others, make a new page on the `CKAN wiki`_, and tell us about it on
+`ckan-dev`_.
+
+.. _ckan-dev: http://lists.okfn.org/mailman/listinfo/ckan-dev
+.. _#ckan: http://webchat.freenode.net/?channels=ckan
+.. _CKAN Wiki: https://github.com/okfn/ckan/wiki
+
 
 Copying and License
 -------------------
 
-This material is copyright (c) 2006-2011 Open Knowledge Foundation.
+This material is copyright (c) 2006-2013 Open Knowledge Foundation.
 
 It is open and licensed under the GNU Affero General Public License (AGPL) v3.0
 whose full text may be found at:

bin/osx-postgres-mem.sh

@@ -13,7 +13,8 @@ case $1 in
 		${PGCTL} -D ${PGDATA} init
 		${PGCTL} -D ${PGDATA} start
 		sleep 2;
-		psql -c "CREATE DATABASE ckantest;" postgres
+		psql -c "CREATE DATABASE ckan_dev;" postgres
+		psql -c "CREATE DATABASE ckan_test;" postgres
 		;;
 	stop)
 		## stop postgres

bin/travis-build

@@ -15,41 +15,38 @@ sudo add-apt-repository -yy ppa:pitti/postgresql
 sudo apt-get update -qq
 sudo apt-get install solr-jetty postgresql-$PGVERSION
 
-# Don't require a password to access DB
-sudo sed -i -e 's/ident/trust/g' /etc/postgresql/$PGVERSION/main/pg_hba.conf 
-
 sudo service postgresql reload
 
+# Setup postgres' users and databases
+sudo -u postgres psql -c "CREATE USER ckan_default WITH PASSWORD 'pass';"
+sudo -u postgres psql -c "CREATE USER datastore_default WITH PASSWORD 'pass';"
+sudo -u postgres psql -c 'CREATE DATABASE ckan_test WITH OWNER ckan_default;'
+sudo -u postgres psql -c 'CREATE DATABASE datastore_test WITH OWNER ckan_default;'
+
 pip install -r pip-requirements.txt --use-mirrors
 pip install -r pip-requirements-test.txt --use-mirrors
 
-psql -c 'CREATE DATABASE ckantest;' -U postgres
-psql -c 'CREATE DATABASE datastore;' -U postgres
-
 python setup.py develop
 
-# Configure CKAN's configuration file
-paster make-config ckan development.ini --no-interactive
-sed -i -e 's/.*solr_url.*/solr_url = http:\/\/127.0.0.1:8983\/solr/' development.ini
-sed -i -e 's/.*ckan\.site_id.*/ckan.site_id = travis_ci/' development.ini
-sed -i -e 's/^sqlalchemy.url.*/sqlalchemy.url = postgresql:\/\/postgres@\/ckantest/' development.ini
-sed -i -e 's/.*datastore.write_url.*/ckan.datastore.write_url = postgresql:\/\/postgres@\/datastore/' development.ini
-
 # Configure Solr
 echo "NO_START=0\nJETTY_HOST=127.0.0.1\nJETTY_PORT=8983\nJAVA_HOME=$JAVA_HOME" | sudo tee /etc/default/jetty
+# FIXME the solr schema cannot be hardcoded as it is dependent on the ckan version
 sudo cp ckan/config/solr/schema-2.0.xml /etc/solr/conf/schema.xml
 sudo service jetty restart
 
-paster --plugin=ckan db init
+paster db init -c test-core.ini
 
 # If Postgres >= 9.0, we don't need to use datastore's legacy mode.
 if [ $PGVERSION != '8.4' ]
 then
-  psql -c 'CREATE USER readonlyuser;' -U postgres
-  sed -i -e 's/.*datastore.read_url.*/ckan.datastore.read_url = postgresql:\/\/readonlyuser@\/datastore/' development.ini
-  paster datastore set-permissions postgres
+  psql -c 'CREATE USER datastore_default;' -U postgres
+  sed -i -e 's/.*datastore.read_url.*/ckan.datastore.read_url = postgresql:\/\/datastore_default@\/datastore_test/' test-core.ini
+  paster datastore set-permissions postgres -c test-core.ini
+else
+  sed -i -e 's/.*datastore.read_url.*//' test-core.ini
 fi
 
+cat test-core.ini
 
 # And finally, run the tests
 nosetests --ckan --with-pylons=test-core.ini --nologcapture ckan ckanext