[sync]

update to latest
ckan · Jul 30, 2018 · 93470d6 · 93470d6
2 parents 867f3fa + f1a24c2
commit 93470d6
Show file tree

Hide file tree

Showing 680 changed files with 27,553 additions and 10,739 deletions.
diff --git a/.tx/config b/.tx/config
@@ -1,7 +1,7 @@
 [main]
 host = https://www.transifex.com
 
-[ckan.2-4]
+[ckan.2-8]
 file_filter = ckan/i18n/<lang>/LC_MESSAGES/ckan.po
 source_file = ckan/i18n/ckan.pot
 source_lang = en

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -7,11 +7,63 @@
 Changelog
 ---------
 
-v?? (TBA)
-=========
+v.2.9.0 TBA
+==================
 
-Note: This version requires re-running the 'datastore set-permissions' command
-   (assuming you run DataStore). See: :ref:`datastore-set-permissions`
+Deprecations:
+ * ``c.action`` and ``c.controller`` variables should be avoided.
+   ``ckan.plugins.toolkit.get_endpoint`` can be used instead. This function
+   returns tuple of two items(depending on request handler):
+   1. Flask blueprint name / Pylons controller name
+   2. Flask view name / Pylons action name
+   In some cases, Flask blueprints have names that are differs from their
+   Pylons equivalents. For example, 'package' controller is divided between
+   'dataset' and 'resource' blueprints. For such cases you may need to perform
+   additional check of returned value:
+
+   >>> if toolkit.get_endpoint()[0] in ['dataset', 'package']:
+   >>>     do_something()
+
+   In this code snippet, will be called if current request is handled via Flask's
+   dataset blueprint in CKAN>=2.9, and, in the same time, it's still working for
+   Pylons package controller in CKAN<2.9
+
+
+v.2.8.1 2018-07-25
+==================
+
+General notes:
+ * Note: This version does not requires a requirements upgrade on source installations
+ * Note: This version does not requires a database upgrade
+ * Note: This version does not require a Solr schema upgrade
+
+Fixes:
+
+ * "Add Filter" Performance Issue (`#4162 <https://github.com/ckan/ckan/issues/4162>`_)
+ * Error handler update (`#4257 <https://github.com/ckan/ckan/issues/4257>`_)
+ * "New view" button does not work (`#4260 <https://github.com/ckan/ckan/issues/4260>`_)
+ * Upload logo is not working (`#4262 <https://github.com/ckan/ckan/issues/4262>`_)
+ * Unable to pip install ckan (`#4271 <https://github.com/ckan/ckan/issues/4271>`_)
+ * The "License" Icon in 2.8 is wrong (`#4272 <https://github.com/ckan/ckan/issues/4272>`_)
+ * Search - input- border color is overly specific in CSS (`#4273 <https://github.com/ckan/ckan/issues/4273>`_)
+ * Site logo image does not scale down when very large (`#4283 <https://github.com/ckan/ckan/issues/4283>`_)
+ * Validation Error on datastore_search when sorting timestamp fields (`#4288 <https://github.com/ckan/ckan/issues/4288>`_)
+ * Undocumented changes breaking error_document_template (`#4303 <https://github.com/ckan/ckan/issues/4303>`_)
+ * Internal server error when viewing /dashboard when logged out (`#4305 <https://github.com/ckan/ckan/issues/4305>`_)
+ * Missing c.action attribute in 2.8.0 templates (`#4310 <https://github.com/ckan/ckan/issues/4310>`_)
+ * [multilingual] AttributeError: '_Globals' object has no attribute 'fields' (`#4338 <https://github.com/ckan/ckan/issues/4338>`_)
+ * `search` legacy route missing (`#4346 <https://github.com/ckan/ckan/issues/4346>`_)
+
+
+v.2.8.0 2018-05-09
+==================
+
+General notes:
+ * This version requires a requirements upgrade on source installations
+ * This version requires a database upgrade
+ * This version requires a Solr schema upgrade
+ * This version requires re-running the ``datastore set-permissions`` command
+   (assuming you are using the DataStore). See: :ref:`datastore-set-permissions`
 
    Otherwise new and updated datasets will not be searchable in DataStore and
    the logs will contain this error::
@@ -21,6 +73,178 @@ Note: This version requires re-running the 'datastore set-permissions' command
    CKAN developers should also re-run set-permissions on the test database:
    :ref:`datastore-test-set-permissions`
 
+ * There are several old features being officially deprecated starting from
+   this version. Check the *Deprecations* section to be prepared.
+
+Major changes:
+ * New revamped frontend templates based on Bootstrap 3, see "Changes and deprecations" (#3547)
+ * Allow datastore_search_sql on private datasets (#2562)
+ * New Flask blueprints migrated from old Pylons controllers: user, dashboard, feeds, admin and home (#3927, #3870, #3775, #3762)
+ * Improved support for custom groups and organization types (#4032)
+ * Hide user details to anonymous users (#3915)
+
+Minor changes:
+ * Allow chaining of authentication functions (#3679)
+ * Show custom dataset types in search pages (#3807)
+ * Overriding datastore authorization system (#3679)
+ * Standardize on url_for (#3831)
+ * Deprecate notify_after_commit (#3633)
+ *  _mail_recipient header override (#3781)
+ * Restrict access to member forms (#3684)
+ * Clean up template rendering code (#3923)
+ * Permission labels are indexed by type text in SOLR (#3863)
+ * CLI commands require a Flask test request context (#3760)
+ * Allow IValidator to override existing validators (#3865)
+ * Shrink datastore_create response size (#3810)
+ * Stable version URLs CKAN for documentation (#4209)
+ * API Documentation update (#4136)
+ * Documentation of Data Dictionary  (#3989)
+ * Remove datastore legacy mode (#4041)
+ * Map old Pylons routes to Flask ones (#4066)
+
+Bug fixes:
+ * File uploads don't work on new Flask based API (#3869)
+ * {% ckan_extends %} not working on templates served by Flask (#4044)
+ * Problems in background workers with non-core database relations (#3606)
+ * Render_datetime can't handle dates before year 1900 (#2228)
+ * DatapusherPlugin implementation of notify() can call 'datapusher_submit' multiple times (#2334)
+ * Dataset creation page generates incorrect URLs with Chrome autocomplete (#2501)
+ * Search buttons need accessible labels (#2550)
+ * Column name length limit for datastore upload (#2804)
+ * #2373: Do not validate packages or resources from database to views (#3016)
+ * Creation of dataset - different behaviour between Web API & CKAN Interface functionality (#3528)
+ * Redirecting to same page in non-root hosted ckan adds extra root_path to url  (#3499)
+ * Beaker 1.8.0 exception when the code is served from OSX via Vagrant (#3512)
+ * Add "Add Dataset" button to user's and group's page (#2794)
+ * Some links in CKAN is not reachable (#2898)
+ * Exception when specifying a directory in the ckan.i18n_directory option (#3539)
+ * Resource view filter user filters JS error (#3590)
+ * Recaptcha v1 will stop working 2018-3-31 (#4061)
+ * "Testing coding standards" page in docs is missing code snippets (#3635)
+ * Followers count not updated immediately on UI (#3639)
+ * Increase jQuery version (#3665)
+ * Search icon on many pages is not properly vertically aligned (#3654)
+ * Datatables view can't be used as a default view (#3669)
+ * Resource URL is not validated on create/update (#3660)
+ * Upload to Datastore tab shows incorrect time at Upload Log (#3588)
+ * Filter results button is not working (#3593)
+ * Broken link in "Upgrading CKAN’s dependencies" doc page (#3637)
+ * Default logo image not properly saved (#3656)
+ * Activity test relies on datetime.now() (#3644)
+ * Info block text for Format field not properly aligned in resource form page (#3663)
+ * Issue upon creating new organization/group through UI form (#3661)
+ * In API docs "package_create" lists "owner_org" as optional (#3647)
+ * Embed modal window not working (#3731)
+ * Frontent build command does not work on master (#3688)
+ * Loading image duplicated  (#3716)
+ * Datastore set-up error - logging getting in the way (#3694)
+ * Registering a new account redirects to an unprefixed url (#3834)
+ * Exception in search page when not authorized (#4081)
+ * Datastore full-text-search column is populated by postgres trigger rather than python (#3785)
+ * Datastore dump results are not the same as data in database (#4150)
+ * Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+ * No such file or directory: '/usr/lib/ckan/default/src/ckan/requirement-setuptools.txt' during installation from source (#3641)
+ * Register user form missing required field indicators (#3658)
+ * Datastore full-text-search column is populated by postgres trigger rather than python  (#3786)
+ * Add missing major changes to change log (#3799)
+ * Paster/CLI config-tool requires _get_test_app which in turn requires a dev-only dependency (#3806)
+ * Change log doesn't mention necessary Solr scheme upgrade (#3851)
+ * TypeError: expected byte string object, value of type unicode found (#3921)
+ * CKAN's state table clashes with PostGIS generated TIGER state table (#3929)
+ * [Docker] entrypoint initdb.d sql files copied to root (#3939)
+ * DataStore status page throws TypeError - Bleach upgrade regression (#3968)
+ * Source install error with who.ini (#4020)
+ * making a JSONP call to the CKAN API returns the wrong mime type (#4022)
+ * Deleting a resource sets datastore_active=False to all resources and overrides their extras (#4042)
+ * Deleting first Group and Organization custom field is not possible (#4094)
+
+Changes and deprecations:
+ * The default templates included in CKAN core have been updated to use Bootstrap 3. Extensions
+   implementing custom themes are encouraged to update their templates, but they can still
+   make CKAN load the old Bootstrap 2 templates during the transition using the following
+   configuration options::
+
+        ckan.base_public_folder = public-bs2
+        ckan.base_templates_folder = templates-bs2
+
+ * The API versions 1 and 2 (also known as the REST API), ie ``/api/rest/*`` have been
+   completely removed in favour of the version 3 (action API, ``/api/action/*``).
+ * The old Celery based background jobs have been removed in CKAN 2.8 in favour of the new RQ based
+   jobs (http://docs.ckan.org/en/latest/maintaining/background-tasks.html). Extensions can still
+   of course use Celery but they will need to handle the management themselves.
+ * `ckan.recaptcha.version` config option is removed, since v2 is the only valid
+    version now (#4061)
+ * After introducing dataset blueprint, `h.get_facet_items_dict` takes search_facets as second argument.
+   This change is aimed to reduce usage of global variables in context. For a while, it has default value
+   of None, in which case, `c.search_facets` will be used. But all template designers are strongly advised
+   to specify this argument explicitly, as in future it'll become required.
+ * The ``ckan.recaptcha.version`` config option is now removed, since v2 is the only valid version now (#4061)
+
+v2.7.4 2018-05-09
+=================
+
+ * Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+ * Datastore dump results are not the same as data in database (#4150)
+
+v2.7.3 2018-03-15
+=================
+
+General notes:
+ * As with all patch releases this one does not include requirement changes.
+   However in some scenarios you might encounter the following error while
+   installing or upgrading this version of CKAN::
+
+     Error: could not determine PostgreSQL version from '10.2'
+
+   This is due to a bug in the psycopg2 version pinned to the release. To solve
+   it, upgrade psycopg2 with the following command::
+
+     pip install --upgrade psycopg2==2.7.3.2
+
+ * This release does not require a Solr schema upgrade, but if you are having the
+   issues described in #3863 (datasets wrongly indexed in multilingual setups),
+   you can upgrade the Solr schema and reindex to solve them.
+
+ * #3422 (implemented in #3425) introduced a major bug where if a resource was
+   deleted and the DataStore was active extras from all resources on the site where
+   changed. This is now fixed as part of this release but if your database is already
+   affected you will need to run a script to restore the extras to their
+   previous state. Remember, you only need to run the script if all the following are
+   true:
+
+   1. You are currently running CKAN 2.7.0 or 2.7.2, and
+   2. You have enabled the DataStore, and
+   3. One or more resources with data on the DataStore have been deleted (or you
+      suspect they might have been)
+
+   If all these are true you can run the following script to restore the extras to
+   their previous state:
+
+   https://github.com/ckan/ckan/blob/dev-v2.7/scripts/4042_fix_resource_extras.py
+
+   This issue is described in #4042
+
+Fixes:
+ * Fix toggle bars header icon (#3880)
+ * Change CORS header keys and values to string instead of unicode (#3855)
+ * Fix cors header when all origins are allowed (#3898)
+ * Update SOLR schema.xml reference in Dockerfile
+ * Build local SOLR container by default
+ * Create datastore indexes only if they are not exist
+ * Properly close file responses
+ * Use javascript content-type for jsonp responses (#4022)
+ * Add Data Dictionary documentation (#3989)
+ * Fix SOLR index delete_package implementation
+ * Add second half of DataStore set-permissions command(Docs)
+ * Fix extras overriding for removed resources (#4042)
+ * Return a 403 if not authorized on the search page (#4081)
+ * Add support for user/pass for Solr as ENV var
+ * Change permission_labels type to string in schema.xml (#3863)
+ * Disallow solr local parameters
+ * Improve text view rendering
+ * Update Orgs/Groups logic for custom fields delete and update (#4094)
+ * Upgrade Solr Docker image
+
 v2.7.2 2017-09-28
 =================
 
@@ -139,6 +363,35 @@ Deprecations:
    jobs (http://docs.ckan.org/en/latest/maintaining/background-tasks.html). Extensions can still
    of course use Celery but they will need to handle the management themselves.
 
+v2.6.6 2018-05-09
+=================
+
+* Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+* Stable version URLs CKAN for documentation (#4209)
+* Add Warning in docs sidebar (#4209)
+
+v2.6.5 2018-03-15
+=================
+
+Note: This version requires a database upgrade
+
+* Activity Time stored in UTC (#2882)
+* Migration script to adjust current activity timestamps to UTC
+* Change CORS header keys and values to string instead of unicode (#3855)
+* Fix cors header when all origins are allowed (#3898)
+* Update SOLR schema.xml reference in Dockerfile
+* Build local SOLR container by default
+* Create datastore indexes only if they don't exist
+* Properly close file responses
+* Use javascript content-type for jsonp responses (#4022)
+* Fix SOLR index delete_package implementation
+* Add second half of DataStore set-permissions command (Docs)
+* Return a 403 if not authorized on the search page (#4081)
+* Add support for user/pass for Solr as ENV var
+* Disallow solr local parameters
+* Improve text view rendering
+* Update Orgs/Groups logic for custom fields delete and update (#4094)
+
 v2.6.4 2017-09-27
 =================
 
@@ -288,6 +541,34 @@ Bug fixes:
 API changes and deprecations:
  * Replace `c.__version__` with new helper `h.ckan_version()` (`#3103 <https://github.com/ckan/ckan/pull/3103>`_)
 
+v2.5.9 2018-05-09
+=================
+
+* Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+* Add Warning in docs sidebar (#4209)
+* Point API docs to stable URL (#4209)
+
+v2.5.8 2018-03-15
+=================
+
+Note: This version requires a database upgrade
+
+* Fix language switcher
+* Activity Time stored in UTC (#2882)
+* Migration script to adjust current activity timestamps to UTC
+* Change CORS header keys and values to string instead of unicode (#3855)
+* Fix cors header when all origins are allowed (#3898)
+* Create datastore indexes only if they are not exist
+* Use javascript content-type for jsonp responses (#4022)
+* Fix SOLR index delete_package implementation
+* Add second half of DataStore set-permissions command(Docs)
+* Update SOLR client (pysolr -> solrpy)
+* Return a 403 if not authorized on the search page (#4081)
+* Add support for user/pass for Solr as ENV var
+* Disallow solr local parameters
+* Improve text view rendering
+* Update Orgs/Groups logic for custom fields delete and update (#4094)
+
 v2.5.7 2017-09-27
 =================
 

diff --git a/Dockerfile b/Dockerfile
@@ -45,6 +45,7 @@ RUN mkdir -p $CKAN_VENV $CKAN_CONFIG $CKAN_STORAGE_PATH && \
 # Setup CKAN
 ADD . $CKAN_VENV/src/ckan/
 RUN ckan-pip install -U pip && \
+    ckan-pip install --upgrade --no-cache-dir -r $CKAN_VENV/src/ckan/requirement-setuptools.txt && \
     ckan-pip install --upgrade --no-cache-dir -r $CKAN_VENV/src/ckan/requirements.txt && \
     ckan-pip install -e $CKAN_VENV/src/ckan/ && \
     ln -s $CKAN_VENV/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini && \

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -19,4 +19,4 @@ include CHANGELOG.txt
 include ckan/migration/migrate.cfg
 include ckan/migration/README
 recursive-include ckan/migration/versions *.sql
-recursive-include ckan_deb *
+include requirement-setuptools.txt
diff --git a/README.rst b/README.rst
@@ -35,8 +35,6 @@ See the `CKAN Documentation <http://docs.ckan.org>`_ for installation instructio
 
 Support
 -------
-For general discussion around CKAN, you can write to the `Google Group`_.
-
 If you need help with CKAN or want to ask a question, use either the
 `ckan-dev`_ mailing list or the `CKAN tag on Stack Overflow`_ (try
 searching the Stack Overflow and ckan-dev `archives`_ for an answer to your
@@ -48,7 +46,6 @@ searching first to see if there's already an issue for your bug).
 If you find a potential security vulnerability please email security@ckan.org,
 rather than creating a public issue on GitHub.
 
-.. _Google Group: https://groups.google.com/forum/#!forum/ckan-global-user-group
 .. _CKAN tag on Stack Overflow: http://stackoverflow.com/questions/tagged/ckan
 .. _ckan-dev: https://lists.okfn.org/mailman/listinfo/ckan-dev
 .. _archives: https://www.google.com/search?q=%22%5Bckan-dev%5D%22+site%3Alists.okfn.org.

diff --git a/ckan/__init__.py b/ckan/__init__.py
@@ -1,6 +1,6 @@
 # encoding: utf-8
 
-__version__ = '2.8.0a'
+__version__ = '2.9.0a'
 
 __description__ = 'CKAN Software'
 __long_description__ = \

diff --git a/ckan/authz.py b/ckan/authz.py
@@ -197,10 +197,11 @@ def is_authorized(action, context, data_dict=None):
         # access straight away
         if not getattr(auth_function, 'auth_allow_anonymous_access', False) \
            and not context.get('auth_user_obj'):
-            return {'success': False,
-                    'msg': '{0} requires an authenticated user'
-                            .format(auth_function)
-                   }
+            return {
+                'success': False,
+                'msg': 'Action {0} requires an authenticated user'.format(
+                    auth_function.__name__)
+            }
 
         return auth_function(context, data_dict)
     else:

diff --git a/ckan/ckan_nose_plugin.py b/ckan/ckan_nose_plugin.py
@@ -94,7 +94,7 @@ def wantFunction(self, fn):
 
     def finalize(self, report):
         if self.segments:
-            print 'Segments: %s' % self.segments
+            print('Segments: %s' % self.segments)
 
     def configure(self, settings, config):
         CkanNose.settings = settings

diff --git a/ckan/common.py b/ckan/common.py
@@ -24,6 +24,8 @@
 
 import simplejson as json
 
+current_app = flask.current_app
+
 try:
     from collections import OrderedDict  # from python 2.7
 except ImportError: