Merge branch 'smotornyuk-package-blueprint' into package-blueprint-tests

CHANGELOG.rst

@@ -7,11 +7,15 @@
 Changelog
 ---------
 
-v?? (TBA)
-=========
+v.2.8.0 2018-05-09
+==================
 
-Note: This version requires re-running the 'datastore set-permissions' command
-   (assuming you run DataStore). See: :ref:`datastore-set-permissions`
+General notes:
+ * This version requires a requirements upgrade on source installations
+ * This version requires a database upgrade
+ * This version requires a Solr schema upgrade
+ * This version requires re-running the ``datastore set-permissions`` command
+   (assuming you are using the DataStore). See: :ref:`datastore-set-permissions`
 
    Otherwise new and updated datasets will not be searchable in DataStore and
    the logs will contain this error::
@@ -21,7 +25,102 @@ Note: This version requires re-running the 'datastore set-permissions' command
    CKAN developers should also re-run set-permissions on the test database:
    :ref:`datastore-test-set-permissions`
 
+ * There are several old features being officially deprecated starting from
+   this version. Check the *Deprecations* section to be prepared.
+
+Major changes:
+ * New revamped frontend templates based on Bootstrap 3, see "Changes and deprecations" (#3547)
+ * Allow datastore_search_sql on private datasets (#2562)
+ * New Flask blueprints migrated from old Pylons controllers: user, dashboard, feeds, admin and home (#3927, #3870, #3775, #3762)
+ * Improved support for custom groups and organization types (#4032)
+ * Hide user details to anonymous users (#3915)
+
+Minor changes:
+ * Allow chaining of authentication functions (#3679)
+ * Show custom dataset types in search pages (#3807)
+ * Overriding datastore authorization system (#3679)
+ * Standardize on url_for (#3831)
+ * Deprecate notify_after_commit (#3633)
+ *  _mail_recipient header override (#3781)
+ * Restrict access to member forms (#3684)
+ * Clean up template rendering code (#3923)
+ * Permission labels are indexed by type text in SOLR (#3863)
+ * CLI commands require a Flask test request context (#3760)
+ * Allow IValidator to override existing validators (#3865)
+ * Shrink datastore_create response size (#3810)
+ * Stable version URLs CKAN for documentation (#4209)
+ * API Documentation update (#4136)
+ * Documentation of Data Dictionary  (#3989)
+ * Remove datastore legacy mode (#4041)
+ * Map old Pylons routes to Flask ones (#4066)
+
+Bug fixes:
+ * File uploads don't work on new Flask based API (#3869)
+ * {% ckan_extends %} not working on templates served by Flask (#4044)
+ * Problems in background workers with non-core database relations (#3606)
+ * Render_datetime can't handle dates before year 1900 (#2228)
+ * DatapusherPlugin implementation of notify() can call 'datapusher_submit' multiple times (#2334)
+ * Dataset creation page generates incorrect URLs with Chrome autocomplete (#2501)
+ * Search buttons need accessible labels (#2550)
+ * Column name length limit for datastore upload (#2804)
+ * #2373: Do not validate packages or resources from database to views (#3016)
+ * Creation of dataset - different behaviour between Web API & CKAN Interface functionality (#3528)
+ * Redirecting to same page in non-root hosted ckan adds extra root_path to url  (#3499)
+ * Beaker 1.8.0 exception when the code is served from OSX via Vagrant (#3512)
+ * Add "Add Dataset" button to user's and group's page (#2794)
+ * Some links in CKAN is not reachable (#2898)
+ * Exception when specifying a directory in the ckan.i18n_directory option (#3539)
+ * Resource view filter user filters JS error (#3590)
+ * Recaptcha v1 will stop working 2018-3-31 (#4061)
+ * "Testing coding standards" page in docs is missing code snippets (#3635)
+ * Followers count not updated immediately on UI (#3639)
+ * Increase jQuery version (#3665)
+ * Search icon on many pages is not properly vertically aligned (#3654)
+ * Datatables view can't be used as a default view (#3669)
+ * Resource URL is not validated on create/update (#3660)
+ * Upload to Datastore tab shows incorrect time at Upload Log (#3588)
+ * Filter results button is not working (#3593)
+ * Broken link in "Upgrading CKAN’s dependencies" doc page (#3637)
+ * Default logo image not properly saved (#3656)
+ * Activity test relies on datetime.now() (#3644)
+ * Info block text for Format field not properly aligned in resource form page (#3663)
+ * Issue upon creating new organization/group through UI form (#3661)
+ * In API docs "package_create" lists "owner_org" as optional (#3647)
+ * Embed modal window not working (#3731)
+ * Frontent build command does not work on master (#3688)
+ * Loading image duplicated  (#3716)
+ * Datastore set-up error - logging getting in the way (#3694)
+ * Registering a new account redirects to an unprefixed url (#3834)
+ * Exception in search page when not authorized (#4081)
+ * Datastore full-text-search column is populated by postgres trigger rather than python (#3785)
+ * Datastore dump results are not the same as data in database (#4150)
+ * Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+ * No such file or directory: '/usr/lib/ckan/default/src/ckan/requirement-setuptools.txt' during installation from source (#3641)
+ * Register user form missing required field indicators (#3658)
+ * Datastore full-text-search column is populated by postgres trigger rather than python  (#3786)
+ * Add missing major changes to change log (#3799)
+ * Paster/CLI config-tool requires _get_test_app which in turn requires a dev-only dependency (#3806)
+ * Change log doesn't mention necessary Solr scheme upgrade (#3851)
+ * TypeError: expected byte string object, value of type unicode found (#3921)
+ * CKAN's state table clashes with PostGIS generated TIGER state table (#3929)
+ * [Docker] entrypoint initdb.d sql files copied to root (#3939)
+ * DataStore status page throws TypeError - Bleach upgrade regression (#3968)
+ * Source install error with who.ini (#4020)
+ * making a JSONP call to the CKAN API returns the wrong mime type (#4022)
+ * Deleting a resource sets datastore_active=False to all resources and overrides their extras (#4042)
+ * Deleting first Group and Organization custom field is not possible (#4094)
+
 Changes and deprecations:
+ * The default templates included in CKAN core have been updated to use Bootstrap 3. Extensions
+   implementing custom themes are encouraged to update their templates, but they can still
+   make CKAN load the old Bootstrap 2 templates during the transition using the following
+   configuration options::
+
+        ckan.base_public_folder = public-bs2
+        ckan.base_templates_folder = templates-bs2
+
+ * The API versions 1 and 2 (also known as the REST API), ie ``/api/rest/*`` have been
+   completely removed in favour of the version 3 (action API, ``/api/action/*``).
  * The old Celery based background jobs have been removed in CKAN 2.8 in favour of the new RQ based
    jobs (http://docs.ckan.org/en/latest/maintaining/background-tasks.html). Extensions can still
    of course use Celery but they will need to handle the management themselves.
@@ -31,6 +130,13 @@ Changes and deprecations:
    This change is aimed to reduce usage of global variables in context. For a while, it has default value
    of None, in which case, `c.search_facets` will be used. But all template designers are strongly advised
    to specify this argument explicitly, as in future it'll become required.
+ * The ``ckan.recaptcha.version`` config option is now removed, since v2 is the only valid version now (#4061)
+
+v2.7.4 2018-05-09
+=================
+
+ * Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+ * Datastore dump results are not the same as data in database (#4150)
 
 v2.7.3 2018-03-15
 =================
@@ -209,6 +315,13 @@ Deprecations:
    jobs (http://docs.ckan.org/en/latest/maintaining/background-tasks.html). Extensions can still
    of course use Celery but they will need to handle the management themselves.
 
+v2.6.6 2018-05-09
+=================
+
+* Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+* Stable version URLs CKAN for documentation (#4209)
+* Add Warning in docs sidebar (#4209)
+
 v2.6.5 2018-03-15
 =================
 
@@ -380,6 +493,13 @@ Bug fixes:
 API changes and deprecations:
  * Replace `c.__version__` with new helper `h.ckan_version()` (`#3103 <https://github.com/ckan/ckan/pull/3103>`_)
 
+v2.5.9 2018-05-09
+=================
+
+* Adding filter at resoruce preview doesn't work while site is setup with ckan.root_path param (#4140)
+* Add Warning in docs sidebar (#4209)
+* Point API docs to stable URL (#4209)
+
 v2.5.8 2018-03-15
 =================
 

MANIFEST.in

@@ -19,4 +19,4 @@ include CHANGELOG.txt
 include ckan/migration/migrate.cfg
 include ckan/migration/README
 recursive-include ckan/migration/versions *.sql
-recursive-include ckan_deb *
+include requirement-setuptools.txt

ckan/config/routing.py

@@ -238,10 +238,6 @@ def make_map():
         m.connect('/revision/list', action='list')
         m.connect('/revision/{id}', action='read')
 
-    with SubMapper(map, controller='ckan.controllers.storage:StorageController') as m:
-        m.connect('storage_file', '/storage/f/{label:.*}',
-                  action='file')
-
     with SubMapper(map, controller='util') as m:
         m.connect('/i18n/strings_{lang}.js', action='i18n_js_strings')
         m.connect('/util/redirect', action='redirect')

ckan/lib/helpers.py

@@ -274,7 +274,7 @@ def url_for(*args, **kw):
 
     URLs built by Pylons use the Routes syntax::
 
-        url_for(controller='my_controller', action='my_action', id='my_dataset')
+        url_for(controller='my_ctrl', action='my_action', id='my_dataset')
         # Returns '/dataset/my_dataset'
 
     Or, using a named route::

ckan/public-bs2/base/javascript/modules/resource-view-filters-form.js

@@ -19,26 +19,30 @@ ckan.module('resource-view-filters-form', function (jQuery) {
                 query;
 
             query = {
-              plain: false,
               resource_id: resourceId,
               limit: queryLimit,
               offset: offset,
               fields: filterName,
               distinct: true,
-              sort: filterName
+              sort: filterName,
+              include_total: false
             };
 
             if (term !== '') {
               var q = {};
-              q[filterName] = term + ':*';
+              if (term.indexOf(' ') == -1) {
+                term = term + ':*';
+                query.plain = false;
+              }
+              q[filterName] = term;
               query.q = JSON.stringify(q);
             }
 
             return query;
         },
         results: function (data, page) {
           var records = data.result.records,
-              hasMore = (records.length < data.result.total),
+              hasMore = (records.length == queryLimit),
               results;
 
           results = $.map(records, function (record) {

ckan/public-bs2/base/javascript/modules/resource-view-filters.js

@@ -103,26 +103,31 @@ this.ckan.module('resource-view-filters', function (jQuery) {
                 query;
 
             query = {
-              plain: false,
               resource_id: resourceId,
               limit: queryLimit,
               offset: offset,
               fields: filterName,
               distinct: true,
-              sort: filterName
+              sort: filterName,
+              include_total: false
             };
 
+
             if (term !== '') {
               var q = {};
-              q[filterName] = term + ':*';
+              if (term.indexOf(' ') == -1) {
+                term = term + ':*';
+                query.plain = false;
+              }
+              q[filterName] = term;
               query.q = JSON.stringify(q);
             }
 
             return query;
           },
           results: function (data, page) {
             var records = data.result.records,
-                hasMore = (records.length < data.result.total),
+                hasMore = (records.length == queryLimit),
                 results;
 
             results = $.map(records, function (record) {

ckan/public/base/javascript/modules/autocomplete.js

@@ -2,14 +2,15 @@
  * a list of terms from an API endpoint (provided using data-module-source).
  *
  * source   - A url pointing to an API autocomplete endpoint.
- * interval - The interval between requests in milliseconds (default: 1000).
+ * interval - The interval between requests in milliseconds (default: 300).
  * items    - The max number of items to display (default: 10)
  * tags     - Boolean attribute if true will create a tag input.
  * key      - A string of the key you want to be the form value to end up on
  *            from the ajax returned results
  * label    - A string of the label you want to appear within the dropdown for
  *            returned results
- *
+ * tokensep - A string that contains characters which will be interpreted
+ *            as separators for tags when typed or pasted (default ",").
  * Examples
  *
  *   // <input name="tags" data-module="autocomplete" data-module-source="http://" />
@@ -24,6 +25,7 @@ this.ckan.module('autocomplete', function (jQuery) {
       label: false,
       items: 10,
       source: null,
+      tokensep: ',',
       interval: 300,
       dropdownClass: '',
       containerClass: ''
@@ -50,7 +52,8 @@ this.ckan.module('autocomplete', function (jQuery) {
         formatNoMatches: this.formatNoMatches,
         formatInputTooShort: this.formatInputTooShort,
         dropdownCssClass: this.options.dropdownClass,
-        containerCssClass: this.options.containerClass
+        containerCssClass: this.options.containerClass,
+        tokenSeparators: this.options.tokensep.split('')
       };
 
       // Different keys are required depending on whether the select is