Merge branch 'master' into 2493-stats

Conflicts:
	ckanext/stats/templates_legacy/ckanext/stats/index.html

CHANGELOG.rst

@@ -7,12 +7,94 @@
 Changelog
 ---------
 
-v2.4
-====
+v2.5.0 XXXX-XX-XX
+=================
+
+Changes and deprecations
+------------------------
+
+* The old RDF templates to output a dataset in RDF/XML or N3 format have been
+  removed. These can be now enabled using the ``dcat`` plugin on *ckanext-dcat*:
+
+    https://github.com/ckan/ckanext-dcat#rdf-dcat-endpoints
+
+
+v2.4.1 2015-09-02
+=================
+
+Note: #2554 fixes a regression where ``group_list`` and ``organization_list``
+      where returning extra additional fields by default, causing performance
+      issues. This is now fixed, so the output for these actions no longer returns
+      ``users``, ``extras``, etc.
+      Also, on the homepage template the ``c.groups`` and ``c.group_package_stuff``
+      context variables are no longer available.
+
+
+Bug fixes:
+
+* Fix dataset count in templates and show datasets on featured org/group (#2557)
+* Fix autodetect for TSV resources (#2553)
+* Improve character escaping in DataStore parameters
+* Fix "paster db init" when celery is configured with a non-database backend
+* Fix severe performance issues with groups and orgs listings (#2554)
+
+
+v2.4.0 2015-07-22
+=================
+
+Note: This version requires a database upgrade
+
+Note: This version requires a Solr schema upgrade
+
+Major:
+ * CKAN config can now be set from environment variables and via the API (#2429)
+
+Minor:
+ * API calls now faster: ``group_show``, ``organization_show``, ``user_show``,
+   ``package_show``, ``vocabulary_show`` & ``tag_show`` (#1886, #2206, #2207,
+   #2376)
+ * Require/validate current password before allowing a password change (#1940)
+ * Added ``organization_autocomplete`` action (#2125)
+ * Default authorization no longer allows anyone to create datasets etc (#2164)
+ * ``organization_list_for_user`` now returns organizations in hierarchy if they
+   exist for roles set in ``ckan.auth.roles_that_cascade_to_sub_groups`` (#2199)
+ * Improved accessibility (text based browsers) focused on the page header
+   (#2258)
+ * Improved IGroupForm for better customizing groups and organization behaviour
+   (#2354)
+ * Admin page can now be extended to have new tabs (#2351)
+
+
+Bug fixes:
+ * Command line ``paster user`` failed for non-ascii characters (#1244)
+ * Memory leak fixed in datastore API (#1847)
+ * Modifying resource didn't update it's last updated timestamp (#1874)
+ * Datastore didn't update if you uploaded a new file of the same name as the
+   existing file (#2147)
+ * Files with really long file were skipped by datapusher (#2057)
+ * Multi-lingual Solr schema is now updated so it works again (#2161)
+ * Resource views didn't display when embedded in another site (#2238)
+ * ``resource_update`` failed if you supplied a revision_id (#2340)
+ * Recline could not plot GeoJSON on a map (#2387)
+ * Dataset create form 404 error if you added a resource but left it blank (#2392)
+ * Editing a resource view for a file that was UTF-8 and had a BOM gave an
+   error (#2401)
+ * Email invites had the email address changed to lower-case (#2415)
+ * Default resource views not created when using a custom dataset schema (#2421,
+   #2482)
+ * If the licenses pick-list was customized to remove some, datasets with old
+   values had them overwritten when edited (#2472)
+ * Recline views failed on some non-ascii characters (#2490)
+ * Resource proxy failed if HEAD responds with 403 (#2530)
+ * Resource views for non-default dataset types couldn't be created (#2532)
 
 Changes and deprecations
 ------------------------
 
+* The default of allowing anyone to create datasets, groups and organizations
+  has been changed to False. It is advised to ensure you set all of the
+  :ref:`config-authorization` options explicitly in your CKAN config. (#2164)
+
 * The ``package_show`` API call does not return the ``tracking_summary``,
   keys in the dataset or resources by default any more.
 
@@ -23,27 +105,49 @@ Changes and deprecations
   `new_tests` directory has moved to `tests` and the `new_authz.py`
   module has been renamed `authz.py`. Code that imports names from the
   old locations will continue to work in this release but will issue
-  a deprecation warning.
+  a deprecation warning. (#1753)
 
-* Add text to account links in header, fixes text based browser support #2258
+* ``group_show`` and ``organization_show`` API calls no longer return the
+  datasets by default (#2206)
 
-* Add middleware that cleans up the response string after it has been
-  served, stabilizes memory usage for large requests #1847
+  Custom templates or users of this API call will need to pass
+  ``include_datasets=True`` to include datasets in the response.
 
 * The ``vocabulary_show`` and ``tag_show`` API calls no longer returns the 
   ``packages`` key - i.e. datasets that use the vocabulary or tag. 
   However ``tag_show`` now has an ``include_datasets`` option. (#1886)
 
-* `organization_list_for_user` now returns organizations in hierarchy if they
-  exist for roles set in `ckan.auth.roles_that_cascade_to_sub_groups`.
+* Config option ``site_url`` is now required - CKAN will not abort during
+  start-up if it is not set. (#1976)
 
-* Update license keys to match opendefinition.org #2110
 
-* The ``group_show`` and ``organization_show`` API calls do not return
-  ``datasets`` by default any more.
+v2.3.2 2015-09-02
+=================
+
+Bug fixes:
+* Fix autodetect for TSV resources (#2553)
+* Improve character escaping in DataStore parameters
+* Fix "paster db init" when celery is configured with a non-database backend
+
+
+v2.3.1 2015-07-22
+=================
+
+Bug fixes:
+ * Resource views won't display when embedded in another site (#2238)
+ * ``resource_update`` failed if you supplied a revision_id (#2340)
+ * Recline could not plot GeoJSON on a map (#2387)
+ * Dataset create form 404 error if you added a resource but left it blank (#2392)
+ * Editing a resource view for a file that was UTF-8 and had a BOM gave an
+   error (#2401)
+ * Email invites had the email address changed to lower-case (#2415)
+ * Default resource views not created when using a custom dataset schema (#2421,
+   #2482)
+ * If the licenses pick-list was customized to remove some, datasets with old
+   values had them overwritten when edited (#2472)
+ * Recline views failed on some non-ascii characters (#2490)
+ * Resource views for non-default dataset types couldn't be created (#2532)
 
-  Custom templates or users of this API call will need to pass
-  ``include_datasets=True`` to include datasets in the response.
 
 v2.3 2015-03-04
 ===============
@@ -200,7 +304,6 @@ Bug fixes:
  * Make resource_create auth work against package_update (#2037)
  * Fix DataStore permissions check on startup (#1374)
  * Fix datastore docs link (#2044)
- * Fix resource extras getting lost on resource update (#2158)
  * Clean up field names before rendering the Recline table (#2319)
  * Don't "normalize" resource URL in recline view (#2324)
  * Don't assume resource format is there on text preview (#2320)
@@ -341,6 +444,17 @@ Troubleshooting:
 
   Also see the previous point for other ``who.ini`` changes.
 
+v2.2.3 2015-07-22
+=================
+
+Bug fixes:
+ * Allow uppercase emails on user invites (#2415)
+ * Fix broken boolean validator (#2443)
+ * Fix auth check in resources_list.html (#2037)
+ * Key error on resource proxy (#2425)
+ * Ignore revision_id passed to resources (#2340)
+ * Add reset for reset_key on successful password change (#2379)
+
 v2.2.2 2015-03-04
 =================
 
@@ -562,6 +676,15 @@ Troubleshooting:
    leaving the fields empty. Also make sure to restart running processes like
    harvesters after the update to make sure they use the new code base.
 
+v2.1.5 2015-07-22
+=================
+
+Bug fixes:
+ * Fix broken boolean validator (#2443)
+ * Key error on resource proxy (#2425)
+ * Ignore revision_id passed to resources (#2340)
+ * Add reset for reset_key on successful password change (#2379)
+
 v2.1.4 2015-03-04
 =================
 
@@ -720,6 +843,15 @@ Known issues:
  * Under certain authorization setups the frontend for the groups functionality
    may not work as expected (See #1176 #1175).
 
+v2.0.7 2015-07-22
+=================
+
+Bug fixes:
+ * Fix broken boolean validator (#2443)
+ * Key error on resource proxy (#2425)
+ * Ignore revision_id passed to resources (#2340)
+ * Add reset for reset_key on successful password change (#2379)
+
 v2.0.6 2015-03-04
 =================
 

bin/less

@@ -15,10 +15,6 @@ function compile(event, filename) {
   var start = Date.now(),
       filename = 'main.css';
 
-  if (debug) {
-    filename = 'main.debug.css';
-  }
-
   exec('`npm bin`/lessc ' + __dirname + '/../ckan/public/base/less/main.less > ' + __dirname + '/../ckan/public/base/css/' + filename, function (err, stdout, stderr) {
     var duration = Date.now() - start;
 

ckan/authz.py

@@ -81,7 +81,6 @@ def _build(self):
                             resolved_auth_function_plugins[name]
                         )
                     )
-                log.debug('Auth function {0} from plugin {1} was inserted'.format(name, plugin.name))
                 resolved_auth_function_plugins[name] = plugin.name
                 fetched_auth_functions[name] = auth_function
         # Use the updated ones in preference to the originals.

ckan/config/deployment.ini_tmpl

@@ -111,6 +111,7 @@ ckan.favicon = /images/icons/ckan.ico
 ckan.gravatar_default = identicon
 ckan.preview.direct = png jpg gif
 ckan.preview.loadable = html htm rdf+xml owl+xml xml n3 n-triples turtle plain atom csv tsv rss txt json
+ckan.display_timezone = server
 
 # package_hide_extras = for_search_index_only
 #package_edit_return_url = http://another.frontend/dataset/<NAME>

ckan/config/resource_formats.json

@@ -18,7 +18,7 @@
   ["MDB", "Access Database", "application/x-msaccess", []],
   ["NetCDF", "NetCDF File", "application/netcdf", []],
   ["ArcGIS Map Service", "ArcGIS Map Service", "ArcGIS Map Service", ["arcgis map service"]],
-  ["TSV", "Tab Separated Values File", "text/tsv", []],
+  ["TSV", "Tab Separated Values File", "text/tab-separated-values", ["text/tsv"]],
   ["WFS", "Web Feature Service", null, []],
   ["ArcGIS Online Map", "ArcGIS Online Map", "ArcGIS Online Map", ["web map application"]],
   ["Perl", "Perl Script", "text/x-perl", []],

ckan/controllers/api.py

@@ -83,7 +83,9 @@ def _finish(self, status_int, response_data=None,
         if response_data is not None:
             response.headers['Content-Type'] = CONTENT_TYPES[content_type]
             if content_type == 'json':
-                response_msg = h.json.dumps(response_data)
+                response_msg = h.json.dumps(
+                    response_data,
+                    for_json=True) # handle objects with for_json methods
             else:
                 response_msg = response_data
             # Support "JSONP" callback.
@@ -164,7 +166,8 @@ def action(self, logic_function, ver=None):
                 _('Action name not known: %s') % logic_function)
 
         context = {'model': model, 'session': model.Session, 'user': c.user,
-                   'api_version': ver, 'auth_user_obj': c.userobj}
+                   'api_version': ver, 'return_type': 'LazyJSONObject',
+                   'auth_user_obj': c.userobj}
         model.Session()._context = context
 
         return_dict = {'help': h.url_for(controller='api',
@@ -730,7 +733,8 @@ def dataset_autocomplete(self):
         return self._finish_ok(resultSet)
 
     def tag_autocomplete(self):
-        q = request.params.get('incomplete', '')
+        q = request.str_params.get('incomplete', '')
+        q = unicode(urllib.unquote(q), 'utf-8')
         limit = request.params.get('limit', 10)
         tag_names = []
         if q:

ckan/controllers/group.py

@@ -150,15 +150,15 @@ def add_group_type(cls, group_type):
     def index(self):
         group_type = self._guess_group_type()
 
+        page = self._get_page_number(request.params) or 1
+        items_per_page = 21
+
         context = {'model': model, 'session': model.Session,
                    'user': c.user or c.author, 'for_view': True,
                    'with_private': False}
 
         q = c.q = request.params.get('q', '')
-        data_dict = {'all_fields': True, 'q': q, 'type': group_type or 'group'}
         sort_by = c.sort_by_selected = request.params.get('sort')
-        if sort_by:
-            data_dict['sort'] = sort_by
         try:
             self._check_access('site_read', context)
         except NotAuthorized:
@@ -170,14 +170,34 @@ def index(self):
             context['user_id'] = c.userobj.id
             context['user_is_admin'] = c.userobj.sysadmin
 
-        results = self._action('group_list')(context, data_dict)
+        data_dict_global_results = {
+            'all_fields': False,
+            'q': q,
+            'sort': sort_by,
+            'type': group_type or 'group',
+        }
+        global_results = self._action('group_list')(context,
+                                                    data_dict_global_results)
+
+        data_dict_page_results = {
+            'all_fields': True,
+            'q': q,
+            'sort': sort_by,
+            'type': group_type or 'group',
+            'limit': items_per_page,
+            'offset': items_per_page * (page - 1),
+        }
+        page_results = self._action('group_list')(context,
+                                                  data_dict_page_results)
 
         c.page = h.Page(
-            collection=results,
-            page = self._get_page_number(request.params),
+            collection=global_results,
+            page=page,
             url=h.pager_url,
-            items_per_page=21
+            items_per_page=items_per_page,
         )
+
+        c.page.items = page_results
         return render(self._index_template(group_type),
                       extra_vars={'group_type': group_type})
 
@@ -625,9 +645,9 @@ def delete(self, id):
         try:
             if request.method == 'POST':
                 self._action('group_delete')(context, {'id': id})
-                if self.group_type == 'organization':
+                if group_type == 'organization':
                     h.flash_notice(_('Organization has been deleted.'))
-                elif self.group_type == 'group':
+                elif group_type == 'group':
                     h.flash_notice(_('Group has been deleted.'))
                 else:
                     h.flash_notice(_('%s has been deleted.')