Merged master

.tx/config

@@ -1,7 +1,7 @@
 [main]
 host = https://www.transifex.com
 
-[ckan.2-0]
+[ckan.2-1]
 file_filter = ckan/i18n/<lang>/LC_MESSAGES/ckan.po
 source_file = ckan/i18n/ckan.pot
 source_lang = en

CHANGELOG.rst

@@ -16,6 +16,105 @@ API changes and deprecations:
   called when updating a resource using the web frontend or the
   resource_update API action [#1052]
 
+
+v2.1 2013-08-13
+===============
+
+Note: This version requires a requirements upgrade on source installations
+
+Note: This version requires a database upgrade
+
+Note: This version does not require a Solr schema upgrade
+
+.. note:: The ``json_preview`` plugin has been renamed to ``text_preview``
+ (see #266). If you are upgrading CKAN from a previous version you need
+ to change the plugin name on your CKAN config file after upgrading to avoid
+ a PluginNotFound exception.
+
+
+Major:
+ * Bulk updates of datasets within organizations (delete, make public/private) (#278)
+ * Organizations and Groups search (#303)
+ * Generic text preview extension for JSON, XML and plain text files (#226)
+ * Improve consistency of the Action API (#473)
+ * IAuthenticator interface for plugging into authorization platforms (Work
+   in progress) (#1007)
+ * New clearer dashboard with more information easier to access (#626)
+ * New ``rebuild_fast`` command to speed up reindex using multiple cores (#700)
+ * Complete restructure of the documentation, with updated sections on
+   installation, upgrading, release process, etc and guidelines on how to write
+   new documentation (#769 and multiple others)
+
+Minor:
+ * Add group members page to templates (#844)
+ * Show search facets on organization page (#776)
+ * Changed default sort ordering (#869)
+ * More consistent display of buttons across pages (#890)
+ * History page ported to new templates (#368)
+ * More blocks to templates to allow furhter customization (#688)
+ * Improve imports from lib.helpers (#262)
+ * Add support for callback parameter on Action API (#414)
+ * Create site_user at startup (#952)
+ * Add warning before deleting an organization (#803)
+ * Remove flags from language selector (#822)
+ * Hide the Data API button when datastore is disabled (#752)
+ * Pin all requirements and separate minimal requirements in a separate file (#491, #1149)
+ * Better preview plugin selection (#1002)
+ * Add new functions to the plugins toolkit (#1015)
+ * Improve ExampleIDatasetFormPlugin (#2750)
+ * Extend h.sorted_extras() to do substitutions and auto clean keys (#440)
+ * Separate default database for development and testing (#517)
+ * More descriptive Solr exceptions when indexing (#674)
+ * Validate datastore input through schemas (#905)
+
+Bug fixes:
+ * Fix 500 on password reset (#264)
+ * Fix exception when indexing a wrong date on a _date field (#267)
+ * Fix datastore permissions issues (#652)
+ * Placeholder images are not linked with h.url_for_static (#948)
+ * Explore dropdown menu is hidden behind other resources in IE (#915)
+ * Buttons interrupt file uploading (#902)
+ * Fix resource proxy encoding errors (#896)
+ * Enable streaming in resource proxy (#989)
+ * Fix cache_dir and beaker paths on deployment.ini_tmpl (#888)
+ * Fix multiple issues on create dataset form on IE (#881)
+ * Fix internal server error when adding member (#869)
+ * Fix license faceting (#853)
+ * Fix exception in dashboard (#830)
+ * Fix Google Analytics integration (#827)
+ * Fix ValueError when resource size is not an integer (#1009)
+ * Catch NotFound on new resource when package does not exist (#1010)
+ * Fix Celery configuration to allow overriding from config (#1027)
+ * came_from after login is validated to not redidirect to another site (#1039)
+ * And many, many more!
+
+Deprecated and removed:
+ * The ``json_preview`` plugin has been replaced by a new ``text_preview``
+   one. Please update your config files if using it. (#226)
+
+Known issues:
+ * Under certain authorization setups the frontend for the groups functionality
+   may not work as expected (See #1176 #1175).
+
+
+v2.0.2 2013-08-13
+=================
+
+Bug fixes:
+ * Fix markdown in group descriptions (#303)
+ * Fix resource proxy encoding errors (#896)
+ * Fix datastore exception on first run (#907)
+ * Enable streaming in resource proxy (#989)
+ * Fix in user search (#1024)
+ * Fix Celery configuration to allow overriding from config (#1027)
+ * Undefined function on organizations controller (#1036)
+ * Fix license not translated in orgs/groups (#1040)
+ * Fix link to documentation from the footer (#1062)
+ * Fix missing close breadcrumb tag in org templates (#1071)
+ * Fix recently_changed_packages_activity_stream function (#1159)
+ * Fix Recline map sidebar not showing in IE 7-8 (#1133)
+
+
 v2.0.1 2013-06-11
 =================
 
@@ -190,6 +289,15 @@ Deprecated and removed:
  * Removed deprecated datetime_to_datestr() template helper function (#2257)
 
 
+v1.8.2 2013-08-13
+=================
+
+Bug fixes:
+ * Fix for using harvesters with organization setup
+ * Refactor for user update logic
+ * Tweak resources visibility query
+
+
 v1.8.1 2013-05-10
 =================
 
@@ -255,6 +363,14 @@ API changes and deprecation:
    but is deprecated, and will be removed in future versions. (#2313)
 
 
+v1.7.4 2013-08-13
+=================
+
+Bug fixes:
+ * Refactor for user update logic
+ * Tweak resources visibility query
+
+
 v1.7.3 2013-05-10
 =================
 

CONTRIBUTING.rst

@@ -44,6 +44,7 @@ For contributing translations to CKAN, see
 
    i18n
 
+.. _coding standards:
 
 ----------------
 Coding Standards

bin/travis-build

@@ -23,7 +23,9 @@ sudo -u postgres psql -c "CREATE USER datastore_default WITH PASSWORD 'pass';"
 sudo -u postgres psql -c 'CREATE DATABASE ckan_test WITH OWNER ckan_default;'
 sudo -u postgres psql -c 'CREATE DATABASE datastore_test WITH OWNER ckan_default;'
 
-pip install -r requirements.txt -r dev-requirements.txt --use-mirrors
+export PIP_USE_MIRRORS=true
+pip install -r requirements.txt
+pip install -r dev-requirements.txt
 
 python setup.py develop
 
@@ -68,5 +70,5 @@ NOSE_ERROR=$?
 [ "0" -ne "$MOCHA_ERROR" ] && echo MOCKA tests have failed
 [ "0" -ne "$NOSE_ERROR" ] && echo NOSE tests have failed
 
-# If an error occured in our tests make sure travis knows
+# If an error occurred in our tests make sure travis knows
 exit `expr $MOCHA_ERROR + $NOSE_ERROR`

ckan/config/deployment.ini_tmpl

@@ -133,6 +133,21 @@ ckan.feeds.author_link =
 #ofs.aws_access_key_id = ....
 #ofs.aws_secret_access_key = ....
 
+# 'Bucket' to use for file storage
+#ckan.storage.bucket = default
+
+# Prefix for uploaded files (only used for pairtree)
+#ckan.storage.key_prefix = file/
+
+# The maximum content size, in bytes, for uploads
+#ckan.storage.max_content_length = 50000000
+
+## Datapusher settings
+
+# Make sure you have set up the DataStore
+
+datapusher.formats = csv
+datapusher.url = http://datapusher.ckan.org/
 
 ## Activity Streams Settings
 

ckan/config/environment.py

@@ -373,3 +373,6 @@ def template_loaded(template):
     except sqlalchemy.exc.InternalError:
         # The database is not initialised.  Travis hits this
         pass
+    # if an extension or our code does not finish
+    # transaction properly db cli commands can fail
+    model.Session.remove()

ckan/config/middleware.py

@@ -338,8 +338,8 @@ def __call__(self, environ, start_response):
             key = ''.join([
                 environ['HTTP_USER_AGENT'],
                 environ['REMOTE_ADDR'],
-                environ['HTTP_ACCEPT_LANGUAGE'],
-                environ['HTTP_ACCEPT_ENCODING'],
+                environ.get('HTTP_ACCEPT_LANGUAGE', ''),
+                environ.get('HTTP_ACCEPT_ENCODING', ''),
             ])
             key = hashlib.md5(key).hexdigest()
             # store key/data here

ckan/config/solr/schema-2.0.xml

@@ -129,6 +129,7 @@
     <field name="title_string" type="string" indexed="true" stored="false" />
 
     <field name="data_dict" type="string" indexed="false" stored="true" />
+    <field name="validated_data_dict" type="string" indexed="false" stored="true" />
 
     <field name="_version_" type="string" indexed="true" stored="true"/>
 

ckan/controllers/admin.py

@@ -4,6 +4,7 @@
 import ckan.lib.helpers as h
 import ckan.lib.app_globals as app_globals
 import ckan.model as model
+import ckan.logic as logic
 import ckan.new_authz
 
 c = base.c
@@ -19,8 +20,10 @@ class AdminController(base.BaseController):
     def __before__(self, action, **params):
         super(AdminController, self).__before__(action, **params)
         context = {'model': model,
-                   'user': c.user}
-        if not ckan.new_authz.is_authorized('sysadmin', context, {})['success']:
+                   'user': c.user, 'auth_user_obj': c.userobj}
+        try:
+            logic.check_access('sysadmin', context, {})
+        except logic.NotAuthorized:
             base.abort(401, _('Need to be system administrator to administer'))
         c.revision_change_state_allowed = True
 
@@ -155,4 +158,4 @@ def trash(self):
 
             for msg in msgs:
                 h.flash_error(msg)
-            h.redirect_to(h.url_for('ckanadmin', action='trash'))
+            h.redirect_to(controller='admin', action='trash')

ckan/controllers/api.py

@@ -52,7 +52,8 @@ def __call__(self, environ, start_response):
 
         self._identify_user()
         try:
-            context = {'model': model, 'user': c.user or c.author}
+            context = {'model': model, 'user': c.user or c.author,
+                       'auth_user_obj': c.userobj}
             logic.check_access('site_read', context)
         except NotAuthorized:
             response_msg = self._finish(403,
@@ -161,7 +162,7 @@ def action(self, logic_function, ver=None):
                 _('Action name not known: %s') % logic_function)
 
         context = {'model': model, 'session': model.Session, 'user': c.user,
-                   'api_version': ver}
+                   'api_version': ver, 'auth_user_obj': c.userobj}
         model.Session()._context = context
         return_dict = {'help': function.__doc__}
         try:
@@ -248,7 +249,8 @@ def _get_action_from_map(self, action_map, register, subregister):
 
     def list(self, ver=None, register=None, subregister=None, id=None):
         context = {'model': model, 'session': model.Session,
-                   'user': c.user, 'api_version': ver}
+                   'user': c.user, 'api_version': ver,
+                   'auth_user_obj': c.userobj}
         log.debug('listing: %s' % context)
         action_map = {
             'revision': 'revision_list',
@@ -292,7 +294,7 @@ def show(self, ver=None, register=None, subregister=None,
             action_map[('dataset', type)] = 'package_relationships_list'
 
         context = {'model': model, 'session': model.Session, 'user': c.user,
-                   'api_version': ver}
+                   'api_version': ver, 'auth_user_obj': c.userobj}
         data_dict = {'id': id, 'id2': id2, 'rel': subregister}
 
         log.debug('show: %s' % context)
@@ -327,7 +329,7 @@ def create(self, ver=None, register=None, subregister=None,
             action_map[('dataset', type)] = 'package_relationship_create_rest'
 
         context = {'model': model, 'session': model.Session, 'user': c.user,
-                   'api_version': ver}
+                   'api_version': ver, 'auth_user_obj': c.userobj}
         log.debug('create: %s' % (context))
         try:
             request_data = self._get_request_data()
@@ -390,7 +392,7 @@ def update(self, ver=None, register=None, subregister=None,
             action_map[('dataset', type)] = 'package_relationship_update_rest'
 
         context = {'model': model, 'session': model.Session, 'user': c.user,
-                   'api_version': ver, 'id': id}
+                   'api_version': ver, 'id': id, 'auth_user_obj': c.userobj}
         log.debug('update: %s' % (context))
         try:
             request_data = self._get_request_data()
@@ -442,7 +444,7 @@ def delete(self, ver=None, register=None, subregister=None,
             action_map[('dataset', type)] = 'package_relationship_delete_rest'
 
         context = {'model': model, 'session': model.Session, 'user': c.user,
-                   'api_version': ver}
+                   'api_version': ver, 'auth_user_obj': c.userobj}
 
         data_dict = {'id': id, 'id2': id2, 'rel': subregister}
 
@@ -597,7 +599,7 @@ def tag_counts(self, ver=None):
         c.q = request.params.get('q', '')
 
         context = {'model': model, 'session': model.Session,
-                   'user': c.user or c.author}
+                   'user': c.user or c.author, 'auth_user_obj': c.userobj}
 
         tag_names = get_action('tag_list')(context, {})
         results = []
@@ -632,7 +634,7 @@ def user_autocomplete(self):
         user_list = []
         if q:
             context = {'model': model, 'session': model.Session,
-                       'user': c.user or c.author}
+                       'user': c.user or c.author, 'auth_user_obj': c.userobj}
 
             data_dict = {'q': q, 'limit': limit}
 
@@ -696,7 +698,7 @@ def dataset_autocomplete(self):
         package_dicts = []
         if q:
             context = {'model': model, 'session': model.Session,
-                       'user': c.user or c.author}
+                       'user': c.user or c.author, 'auth_user_obj': c.userobj}
 
             data_dict = {'q': q, 'limit': limit}
 
@@ -712,7 +714,7 @@ def tag_autocomplete(self):
         tag_names = []
         if q:
             context = {'model': model, 'session': model.Session,
-                       'user': c.user or c.author}
+                       'user': c.user or c.author, 'auth_user_obj': c.userobj}
 
             data_dict = {'q': q, 'limit': limit}
 
@@ -731,7 +733,7 @@ def format_autocomplete(self):
         formats = []
         if q:
             context = {'model': model, 'session': model.Session,
-                       'user': c.user or c.author}
+                       'user': c.user or c.author, 'auth_user_obj': c.userobj}
             data_dict = {'q': q, 'limit': limit}
             formats = get_action('format_autocomplete')(context, data_dict)