conflict in cli/cli

.github/ISSUE_TEMPLATE.md

@@ -1,3 +1,6 @@
+**Note:** Are you submitting a security related issue that could be a potential vulnerability? Please send it to security@ckan.org instead.
+
+
 ### CKAN Version if known (or site URL)
 
 

.travis.yml

@@ -1,10 +1,22 @@
 group: travis_latest
 language: python
 
+flake8-steps: &flake8-steps
+  env: FLAKE8=true
+  cache: pip
+  install: pip install flake8
+  before_script:
+    - flake8 --version
+    # stop the build if there are Python syntax errors or undefined names
+    - flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics --exclude ./ckan/include/rjsmin.py
+    # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
+    - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+  script:
+    - true
+
 matrix:
   include:
     - python: "2.7"
-      sudo: required
 
       services:
         - docker
@@ -29,31 +41,8 @@ matrix:
         - docker ps -a
 
     - python: "2.7"
-      env: FLAKE8=true
-      cache: pip
-      install:
-        - pip install flake8
-      before_script:
-        - flake8 --version
-        # stop the build if there are Python syntax errors or undefined names
-        - flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
-        # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
-        - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-      script:
-        - true
+      <<: *flake8-steps
 
     - python: "3.7"
-      env: FLAKE8=true
       dist: xenial    # required for Python 3.7
-      sudo: required  # required for Python 3.7
-      cache: pip
-      install:
-        - pip install flake8
-      before_script:
-        - flake8 --version
-        # stop the build if there are Python syntax errors or undefined names
-        - flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
-        # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
-        - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-      script:
-        - true
+      <<: *flake8-steps

CHANGELOG.rst

@@ -10,22 +10,58 @@ Changelog
 v.2.9.0 TBA
 ==================
 
+ * When upgrading from previous CKAN versions, the Activity Stream needs a
+   migrate_package_activity.py running for displaying the history of dataset
+   changes. This can be performed while CKAN is running or stopped (whereas the
+   standard `paster db upgrade` migrations need CKAN to be stopped). Ideally it
+   is run before CKAN is upgraded, but it can be run afterwards. If running
+   previous versions or this version of CKAN, download and run
+   migrate_package_activity.py like this:
+
+     cd /usr/lib/ckan/default/src/ckan/
+     wget https://raw.githubusercontent.com/ckan/ckan/3484_revision_ui_removal2/ckan/migration/migrate_package_activity.py
+     wget https://raw.githubusercontent.com/ckan/ckan/3484_revision_ui_removal2/ckan/migration/revision_legacy_code.py
+     python migrate_package_activity.py -c /etc/ckan/production.ini
+
+   Future versions of CKAN are likely to need a slightly different procedure.
+   Full info about this migration is found here:
+   https://github.com/ckan/ckan/wiki/Migrate-package-activity
+
+ * A full history of dataset changes is now displayed in the Activity Stream to
+   admins, and optionally to the public. By default this is enabled for new
+   installs, but disabled for sites which upgrade (just in case the history is
+   sensitive). When upgrading, open data CKANs are encouraged to make this
+   history open to the public, by setting this in production.ini:
+   ``ckan.auth.public_activity_stream_detail = true`` (#3972)
+
 Minor changes:
 
  * For navl schemas, the 'default' validator no longer applies the default when
    the value is False, 0, [] or {} (#4448)
  * If you've customized the schema for package_search, you'll need to add to it
    the limiting of ``row``, as per default_package_search_schema now does (#4484)
- * Several logic functions now have new limits to how many items can be
-   returned, notably ``group_list`` and ``organization_list`` when
-   ``all_fields=true``. These are all configurable. (#4484)
+ * Several logic functions now have new upper limits to how many items can be
+   returned, notably ``group_list``, ``organization_list`` when
+   ``all_fields=true``, ``datastore_search`` and ``datastore_search_sql``.
+   These are all configurable. (#4484, #4562)
 
 Bugfixes:
 
  * Action function "datastore_search" would calculate the total, even if you
    set include_total=False (#4448)
 
-Deprecations:
+Removals and deprecations:
+
+ * Revision and History UI is removed: `/revision/*` & `/dataset/{id}/history`
+   in favour of `/dataset/changes/` visible in the Activity Stream. (#3972)
+ * Logic functions removed:
+   ``dashboard_activity_list_html`` ``organization_activity_list_html``
+   ``user_activity_list_html`` ``package_activity_list_html``
+   ``group_activity_list_html`` ``organization_activity_list_html``
+   ``recently_changed_packages_activity_list_html``
+   ``dashboard_activity_list_html`` ``activity_detail_list`` (#4627/#3972)
+ * ``model.ActivityDetail`` is no longer used and will be removed in the next
+   CKAN release. (#3972)
  * ``c.action`` and ``c.controller`` variables should be avoided.
    ``ckan.plugins.toolkit.get_endpoint`` can be used instead. This function
    returns tuple of two items(depending on request handler):

bin/remove-old-git-branches.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+# Change this variable if you are using another name for ckan/ckan's remote
+REMOTE=origin
+PR_ENDPOINT=https://api.github.com/repos/ckan/ckan/pulls
+
+prepare_branches() {
+  echo List existing branches
+  # awk filters out `master`, `release-something`, `dev-something` and `MAJOR.MINOR-something` branches
+  custom_branches=$(git fetch $REMOTE && git branch -r --list $REMOTE/* | awk '{ if ($2 !~ /^[0-9]\.[0-9]|^release|^dev-|^master$|HEAD/) print $2}' FS=/)
+
+  echo Calculate total amount of active PRs
+  # github adds Link header, that contains reference to last page with results. And that helps us to define total number of pages
+  last_page=$(curl -s -I $PR_ENDPOINT | grep '^Link' | sed -E 's/.*page=([[:digit:]]+).*rel="last".*/\1/')
+
+  echo Compute branches that are in use by active PRs
+  for ((i=1; i<=$last_page; i++))
+  do
+    # Take JSON, get name of branch with feature, leave only those ones, that starts with `ckan:`. I think, that jq is much more nice way to process json, but i'm not sure, that every one has it. But if you do,
+    # you can replace python's section with something like `| jq '.[] | .head.label' -r |`
+    next_portion=$(curl -s "$PR_ENDPOINT?page=$i" | python -c 'import json, sys; raw=sys.stdin.read(); labels=[str(item["head"]["label"]) for item in json.loads(raw)]; print("\n".join(labels))' | grep '^ckan' | sed 's/^ckan://')
+    used_branches="$used_branches\n$next_portion"
+    echo [$i of $last_page parts processed]
+  done;
+
+  echo Compute list of branches that can be safetely removed
+  tmp_file=$(mktemp)
+  diff <(echo -e "$used_branches" | sort) <(echo -e "$custom_branches" | sort) --unchanged-line-format="" >> $tmp_file
+
+  echo
+  echo List of all branches that can be removed stored into $tmp_file. Review it\'s content and run \'$0 --apply $tmp_file\'.
+  echo NOTE. This command requires write access to CKAN repository and will actually remove all branches that are listed in $tmp_file.
+}
+
+if [[ $# -gt 0 ]];
+then
+  case "$1" in
+    -h|--help)
+      echo Compute/remove unused branches
+      echo Usage: $0 [--apply FILE_WITH_BRANCHES]
+      ;;
+    --apply)
+      shift
+      if [[ -f $1 ]]
+      then
+        while read branch
+        do
+          git push $REMOTE --delete $branch
+        done < $1
+      else
+        echo File does not exist
+      fi
+      shift
+      ;;
+    *)
+      echo Check usage with: $0 --help
+      ;;
+  esac
+else
+  prepare_branches
+fi

ckan/authz.py

@@ -102,13 +102,19 @@ def _build(self):
                     fetched_auth_functions[name] = auth_function
 
         for name, func_list in chained_auth_functions.iteritems():
-            if name not in fetched_auth_functions:
+            if (name not in fetched_auth_functions and
+                    name not in self._functions):
                 raise Exception('The auth %r is not found for chained auth' % (
                     name))
             # create the chain of functions in the correct order
             for func in reversed(func_list):
-                prev_func = fetched_auth_functions[name]
-                fetched_auth_functions[name] = functools.partial(func, prev_func)
+                if name in fetched_auth_functions:
+                    prev_func = fetched_auth_functions[name]
+                else:
+                    # fallback to chaining off the builtin auth function
+                    prev_func = self._functions[name]
+                fetched_auth_functions[name] = (
+                    functools.partial(func, prev_func))
 
         # Use the updated ones in preference to the originals.
         self._functions.update(fetched_auth_functions)
@@ -200,7 +206,7 @@ def is_authorized(action, context, data_dict=None):
             return {
                 'success': False,
                 'msg': 'Action {0} requires an authenticated user'.format(
-                    auth_function.__name__)
+                    action)
             }
 
         return auth_function(context, data_dict)
@@ -401,6 +407,7 @@ def get_user_id_for_username(user_name, allow_none=False):
     'create_user_via_api': False,
     'create_user_via_web': True,
     'roles_that_cascade_to_sub_groups': 'admin',
+    'public_activity_stream_detail': False,
 }
 
 

ckan/cli/cli.py

@@ -3,11 +3,15 @@
 import logging
 
 import click
-
+from ckan.cli import config_tool
 from ckan.cli import (
-    click_config_option, db, load_config, search_index, server, asset
+    click_config_option, db, load_config, search_index, server,
+    asset,
+    translation,
 )
+
 from ckan.config.middleware import make_app
+from ckan.cli import seed
 
 log = logging.getLogger(__name__)
 
@@ -27,7 +31,10 @@ def ckan(ctx, config, *args, **kwargs):
     ctx.obj = CkanCommand(config)
 
 
+ckan.add_command(config_tool.config_tool)
 ckan.add_command(server.run)
+ckan.add_command(seed.seed)
 ckan.add_command(db.db)
 ckan.add_command(search_index.search_index)
 ckan.add_command(asset.asset)
+ckan.add_command(translation.translation)

ckan/cli/config_tool.py

@@ -0,0 +1,77 @@
+# encoding: utf-8
+
+import logging
+
+import click
+
+from ckan.cli import error_shout
+import ckan.lib.config_tool as ct
+
+log = logging.getLogger(__name__)
+
+
+class ConfigOption(click.ParamType):
+    name = u'config-option'
+
+    def convert(self, value, param, ctx):
+        if u'=' not in value:
+            self.fail(
+                u'An option does not have an equals sign. '
+                u'It should be \'key=value\'. If there are spaces '
+                u'you\'ll need to quote the option.\n'
+            )
+        return value
+
+
+@click.command(
+    name=u'config-tool',
+    short_help=u'Tool for editing options in a CKAN config file.'
+)
+@click.option(
+    u'--section',
+    u'-s',
+    default=u'app:main',
+    help=u'Section of the config file'
+)
+@click.option(
+    u'--edit',
+    u'-e',
+    is_flag=True,
+    help=u'Checks the option already exists in the config file.'
+)
+@click.option(
+    u'--file',
+    u'-f',
+    u'merge_filepath',
+    help=u'Supply an options file to merge in.'
+)
+@click.argument(u'config_filepath', type=click.Path(exists=True))
+@click.argument(u'options', nargs=-1, type=ConfigOption())
+def config_tool(config_filepath, options, section, edit, merge_filepath):
+    u'''Tool for editing options in a CKAN config file
+
+    paster config-tool <default.ini> <key>=<value> [<key>=<value> ...]
+
+    paster config-tool <default.ini> -f <custom_options.ini>
+
+    Examples:
+
+      paster config-tool default.ini sqlalchemy.url=123 'ckan.site_title=ABC'
+
+      paster config-tool default.ini -s server:main -e port=8080
+
+      paster config-tool default.ini -f custom_options.ini
+    '''
+
+    if merge_filepath:
+        ct.config_edit_using_merge_file(
+            config_filepath, merge_filepath
+        )
+    if not (options or merge_filepath):
+        return error_shout(u'No options provided')
+    try:
+        ct.config_edit_using_option_strings(
+            config_filepath, options, section, edit=edit
+        )
+    except ct.ConfigToolError as e:
+        error_shout(e)

ckan/cli/search_index.py

@@ -93,9 +93,9 @@ def start(ids):
     def chunks(l, n):
         u""" Yield n successive chunks from l."""
         newn = int(len(l) / n)
-        for i in range(0, n-1):
-            yield l[i*newn:i*newn+newn]
-        yield l[n*newn-newn:]
+        for i in range(0, n - 1):
+            yield l[i * newn:i * newn + newn]
+        yield l[n * newn - newn:]
 
     processes = []
     with flask_app.test_request_context():