Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/master' into 4556-view-filter-co…
Browse files Browse the repository at this point in the history
…ntrols
  • Loading branch information
wardi committed Jun 21, 2019
2 parents 69864d7 + 9783a3d commit f3a78e2
Show file tree
Hide file tree
Showing 1,151 changed files with 24,270 additions and 144,146 deletions.
3 changes: 3 additions & 0 deletions .github/ISSUE_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
**Note:** Are you submitting a security related issue that could be a potential vulnerability? Please send it to security@ckan.org instead.


### CKAN Version if known (or site URL)


Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ tmp/*
solr_runtime/*
fl_notes.txt
*.ini
!ckan/migration/alembic.ini
.noseids
*~
.idea
Expand Down
41 changes: 15 additions & 26 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,22 @@
group: travis_latest
language: python

flake8-steps: &flake8-steps
env: FLAKE8=true
cache: pip
install: pip install flake8
before_script:
- flake8 --version
# stop the build if there are Python syntax errors or undefined names
- flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics --exclude ./ckan/include/rjsmin.py
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
- flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
script:
- true

matrix:
include:
- python: "2.7"
sudo: required

services:
- docker
Expand All @@ -29,31 +41,8 @@ matrix:
- docker ps -a

- python: "2.7"
env: FLAKE8=true
cache: pip
install:
- pip install flake8
before_script:
- flake8 --version
# stop the build if there are Python syntax errors or undefined names
- flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
- flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
script:
- true
<<: *flake8-steps

- python: "3.7"
env: FLAKE8=true
dist: xenial # required for Python 3.7
sudo: required # required for Python 3.7
cache: pip
install:
- pip install flake8
before_script:
- flake8 --version
# stop the build if there are Python syntax errors or undefined names
- flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
- flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
script:
- true
<<: *flake8-steps
88 changes: 87 additions & 1 deletion CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,58 @@ Changelog
v.2.9.0 TBA
==================

* When upgrading from previous CKAN versions, the Activity Stream needs a
migrate_package_activity.py running for displaying the history of dataset
changes. This can be performed while CKAN is running or stopped (whereas the
standard `paster db upgrade` migrations need CKAN to be stopped). Ideally it
is run before CKAN is upgraded, but it can be run afterwards. If running
previous versions or this version of CKAN, download and run
migrate_package_activity.py like this:

cd /usr/lib/ckan/default/src/ckan/
wget https://raw.githubusercontent.com/ckan/ckan/3484_revision_ui_removal2/ckan/migration/migrate_package_activity.py
wget https://raw.githubusercontent.com/ckan/ckan/3484_revision_ui_removal2/ckan/migration/revision_legacy_code.py
python migrate_package_activity.py -c /etc/ckan/production.ini

Future versions of CKAN are likely to need a slightly different procedure.
Full info about this migration is found here:
https://github.com/ckan/ckan/wiki/Migrate-package-activity

* A full history of dataset changes is now displayed in the Activity Stream to
admins, and optionally to the public. By default this is enabled for new
installs, but disabled for sites which upgrade (just in case the history is
sensitive). When upgrading, open data CKANs are encouraged to make this
history open to the public, by setting this in production.ini:
``ckan.auth.public_activity_stream_detail = true`` (#3972)

Minor changes:

* For navl schemas, the 'default' validator no longer applies the default when
the value is False, 0, [] or {} (#4448)
* If you've customized the schema for package_search, you'll need to add to it
the limiting of ``row``, as per default_package_search_schema now does (#4484)
* Several logic functions now have new upper limits to how many items can be
returned, notably ``group_list``, ``organization_list`` when
``all_fields=true``, ``datastore_search`` and ``datastore_search_sql``.
These are all configurable. (#4484, #4562)

Bugfixes:

* Action function "datastore_search" would calculate the total, even if you
set include_total=False (#4448)

Deprecations:
Removals and deprecations:

* Revision and History UI is removed: `/revision/*` & `/dataset/{id}/history`
in favour of `/dataset/changes/` visible in the Activity Stream. (#3972)
* Logic functions removed:
``dashboard_activity_list_html`` ``organization_activity_list_html``
``user_activity_list_html`` ``package_activity_list_html``
``group_activity_list_html`` ``organization_activity_list_html``
``recently_changed_packages_activity_list_html``
``dashboard_activity_list_html`` ``activity_detail_list`` (#4627/#3972)
* ``model.ActivityDetail`` is no longer used and will be removed in the next
CKAN release. (#3972)
* ``c.action`` and ``c.controller`` variables should be avoided.
``ckan.plugins.toolkit.get_endpoint`` can be used instead. This function
returns tuple of two items(depending on request handler):
Expand All @@ -38,6 +79,29 @@ Deprecations:
dataset blueprint in CKAN>=2.9, and, in the same time, it's still working for
Pylons package controller in CKAN<2.9

v.2.8.2 2018-12-12
==================

General notes:
* This version requires a requirements upgrade on source installations
* Note: This version does not requires a database upgrade
* Note: This version does not require a Solr schema upgrade

Fixes:

* Strip full URL on uploaded resources before saving to DB (`#4382 <https://github.com/ckan/ckan/issues/4382>`_)
* Fix user not being defined in check_access function (`#4574 <https://github.com/ckan/ckan/issues/4574>`_)
* Remove html5 shim from stats extension (`#4236 <https://github.com/ckan/ckan/issues/4236>`_)
* Fix for datastore_search distinct=true option (`#4236 <https://github.com/ckan/ckan/issues/4236>`_)
* Fix edit slug button (`#4379 <https://github.com/ckan/ckan/issues/4379>`_)
* Don't re-register plugin helpers on flask_app (`#4414 <https://github.com/ckan/ckan/issues/4414>`_)
* Fix for Resouce View Re-order (`#4416 <https://github.com/ckan/ckan/issues/4416>`_)
* autocomplete.js: fix handling of comma key codes (`#4421 <https://github.com/ckan/ckan/issues/4421>`_)
* Flask patch update (`#4426 <https://github.com/ckan/ckan/issues/4426>`_)
* Allow plugins to define multiple blueprints (`#4495 <https://github.com/ckan/ckan/issues/4495>`_)
* Fix i18n API encoding (`#4505 <https://github.com/ckan/ckan/issues/4505>`_)
* Allow to defined legacy route mappings as a dict in config (`#4521 <https://github.com/ckan/ckan/issues/4521>`_)
* group_patch does not reset packages (`#4557 <https://github.com/ckan/ckan/issues/4557>`_)

v.2.8.1 2018-07-25
==================
Expand Down Expand Up @@ -188,6 +252,21 @@ Changes and deprecations:
to specify this argument explicitly, as in future it'll become required.
* The ``ckan.recaptcha.version`` config option is now removed, since v2 is the only valid version now (#4061)

v2.7.5 2018-12-12
=================

* Strip full URL on uploaded resources before saving to DB (`#4382 <https://github.com/ckan/ckan/issues/4382>`_)
* Fix for datastore_search distinct=true option (`#4236 <https://github.com/ckan/ckan/issues/4236>`_)
* Fix edit slug button (`#4379 <https://github.com/ckan/ckan/issues/4379>`_)
* Don't re-register plugin helpers on flask_app (`#4414 <https://github.com/ckan/ckan/issues/4414>`_)
* Fix for Resouce View Re-order (`#4416 <https://github.com/ckan/ckan/issues/4416>`_)
* autocomplete.js: fix handling of comma key codes (`#4421 <https://github.com/ckan/ckan/issues/4421>`_)
* Flask patch update (`#4426 <https://github.com/ckan/ckan/issues/4426>`_)
* Allow plugins to define multiple blueprints (`#4495 <https://github.com/ckan/ckan/issues/4495>`_)
* Fix i18n API encoding (`#4505 <https://github.com/ckan/ckan/issues/4505>`_)
* Allow to defined legacy route mappings as a dict in config (`#4521 <https://github.com/ckan/ckan/issues/4521>`_)
* group_patch does not reset packages (`#4557 <https://github.com/ckan/ckan/issues/4557>`_)

v2.7.4 2018-05-09
=================

Expand Down Expand Up @@ -371,6 +450,13 @@ Deprecations:
jobs (http://docs.ckan.org/en/latest/maintaining/background-tasks.html). Extensions can still
of course use Celery but they will need to handle the management themselves.

v2.6.7 2018-12-12
=================

* Fix for Resouce View Re-order (`#4416 <https://github.com/ckan/ckan/issues/4416>`_)
* autocomplete.js: fix handling of comma key codes (`#4421 <https://github.com/ckan/ckan/issues/4421>`_)
* group_patch does not reset packages (`#4557 <https://github.com/ckan/ckan/issues/4557>`_)

v2.6.6 2018-05-09
=================

Expand Down
4 changes: 4 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ RUN apt-get -q -y update \
python-pip \
python-virtualenv \
python-wheel \
python3-dev \
python3-pip \
python3-virtualenv \
python3-wheel \
libpq-dev \
libxml2-dev \
libxslt-dev \
Expand Down
2 changes: 1 addition & 1 deletion LICENSE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ License
+++++++

CKAN - Data Catalogue Software
Copyright (c) 2006-2018 Open Knowledge International and contributors
Copyright (c) 2006-2018 Open Knowledge Foundation and contributors

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
Expand Down
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ ckan-dev mailing list or on Gitter.
Copying and License
-------------------

This material is copyright (c) 2006-2018 Open Knowledge International and contributors.
This material is copyright (c) 2006-2018 Open Knowledge Foundation and contributors.

It is open and licensed under the GNU Affero General Public License (AGPL) v3.0
whose full text may be found at:
Expand Down
61 changes: 61 additions & 0 deletions bin/remove-old-git-branches.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env bash

# Change this variable if you are using another name for ckan/ckan's remote
REMOTE=origin
PR_ENDPOINT=https://api.github.com/repos/ckan/ckan/pulls

prepare_branches() {
echo List existing branches
# awk filters out `master`, `release-something`, `dev-something` and `MAJOR.MINOR-something` branches
custom_branches=$(git fetch $REMOTE && git branch -r --list $REMOTE/* | awk '{ if ($2 !~ /^[0-9]\.[0-9]|^release|^dev-|^master$|HEAD/) print $2}' FS=/)

echo Calculate total amount of active PRs
# github adds Link header, that contains reference to last page with results. And that helps us to define total number of pages
last_page=$(curl -s -I $PR_ENDPOINT | grep '^Link' | sed -E 's/.*page=([[:digit:]]+).*rel="last".*/\1/')

echo Compute branches that are in use by active PRs
for ((i=1; i<=$last_page; i++))
do
# Take JSON, get name of branch with feature, leave only those ones, that starts with `ckan:`. I think, that jq is much more nice way to process json, but i'm not sure, that every one has it. But if you do,
# you can replace python's section with something like `| jq '.[] | .head.label' -r |`
next_portion=$(curl -s "$PR_ENDPOINT?page=$i" | python -c 'import json, sys; raw=sys.stdin.read(); labels=[str(item["head"]["label"]) for item in json.loads(raw)]; print("\n".join(labels))' | grep '^ckan' | sed 's/^ckan://')
used_branches="$used_branches\n$next_portion"
echo [$i of $last_page parts processed]
done;

echo Compute list of branches that can be safetely removed
tmp_file=$(mktemp)
diff <(echo -e "$used_branches" | sort) <(echo -e "$custom_branches" | sort) --unchanged-line-format="" >> $tmp_file

echo
echo List of all branches that can be removed stored into $tmp_file. Review it\'s content and run \'$0 --apply $tmp_file\'.
echo NOTE. This command requires write access to CKAN repository and will actually remove all branches that are listed in $tmp_file.
}

if [[ $# -gt 0 ]];
then
case "$1" in
-h|--help)
echo Compute/remove unused branches
echo Usage: $0 [--apply FILE_WITH_BRANCHES]
;;
--apply)
shift
if [[ -f $1 ]]
then
while read branch
do
git push $REMOTE --delete $branch
done < $1
else
echo File does not exist
fi
shift
;;
*)
echo Check usage with: $0 --help
;;
esac
else
prepare_branches
fi
16 changes: 12 additions & 4 deletions ckan/authz.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,13 +102,19 @@ def _build(self):
fetched_auth_functions[name] = auth_function

for name, func_list in chained_auth_functions.iteritems():
if name not in fetched_auth_functions:
if (name not in fetched_auth_functions and
name not in self._functions):
raise Exception('The auth %r is not found for chained auth' % (
name))
# create the chain of functions in the correct order
for func in reversed(func_list):
prev_func = fetched_auth_functions[name]
fetched_auth_functions[name] = functools.partial(func, prev_func)
if name in fetched_auth_functions:
prev_func = fetched_auth_functions[name]
else:
# fallback to chaining off the builtin auth function
prev_func = self._functions[name]
fetched_auth_functions[name] = (
functools.partial(func, prev_func))

# Use the updated ones in preference to the originals.
self._functions.update(fetched_auth_functions)
Expand Down Expand Up @@ -200,7 +206,8 @@ def is_authorized(action, context, data_dict=None):
return {
'success': False,
'msg': 'Action {0} requires an authenticated user'.format(
auth_function.__name__)
(auth_function if not isinstance(auth_function, functools.partial)
else auth_function.func).__name__)
}

return auth_function(context, data_dict)
Expand Down Expand Up @@ -401,6 +408,7 @@ def get_user_id_for_username(user_name, allow_none=False):
'create_user_via_api': False,
'create_user_via_web': True,
'roles_that_cascade_to_sub_groups': 'admin',
'public_activity_stream_detail': False,
}


Expand Down
52 changes: 52 additions & 0 deletions ckan/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# encoding: utf-8

import os

import click
import logging
from logging.config import fileConfig as loggingFileConfig

log = logging.getLogger(__name__)


def error_shout(exception):
click.secho(str(exception), fg=u'red', err=True)


click_config_option = click.option(
u'-c',
u'--config',
default=None,
metavar=u'CONFIG',
help=u'Config file to use (default: development.ini)'
)


def load_config(config=None):
from paste.deploy import appconfig
if config:
filename = os.path.abspath(config)
config_source = u'-c parameter'
elif os.environ.get(u'CKAN_INI'):
filename = os.environ.get(u'CKAN_INI')
config_source = u'$CKAN_INI'
else:
default_filename = u'development.ini'
filename = os.path.join(os.getcwd(), default_filename)
if not os.path.exists(filename):
# give really clear error message for this common situation
msg = u'ERROR: You need to specify the CKAN config (.ini) '\
u'file path.'\
u'\nUse the --config parameter or set environment ' \
u'variable CKAN_INI or have {}\nin the current directory.' \
.format(default_filename)
exit(msg)

if not os.path.exists(filename):
msg = u'Config file not found: %s' % filename
msg += u'\n(Given by: %s)' % config_source
exit(msg)

loggingFileConfig(filename)
log.info(u'Using configuration file {}'.format(filename))
return appconfig(u'config:' + filename)

0 comments on commit f3a78e2

Please sign in to comment.