Merge remote-tracking branch 'origin/master' into 6118-table-designer

ckan · Mar 21, 2024 · 4a51415 · 4a51415
2 parents be174d9 + fae6879
commit 4a51415
Show file tree

Hide file tree

Showing 119 changed files with 24,194 additions and 18,861 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -8,7 +8,169 @@ Changelog
 ---------
 
 .. towncrier release notes start
-   
+
+v.2.10.4 2024-03-13
+===================
+
+Migration notes
+---------------
+
+- The default format for accepted uploads for user, groups and organization
+  images is now limited to PNG, GIF anf JPG. If you need to add additional
+  foramts you can use the :ref:`ckan.upload.user.mimetypes` and
+  :ref:`ckan.upload.group.mimetypes`) (`#7028
+  <https://github.com/ckan/ckan/pull/7028>`_)
+- Public user registration is disabled by default, ie users can not create
+  new accounts from the UI. With this default value, new users can be created
+  by being invited by an organization admin, being created directly by a
+  sysadmin in the ``/user/register`` endpoint  or being created in the CLI
+  using ``ckan user add``. To allow public registration see
+  :ref:`ckan.auth.create_user_via_web`, but it's strongly encouraged to put
+  some measures in place to avoid spam. (`#7028
+  <https://github.com/ckan/ckan/pull/7028>`_) (`#7208
+  <https://github.com/ckan/ckan/pull/7208>`_)
+
+Minor changes
+-------------
+- Define allowed alternative Solr query parsers via the :ref:`ckan.search.solr_allowed_query_parsers`
+  config option (`#8053 <https://github.com/ckan/ckan/pull/8053>`_)
+
+Bugfixes
+--------
+- `CVE-2024-27097 <https://github.com/ckan/ckan/security/advisories/GHSA-8g38-3m6v-232j>`_: fixed
+  potential log injection in reset user endpoint.
+- use custom group type from the activity object if it's not supplied, eg on
+  user activity streams (`#7980 <https://github.com/ckan/ckan/pull/7980>`_)
+- Removes extra <<<HEAD from resources list template (`#7998
+  <https://github.com/ckan/ckan/pull/7998>`_)
+- CKAN does not start without ``beaker.session.validate_key`` option introduced
+  in v2.10.3 (`#8023 <https://github.com/ckan/ckan/pull/8023>`_)
+- Editing of resources unavailable from package view page. (`#8025
+  <https://github.com/ckan/ckan/pull/8025>`_)
+- Pass custom package types through to the 'new resource' activity item (`#8034
+  <https://github.com/ckan/ckan/pull/8034>`_)
+- Fix Last Modified sort parameter for bulk-process page (`#8048
+  <https://github.com/ckan/ckan/pull/8048>`_)
+- Detect XLSX mimetypes correctly in uploader (`#8088
+  <https://github.com/ckan/ckan/pull/8088>`_)
+- Remove nginx cache as configuration from documentation (`#8031
+  <https://github.com/ckan/ckan/pull/8031>`_)
+- Fix `clean_db` fixtures breaking when tables are missing (`#8054
+  <https://github.com/ckan/ckan/pull/8054>`_)
+- Fix JS error in flash message when adding a Member (`#8104
+  <https://github.com/ckan/ckan/pull/8104>`_)
+
+
+v.2.10.3 2023-12-13
+===================
+
+
+Minor changes
+-------------
+- New sites now default to cookie-based sessions (the default value for ``beaker.session.type``
+  is now ``cookie``. The ``beaker.session.samesite`` configuration option has been introduced,
+  allowing you to specify the ``SameSite`` attribute for session cookies. This attribute determines
+  how cookies are sent in cross-origin requests, enhancing security and privacy.
+
+  .. note:: When using cookie-based sessions, it is now required to
+    set ``beaker.session.validate_key`` appropriately.
+
+- Skip interactive mode of ``ckan user setpass`` using ``-p``/``--password``
+  option. (`#7530 <https://github.com/ckan/ckan/pull/7530>`_)
+- Added support for Solr 9. Users of the `official Docker images
+  <https://github.com/ckan/ckan-solr>`_ can use the
+  ``ckan/ckan-solr:2.10-solr9`` tag. (`#7693
+  <https://github.com/ckan/ckan/pull/7693>`_)
+- Update requirements to support more Python versions (`#7935
+  <https://github.com/ckan/ckan/issues/7935>`_)
+- Add tooltips when links are truncated, to show the full text. (`#7743
+  <https://github.com/ckan/ckan/pull/7743>`_)
+- Added pages to confirm User delete and Dataset Collaborator delete.
+  Fixed cancellation of Group Member delete. (`#7813
+  <https://github.com/ckan/ckan/pull/7813>`_)
+- The ``validators`` attribute of a declared config option makes tries to parse
+  arguments to validators as python literals. If **all** arguments can be
+  parsed, they are passed to a validator factory with original types. If at least one
+  argument is not a valid Python literal, all values are passed as a string
+  (this was the previous behavior). Space characters are still not allowed inside
+  arguments, use the ``\\x20`` symbol if you need a space in a literal (`#7615
+  <https://github.com/ckan/ckan/pull/7615>`_)::
+
+      # Not changed
+      `validators: v(xxx)` # v("xxx")
+      `validators: v("xxx",yyy)` # v("xxx", "yyy")
+      `validators: v(1,2,none)` # v("1", "2", "none")
+      `validators: v("hello\\x20world")` # v("hello world")
+
+      # Changed
+      `validators: v("xxx")` # v("xxx")
+      `validators: v("xxx",1)` # v("xxx", 1)
+      `validators: v(1,2,None)` # v(1, 2, None)
+
+- Automatically add the ``not_empty`` validator to any config option declared
+  with ``required: true`` (`#7658 <https://github.com/ckan/ckan/pull/7658>`_)
+
+
+Bugfixes
+--------
+- `CVE-2023-50248 <https://github.com/ckan/ckan/security/advisories/GHSA-7fgc-89cx-w8j5>`_: fix potential
+  out of memory error when submitting the dataset form with a specially-crafted field.
+- Fix ``deprecated`` decorator (`#7939
+  <https://github.com/ckan/ckan/pull/7939>`_)
+- Fix for missing Tag facets on Home page (`#7520
+  <https://github.com/ckan/ckan/pull/7520>`_)
+- Fix errors when running the `ckan db upgrade` command (`#7681
+  <https://github.com/ckan/ckan/pull/7681>`_)
+- Fix datastore_search + downloading datastore resources as json with null
+  values (`#6713 <https://github.com/ckan/ckan/pull/6713>`_)
+- ``CONFIG_FROM_ENV_VARS`` takes precedence over config file and extensions but
+  those settings are not normalized. (`#7502
+  <https://github.com/ckan/ckan/pull/7502>`_)
+- Fixed server not recognizing SSL settings in configuration .ini file
+  (`#7758 <https://github.com/ckan/ckan/pull/7758>`_)
+- Fix error when indexing a full ISO date with timezone info (`#7775
+  <https://github.com/ckan/ckan/pull/7775>`_)
+- Aligned `member_create` with `group_member_save` to prevent possible member
+  duplication. (`#7804 <https://github.com/ckan/ckan/pull/7804>`_)
+- datastore-only resources now have a visible download button on the resource
+  page (`#7806 <https://github.com/ckan/ckan/pull/7806>`_)
+- update resource ``datastore_active`` with a single statement on
+  ``datastore_create/delete`` (`#7832 <https://github.com/ckan/ckan/pull/7832>`_)
+- Fixed Octet Streaming for Datastore Dump requests. (`#7839
+  <https://github.com/ckan/ckan/pull/7839>`_)
+- Fixed restricting anonymous users in actions to check user in context.
+  (`#7871 <https://github.com/ckan/ckan/pull/7871>`_)
+- Empty string in ``beaker.session.timeout`` produces an error instead of
+  never-expiring session (`#7881 <https://github.com/ckan/ckan/pull/7881>`_)
+- Updated Bootstrap alert-error class to alert-danger (`#7901
+  <https://github.com/ckan/ckan/pull/7901>`_)
+- Changed dataset query to check for ``+state:`` in the ``fq_list`` as well as the
+  `fq` parameter before forcing ``state:active`` (`#7905
+  <https://github.com/ckan/ckan/pull/7905>`_)
+- View modules use pluggable ``ckan.plugins.toolkit.h`` instead of
+  `ckan.lib.helpers` (`#7923 <https://github.com/ckan/ckan/pull/7923>`_)
+- Fix HTML5 validation failing on resource uploads (`#7925
+  <https://github.com/ckan/ckan/pull/7925>`_)
+- Fixed issues with the ``ckan views create`` CLI sub-command. (`#7944
+  <https://github.com/ckan/ckan/pull/7944>`_)
+- Improve handling of date fields in Solr (`#7775
+  <https://github.com/ckan/ckan/pull/7775>`_)
+- Fix URL validator does not support ":" for specifying ports (`#7891
+  <https://github.com/ckan/ckan/pull/7891>`_)
+- Fix user_show for ``ckan.auth.public_user_details`` (`#7866
+  <https://github.com/ckan/ckan/pull/7866>`_)
+- Add missing translations to aria-label attributes (`#7947
+  <https://github.com/ckan/ckan/pull/7947>`_)
+- Catch AttributeErrors in license retrieval (`#7931
+  <https://github.com/ckan/ckan/pull/7948>`_)
+- Fix downloading datastore resources as json with null values in json columns
+  (`#7545 <https://github.com/ckan/ckan/pull/7545>`_)
+
+v.2.10.2
+========
+
+Unreleased
+
 v.2.10.1 2023-05-24
 ===================
 
@@ -570,6 +732,45 @@ Removals and deprecations
 - ``ckan.route_after_login`` renamed to ``ckan.auth.route_after_login`` (`#7350
   <https://github.com/ckan/ckan/pull/7350>`_)
 
+v.2.9.11 2024-03-13
+===================
+
+Minor changes
+-------------
+- Define allowed alternative Solr query parsers via the :ref:`ckan.search.solr_allowed_query_parsers`
+  config option (`#8053 <https://github.com/ckan/ckan/pull/8053>`_). Note that the 2.9 version of this
+  patch does not use pyparsing to parse the local parameters string, so some limitations are in place,
+  mainly that no quotes are allowed in the local paramaters definition.
+- Get default formats for DataStore views from config (`#8095 <https://github.com/ckan/ckan/pull/8095>`_)
+
+Bugfixes
+--------
+- `CVE-2024-27097 <https://github.com/ckan/ckan/security/advisories/GHSA-8g38-3m6v-232j>`_: fixed
+  potential log injection in reset user endpoint.
+- Fixed Octet Streaming for Datastore Dump requests. (`#7899 <https://github.com/ckan/ckan/pull/7899>`_)
+- Fix Password Reset Keys with multiple accounts (`#8079 <https://github.com/ckan/ckan/pull/8079>`_)
+- Detect XLSX mimetypes correctly in uploader (`#8088 <https://github.com/ckan/ckan/pull/8088>`_)
+
+
+v.2.9.10 2023-12-13
+===================
+
+Bugfixes
+--------
+
+- `CVE-2023-50248 <https://github.com/ckan/ckan/security/advisories/GHSA-7fgc-89cx-w8j5>`_: fix potential
+  out of memory error when submitting the dataset form with a specially-crafted field.
+- Update resource datastore_active with a single statement (`#7833 <https://github.com/ckan/ckan/pull/7833>`_)
+- Fix downloading datastore resources as json with null values in json columns
+  (`#7545 <https://github.com/ckan/ckan/pull/7545>`_)
+- Fix errors when running the `ckan db upgrade` command (`#7681
+  <https://github.com/ckan/ckan/pull/7681>`_)
+- Fix ``deprecated`` decorator (`#7939
+  <https://github.com/ckan/ckan/pull/7939>`_)
+- Changed dataset query to check for ``+state:`` in the ``fq_list`` as well as the
+  `fq` parameter before forcing ``state:active`` (`#7905
+  <https://github.com/ckan/ckan/pull/7905>`_)
+
 v.2.9.9 2023-05-24
 ==================
 

diff --git a/changes/8053.misc b/changes/8053.misc
@@ -0,0 +1 @@
+Define allowed alternative Solr query parsers via the :ref:`ckan.search.solr_allowed_query_parsers` config option 
diff --git a/changes/8124.bugfix b/changes/8124.bugfix
@@ -0,0 +1 @@
+Populate email notification checkbox from the profile it's on, not from the logged-in user
diff --git a/ckan/common.py b/ckan/common.py
@@ -306,6 +306,15 @@ def aslist(obj: Any, sep: Optional[str] = None, strip: bool = True) -> Any:
         return [obj]
 
 
+def repr_untrusted(danger: Any):
+    """
+    repr-format danger and truncate e.g. for logging untrusted input
+    """
+    r = repr(danger)
+    rtrunc = r[:200]
+    return rtrunc + '…' if r != rtrunc else r
+
+
 local = Local()
 
 # This a proxy to the bounded config object

diff --git a/ckan/config/config_declaration.yaml b/ckan/config/config_declaration.yaml
@@ -892,6 +892,15 @@ groups:
           enabled to improve dataset update/create speed (however there may be
           a slight delay before dataset gets seen in results).
 
+      - key: ckan.search.solr_allowed_query_parsers
+        type: list
+        default: []
+        example: ["bool", "knn"]
+        description: |
+          Local parameters are not allowed when passing queries to Solr. An exception to this is when passing local parameters for special query parsers, that need to be enabled explicitly using this config option. For instance, the example provided would allow sending queries like the following::
+            search_params["q"] = "{!bool must=test}..."
+            search_params["q"] = "{!knn field=vector topK=10}..."
+
       - key: ckan.search.show_all_types
         default: dataset
         example: dataset

diff --git a/ckan/i18n/am/LC_MESSAGES/ckan.mo b/ckan/i18n/am/LC_MESSAGES/ckan.mo