Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to CKAN 2.10 #4209

Closed
1 task done
jbrown-xentity opened this issue Feb 9, 2023 · 7 comments
Closed
1 task done

Upgrade to CKAN 2.10 #4209

jbrown-xentity opened this issue Feb 9, 2023 · 7 comments

Comments

@jbrown-xentity
Copy link
Contributor

jbrown-xentity commented Feb 9, 2023

Turning this into an epic

User Story

In order to utilize latest CKAN patches, data.gov admins want CKAN updated to 2.10.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

Background

https://github.com/GSA/catalog.data.gov/blob/main/ckan/requirements.txt#L11
#4113
See email on CKAN mailing list:

Hi all,

If all goes according to plan, next Wednesday 15th February around 13:00 UTC (14:00 CET, 08:00 EST) we will be releasing the next stable version of CKAN, 2.10, as well as patch releases for the 2.9 line (2.9.8).

Users are strongly encouraged to always run on the latest patch release for their version, as these include important security and
stability fixes. The latest patch release is the only one supported by the CKAN team (patch releases don't contain backwards incompatible changes).

More information about CKAN releases can be found here:

http://docs.ckan.org/en/latest/maintaining/upgrading/index.html#ckan-releases

Best,

Adrià 

Security Considerations (required)

None

Sketch

This will be a major lift, and will require picking up the work again here: #4113

Ideally every extension should test against 2.10, and verify things work. If they don't work, a new issue may be created to investigate further.
Final step will be upgrading catalog, and may require docker changes (among others).

@nickumia-reisys
Copy link
Contributor

nickumia-reisys commented Apr 10, 2023

Tenative sketch:

  • Add `"2.10" to the ckan-test.yml template.
  • Test all of our extensions
    • CKAN 2.10 Support - ckanext-datajson
    • CKAN 2.10 Support - ckanext-datagovcatalog
    • CKAN 2.10 Support - ckanext-datagovtheme
    • CKAN 2.10 Support - ckanext-geodatagov
    • CKAN 2.10 Support - ckanext-googleanalyticsbasic
    • CKAN 2.10 Support - ckanext-metrics-dashboard
    • CKAN 2.10 Support - ckanext-usmetadata
    • CKAN 2.10 Support - ckanext-dcat_usmetadata
  • Test all upstream extensions
    • ckanext-envvars
    • ckanext-s3filestore (inventory)
    • ckanext-xloader
    • ckanext-saml2auth
    • ckanext-harvest
    • ckanext-dcat
    • ckanext-spatial
    • ckanext-qa (catalog)
    • ckanext-report
    • ckanext-archiver
  • Make an issue for each extension that takes more than 1-2 hours to get working with CKAN 2.10
  • Upgrade and get all tests passing
  • Test main applications
    • catalog
    • inventory
  • Perform manual user inspection on development

@nickumia-reisys
Copy link
Contributor

General notes on CKAN 2.10 Upgrade:

  • CKAN 2.10 supports Python 3.7 to 3.10
  • This version requires a requirements upgrade on source installations
  • This version requires a database upgrade
  • This version does not require a Solr schema upgrade if you are already using the 2.9 schema, but it is recommended to upgrade to the 2.10 Solr schema.
  • Make sure to check the Migration notes

@Jin-Sun-tts Jin-Sun-tts self-assigned this Apr 19, 2023
nickumia-reisys added a commit to GSA/catalog.data.gov that referenced this issue May 1, 2023
OWSLib was updates so no need for the ignore.  Werkzeug is still being worked on with GSA/data.gov#4209, so give us more time
@nickumia-reisys
Copy link
Contributor

nickumia-reisys commented May 5, 2023

Current status:

@nickumia-reisys
Copy link
Contributor

CKAN 2.10.1 released EIGHT HOURS AGO!!

  • Verified that is has the license_list fix

image

@nickumia-reisys
Copy link
Contributor

Current issue being investigated with the /harvest/new route:

ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 199, in after_dataset_show
ckan_1                  |     source = HarvestSource.get(data_dict["id"])
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/model/__init__.py", line 121, in get
ckan_1                  |     o = cls.filter(**kwds).first()
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/model/__init__.py", line 130, in filter
ckan_1                  |     return query.filter_by(**kwds)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1806, in filter_by
ckan_1                  |     clauses = [
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1807, in <listcomp>
ckan_1                  |     _entity_namespace_key(from_entity, key) == value
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 317, in operate
ckan_1                  |     return op(self.comparator, *other, **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/properties.py", line 423, in operate
ckan_1                  |     return op(self.__clause_element__(), *other, **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/annotation.py", line 221, in __eq__
ckan_1                  |     return self.__element.__class__.__eq__(self, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 869, in operate
ckan_1                  |     return op(self.comparator, *other, **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 77, in operate
ckan_1                  |     return o[0](self.expr, op, *(other + o[1:]), **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/default_comparator.py", line 101, in _boolean_compare
ckan_1                  |     obj = coercions.expect(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 188, in expect
ckan_1                  |     resolved = impl._literal_coercion(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 523, in _literal_coercion
ckan_1                  |     return expr._bind_param(operator, element, type_=bindparam_type)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 4708, in _bind_param
ckan_1                  |     return BindParameter(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 1512, in __init__
ckan_1                  |     self.key = _anonymous_label.safe_construct(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 5375, in safe_construct
ckan_1                  |     body = re.sub(r"[%\(\) \$]+", "_", body).strip("_")
ckan_1                  |   File "/usr/local/lib/python3.8/re.py", line 210, in sub
ckan_1                  |     return _compile(pattern, flags).sub(repl, string, count)
ckan_1                  |   File "/usr/local/lib/python3.8/re.py", line 291, in _compile
ckan_1                  |     if isinstance(flags, RegexFlag):
ckan_1                  | RecursionError: maximum recursion depth exceeded while calling a Python object
ckan_1                  | Fatal Python error: Cannot recover from stack overflow.
ckan_1                  | Python runtime state: initialized
ckan_1                  |
ckan_1                  | Current thread 0x00007f5be02ec740 (most recent call first):
ckan_1                  |   File "/usr/local/lib/python3.8/uuid.py", line 782 in uuid4
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/types.py", line 17 in make_uuid
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 2776 in <lambda>
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1735 in _exec_default
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1871 in get_insert_default
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py", line 3226 in get_insert_default
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1927 in _process_executesingle_defaults
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1025 in _init_compiled
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1800 in _execute_context
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572 in _execute_clauseelement
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 333 in _execute_on_connection
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705 in _execute_20
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1238 in _emit_insert_statements
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 245 in save_obj
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 630 in execute
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456 in execute
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3486 in _flush
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3386 in flush
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 28 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/lib/search/__init__.py", line 167 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 71 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 39 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
db_1                    | 2023-06-01 12:34:19.208 UTC [226] LOG:  unexpected EOF on client connection with an open transaction
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/lib/search/__init__.py", line 167 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 71 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 39 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/lib/search/__init__.py", line 167 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 71 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 65 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   ...
ckan_1                  | [2023-06-01 12:34:19 +0000] [1] [WARNING] Worker with pid 496 was terminated due to signal 6

@nickumia-reisys
Copy link
Contributor

Catalog and Inventory successfully running CKAN 2.10 now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

5 participants