ckanext-datagovtheme Py3 + CKAN 2.9 support

[thejuliekramer]

User Story

As a data.gov developer, I want ckanext-datagovtheme running with CKAN 2.9 and Python 3 so that we can move out of CKAN 2.8 and reduce our technical debt and meet compliance standards.

Acceptance Criteria

• WHEN I view CI
  THEN I see a successful job testing the extension under CKAN 2.9
  AND I see a successful job testing the extension under CKAN 2.8
  AND I see a successful job testing the extension under the legacy test environment
• WHEN I remove the tests
  THEN I see CI fail due to test coverage being below a threshold

Background

CKAN 2.9 extension tracker.

Details / tasks

The goal is to support running this extension against CKAN 2.9 (Python 3) and CKAN 2.8 (Python 2) environments.
Please refer to the main CKAN documentation for Python 3 extensions migration. We're not following this guide exactly. Instead of creating a long-lived py3 branch, we should be merging often. New CI test suites don't need to required or to be fully passing in order to merge. It's only important that changes don't introduce new failures or break the existing test suites.

Below are a list of tasks. Depending on the extension's functionality, not all tasks will be actionable.

Tasks:

• Update spreadsheet to indicate the extension upgrade is in progress
• Rename default branch -> main.
• Remove any CKAN 2.3 tests and functionality
  • Look for check_ckan_version
  • grep for '2.3'
  • See GSA/ckanext-datagovtheme@5968f42 for additional examples
• Dependencies must be python 3 compatible (run caniusepython3 -r requirements.txt from within the virtualenv) and post the results to this issue (docs)
• Add additional CI config for CKAN 2.8 and 2.9 Update CI config. These new CI jobs do not to be required yet (docs)
  • add code coverage reporting and required threshold (example)
• Update documentation with compatibility table (docs)
  • Features
  • "Usage" (how to use/install/configure the extension) vs "Development" (how to test etc)
  • Required extensions and dependencies
  • "Weak" dependencies or how this code interacts with other extensions
• Update/add docker-compose environment in order to run tests locally (example)
• Update test suite (docs)
  • consider copying tests dir to tests/nose in order to preserve existing test environment (example)
  • replace nose with pytest (example)
• Fix Python 3 issues / futurize code (docs)
  • Install future and run futurize --both-stages --write ckanext
• Import from core ckan using toolkit (git grep -w 'paste\|pylons' should return no imports) (docs)
• (optional) Create Mixin plugin implementations for Pylons / Flask (docs) (example)
• Convert controllers to flask views/blueprints (docs)
• Migrate Paster CLI commands to Click (docs)
• Migrate js and css assets to Web assets (docs) (example)
• Update templates (docs) (example)
  • you may want to split the template for CKAN >=2.9 in order to keep templates readable while maintaining backwards compatibility
  • update resource tags to asset for Web Assets
  • update controller='package' -> controller='dataset' or controller='resource'
• Update spreadsheet to indicate the extension upgrade is complete

Additional resources:

• CKAN 2.9 Changelog
• Python 3 migration for CKAN extensions
• Python's porting guide
• 2to3 automatic Python 2 to 3 conversion tool

[adborden]

caniusepython3:

bash-5.0# caniusepython3 -r requirements.txt
/usr/lib/python2.7/site-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography import utils, x509
[WARNING] Skipping u'-e git+https://github.com/ckan/ckan.git@1e10689a4e9936069cb1c0b2fc8bf630c530c246': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/ckan/ckanext-archiver.git@4cb10ac9f96fc78bc7be4b3dee8fbc56049b2865': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/ckanext-datagovcatalog.git@2225231e129e4aacb35939dcd2765ed7f9216bb1': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/ckanext-datagovtheme.git@1853971076dc65c34bbf9864336c8de15bf9e6c9': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/ckanext-datajson.git@4ca1ca31cfcc23d6610c13693049fcfc83696d08': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/ckan/ckanext-dcat@dfeae13248b93fe542486262254cda7e627da2f6': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/ckanext-geodatagov.git@020e54d330c96f17f9ca6cd5429f2aa335ad37d0': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/ckanext-googleanalyticsbasic.git@f0b75cf965e477a84f044885b27880782a48969c': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/ckan/ckanext-harvest.git@44bf78865ff49ff4313c22c6ece8c736e2e41f0f': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/ckan/ckanext-qa.git@d7d384cb18a85243ea62ef0bcc76bc1775f1c806': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/davidread/ckanext-report.git@b67875b2a5b4c9b9ab2cf255f074226255ca9398': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/keitaroinc/ckanext-saml2auth.git@d77d85349c7a6fcfa5a14c19a56b53eacf137a7c': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/ckan/ckanext-spatial.git@4ac25f19aa4eb9c798451f5eeb3084f907ccc003': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/pysaml2.git@c26a4893ff4c8d97e5f7994809ccb96a9d12caa3': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/GSA/webob.git@90d895f9a85c8bea11c80efa4e2f6bdd04eabbe4': could not parse requirement
[WARNING] Skipping u'-e git+https://github.com/asl2/PyZ3950.git@c2282c73182cef2beca0f65b1eb7699c9b24512e': could not parse requirement
Finding and checking dependencies ...
[ERROR] Failed to get external data for https://www.red-dove.com/pypi/projects/P/pairtree/project.json: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/distlib/util.py", line 913, in _get_external_data
    resp = urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
[WARNING] pairtree not found; false-negatives possible
[ERROR] Failed to get external data for https://www.red-dove.com/pypi/projects/P/pyutilib-component-core/project.json: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/distlib/util.py", line 913, in _get_external_data
    resp = urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
[WARNING] pyutilib-component-core not found; false-negatives possible
[ERROR] Failed to get external data for https://www.red-dove.com/pypi/projects/G/geoalchemy/project.json: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/distlib/util.py", line 913, in _get_external_data
    resp = urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
[WARNING] geoalchemy not found; false-negatives possible

You need 12 projects to transition to Python 3.
Of those 12 projects, 12 have no direct dependencies blocking their transition:

  ckanext-envvars
  ckantoolkit
  json-table-schema
  lepl
  messytables
  ofs
  passlib
  pylons
  repoze-who-friendlyform
  vdm
  weberror
  webhelpers

[adborden]

$ pipenv run futurize --both-stages . 2> /dev/null | diffstat
 controllers.py        |    8 +++++---
 helpers.py            |   39 ++++++++++++++++++++++-----------------
 tests/test_ui_data.py |   12 +++++++-----
 3 files changed, 34 insertions(+), 25 deletions(-)

[adborden]

Some hacks and work arounds we should come back to address:

• Add smoke test to catalog.data.gov to ensure dcat link exists in html source ([Task] Implement smoke test for dcat feed on dataset page data.gov#3255)

[mogul]

Waiting on James' return to finish the review and close out the issue.

[nickumia-reisys]

Not sure if this effects anything, but on the main branch, there is still one pip package that isn't python 3 supported?

$ caniusepython3 -r requirements.txt
[WARNING] Skipping '-e git+https://github.com/ckan/ckanext-harvest.git': could not parse requirement
Finding and checking dependencies ...

You need 1 project to transition to Python 3.
Of that 1 project, 1 has no direct dependencies blocking its transition:

  ckantoolkit