Get our apps off of Python 3.6

[seanh]

Background

• Python 3.6 support ends on 23 Dec 2021
• We are likely to start seeing certain libraries move on and possibly stop supporting Python 3.6
• We are missing out on speed benefits and new features of later versions of Python
• We have undergone one large scale upgrade before (2.7 -> 3.6) but the process was not documented (and likely wouldn't apply directly to a 3.x -> 3.y upgrade)

Outcomes

Desired technical outcomes

• All of our libraries and applications:
  • Are on a version of Python above 3.6
  • Are on as high a version of Python as we think is safe
  • Support running tests on multiple versions of Python
• Remove dependencies which obstruct upgrades and can be:
  • Replaced with libraries with better support
  • Consolidated (using one of two libraries which perform the same function)
  • Replaced with our own code (where they bring little value)

Incidental technical outcomes

• We may work with some 3rd party libraries to increase their Python compatibility

Technical consequences

• We will be running a heterogeneous Python environment
• Some products and libraries will support different versions
• Some libraries may have to support multiple versions indefinitely to span the required apps
• Developers will have to manage more instances of Python
• Deploy times and Github costs will increase

Desired documentation outcomes

• Document our procedure for upgrading Python from 3.x -> 3.y
• List our dependencies:
  • Which products they are used in
  • What purposes they serve
  • Whether that are encouraged for use/ acceptable in place / actively being phased out etc.
• Provide repeatable analysis of our dependencies and which versions they support to guide future updates

Desired policy outcomes

• Create a policy for library inclusion
• For example: if the dependency is popular and actively and well-maintained, and solves a significant problem for us, we can add it.

Implementation overview

Phase 1 - Information gathering

The aim of this phase is to:

• Gather information to guide further effort
• Sort the applications and libraries in an order which delivers the most upgraded applications in the shortest period of time
• Prove out examples of each type of action:
  • An application with multi python support
  • A library with multi python support
  • Working with a maintainer to update library support
  • Removing an old library (if a candidate can be identified)
• Result in at least one application which no longer requires Python 3.6
• Gain a gut feeling of effort

Phase 2 - Raise the ceiling

The aim of this phase is to:

• Move every application to the highest version of Python we can achieve with a reasonable effort
• Move every library to support the widest range of Python versions we can
• Provide time to be sure the new running versions are working

Phase 3 - Raise the floor

The aim of this phase is to:

• Raise the floor of each library to the maximum we can to still support our applications
• Remove any unused dual version support in applications
• General tidying and followup as required

Phase 4 - Documentation and policy

The aim of this phase is to:

• Document the procedure so it can be repeated now it has been followed
• Achieve any other policy or documentation goals

It is likely and desirable that many of the documentation goals might be wholly or in part achieved in earlier phases, but this gives an opportunity to recap and ensure they are met.

Implementation details

Sequencing

We should attempt to order our efforts in such a way as to result in the largest number of applications no longer being on Python 3.6 in a given time period. This means:

• Tackling the easiest applications first
• Moving libraries only in the order they are required to support particular applications (unless it proves to be trivial)
• Prefer moving more apps to a lower Python version than few apps to a higher Python version
• Potentially running multiple rounds of upgrades as effort allows

Determining library compatibility

We can determine that a library supports Python 3.x in the following ways (in order of preference):

• The metadata classifiers explicitly advertise support
• An official python version specific wheel is provided
• The documentation states (or implies) compatibility
• Evidence of tests being executed against the target version in the source code, in the latest release that is publicly available
• A locally modified version of the library runs against the target python version
• We have tested the area of our code which the code is used, and it appears to work

Handling incompatible libraries

In the event that a particular library supports a version of Python higher than 3.6, but lower than 3.9, we may decide to only partially upgrade the product in question. This:

• Still gets us some of the benefits of higher versions of Python
• Gives us a longer window of support on Python
• Gives the maintainers of the library more time to improve compatibility (without us expending effort)

If we decide to upgrade or in the event that a library supports at most Python 3.6 we should:

• See if the library can me removed or replaced
• Attempt to work with the maintainer to improve support
• As a last resource see if our product appears to work regardless

If a library has support for Python 3.5 or lower we should:

• Attempt to remove the dependency as a first resort
• Consider whether present experience shows the library works beyond it's spec and move on

Working with maintainers of packages to update Python compatibility

Working with every package maintainer to update their Python compatibility is likely impractical as:

• There are a lot of them below 3.9 (50+)
• Some packages are unmaintained
• Some do not have easily accessible source code / methods for contributing
• We are not necessarily in a place to learn every one

Therefore we should concentrate our efforts on:

• Significant libraries which we use in multiple products, or are singularly holding back an upgrade
• Libraries with active communities which look easy to contribute to
• Libraries where it looks easy to provide a patch

Determining readiness for updates

For any given product (library or application) if we have determined that:

• All of the direct dependencies support our target python version (with any of the methods listed above)
• We are happy that the test provide adequate coverage
• All the tests still pass

Then the product can be deemed to be compatible with the target version and the classifiers should be updated to match.

Note this can be achieved a number of ways including:

• Lowering the target version
• Removing or changing the direct dependencies
• Increasing test coverage

Python version support in our applications and libraries

Applications should:

• Temporarily support the current python version (3.6) to allow us to back out if something goes wrong
• Support the target version for upgrade
• Eventually remove support for the old version when we trust the target version
• At all times retain the ability to run against multiple versions of Python even if there is only one active at the moment
• Optionally target an "aspirational" version of Python we cannot reach yet to:
  • Give us warning of any additions which break compatibility with that version
  • ... or give us notice when we become compatible with that version
• Testing against all versions should be:
  • Mandatory in CI
  • Available in dev (but not the default)
  • Required to pass in all but the "aspirational" target if the "asprirational" target version has never passed
  • Required to pass in all cases if the "aspirational" target has ever passed

Libraries should:

• Temporarily support as wide a range of Python versions as they can (3.6, 3.7, 3.8, 3.9 etc.)
• Eventually drop support for older versions as our products no longer require them
• This behavior should be back ported into the cookiecutter library template
• In all other ways follow the recommendations for applications above

Appendix

Product list for upgrading python

Applications:

• h - Upgrade to Python 3.8 h#6600
• LMS - Upgrade LMS to Python 3.8 lms#2590
• bouncer - Upgrade Python to the highest version bouncer#409
• Via 3 - Update Python to 3.8 via#444
• Via HTML - Upgrade to Python 3.8 viahtml#147
• Checkmate - Upgrade Python to 3.8 checkmate#304
• h-periodic - Upgrade python to the highest possible version and add multi version support h-periodic#72
• Client -Upgrade from Python 3.6 client#3375

Libraries:

• h-vialib - Add multi Python support h-vialib#26
• h-matchers - Upgrade h-matchers to support Python 3.7, 3.8 and 3.9 h-matchers#36
• checkmatelib (requires Python 2 support for now) - Add support for Python 3.7, 3.8 and 3.9 checkmatelib#10
• pyramid-sanity - Add support for Python 3.7, 3.8 and 3.9 pyramid-sanity#12
• h-pyramid-sentry - Update Python support to include 3.9 (if possible) h-pyramid-sentry#19
• h-api - Add support for Python 3.7, 3.8 and 3.9 h-api#6
• h-cookiecutter-pypackage - Export multi python support to allow it to be shared with other libraries h-matchers#37

3rd-party PRs: #1204

Product list for removing Python < 3.8 support:

• See: Raise the minimum version of Python in our libraries #1206