Move to Python 3

[PeterJCLaw]

Python 2 is beyond end of life. Any Python tools we continue to use should be updated to Python 3 in order to ensure that we can continue operating.

This task is a parent task to oversee the general effort.

When upgrading we should ensure wide support for the components remains, which likely means targeting the a version of Python 3 which is widely installed by default. Choosing the version in Debian stable (currently "stretch" "buster") seems a good choice here, so let's stick to supporting Python 3.5 3.7 and above.

Components currently in use where support is either known to be lacking or unknown:

• the ticket/media consent system, though note that that needs other substantial improvements anyway (Upgrade media consent/tickets system #60)
• the reception software (which uses the tickets/media consent endpoints as a backend), see also Upgrade media consent/tickets system #60
• puppet configuration of hosted stuff (mostly blocked on the underlying things changing)

The following components support Python 3:

• SRComp
• sr.tools
• the zip packager used by the IDE
• various scripts installed directly by our puppet config
• the kit-maintenance scripts (turns out Update to Python 3 kit-maintenance-docs#5 did this a while back)
• some utils within the IDE (https://github.com/PeterJCLaw/srobo-ide/issues/4)
• the simulator (sr-turtle; see Support Python 3 sr-turtle#3)
• the server backup scripts
• the team-git repo HTTPD config generating script
• the competitor facing user management system ("nemesis" & dependencies, Upgrade to Python 3 nemesis#5)
• the internal user admin tool ("userman" & dependencies)
• the software running on the kits, currently mainly sr.robot & dependencies; however this is already tracked by Upgrade the SR kit software stack to Python 3 #19, so any discussion of the kit stack should happen there

Deprecated:

• trac -- this is not actively used and is no longer hosted anywhere. We have the archives and trac itself remains an open source project, however we don't expect to be using it any more. Long term it has been replaced by the runbook and various other repo-specific documentation. There's probably stuff we want to extract from the archives, but that's not related to its Python 3 status.
• libkoki has been replaced by zoloto. Therefore while its related tooling hasn't been updated, it is no longer relevant.

Please add other current components or projects to the above lists if they are missing.

[RealOrangeOne]

Need is a strong word. Do we actually need to?

The infrastructure side of this is a much easier argument, because those are internet accessible and security is more of a concern.

Do we need to migrate the kit software too? I get that it's a legacy technology which it would be good to upgrade, and that competitors all learn Python 3, but it's been that case for several years (I was being taught Python 3.2 back in 2014). I'm definitely not against it, I just don't want us to be trying to do too much in a short amount of time. Shipping something legacy is infinitely better than shipping something unstable!

3.5 feels like a reasonable version, although Debian is often behind the curve when it comes to versions, and chances are "buster" (the next version) will ship before the start of SR2020, which has 3.7 by default.

[PeterJCLaw]

I'd actually meant to put "2020" rather than "SR2020" in the description, though I actually think that targeting this being done before the start of the competition year is the right thing to do. Some of the things mentioned can definitely wait a bit longer though.

In terms of Python versions, while Buster does come out soon, Stretch will still be supported for some time after that and many currently supported distros are still using even older versions of Python 3. Debating which version to support could easily become a lengthy discussion without much real benefit to it. Supporting 3.5 doesn't really cost us anything, but should mean that things are likely to work in the vast majority of places.

I'm trying not to worry too much about the kit here -- that's already well covered by #19. My main purpose in creating this ticket was to enumerate all the other things we need to look at, so that we don't get to the point of setting up the competitor services machine for next year and find we need to make changes at that point.

In terms of "need" -- we are fast approaching the point where distros are going to stop including Python 2 and well past the point where all current distros have Python 3 available. I think it's reasonable therefore to argue that this is something we need to do.

[trickeydan]

https://j5.org.uk/j5

[PeterJCLaw]

Sitrep on this.

I've made a good number of the related scripts/programs compatible with Python 3 and just plain converted others. For things which are deployed I'm mostly going for compatible first so that migration can happen smoothly.

The remaining things essentially fall into two buckets: kit related things and user-account related things.

• I'm continuing to mostly ignore kit-related things on this ticket as Upgrade the SR kit software stack to Python 3 #19 exists for that, though I have added some more details where I've discovered them (notably around libkoki things) while trying to upgrade other things.
• The other things I'm deferring for the moment, pending Review our approach to media-consent #286 while I've just chased up on. While that's not strictly a blocker, if we end up changing how media-consent works then that might just plain remove a bunch of the stack which could mean a lot of wasted effort.

I'm fairly happy with progress on this so far. Happily much of it has been running 2to3, though the sheer number of things we have means it's been quite slow.

[PeterJCLaw]

I've made good progress on nemesis which includes the srusers library also used by userman. I think there's a good chance that that stack will be converted before the new year. (See srobo/nemesis#6 and the related PRs specific to Python 3; though also note that both nemesis and libnemesis recently gained Circle CI testing as a stepping stone here 🎉).

I'm now ignoring #286 for the purposes of this conversion. As a result, I'll likely pick up #60 once userman is done, with the aim to have it done during this competition year.

[raccube]

Marking off libkoki generation tool as libkoki has been replaced by zoloto which uses standard markers.

[PeterJCLaw]

Since we haven't used the ticketing/reception software for a couple of competition cycles and likely aren't going to use it, I think we can ignore the need to upgrade them. Similarly the outstanding puppet deployment work was related exclusively to them and was not used at all for the most recent competition cycle.

I'm therefore calling this done.