treewide: Start deprecating Python 2

[diizzyy]

@BKPepe @commodo @jefferyto @cotequeiroz @neheb
It's time to starting thinking about on how to handle Python 2 and related packages as the clock is ticking. https://pythonclock.org/

Perhaps make it a hard requirement to use 3.X for new packages and only allow update PRs for 2.X ones in order to make a smooth transition?

Sorry, if I missed tagging someone ;-)

[BKPepe]

It was partially discussed in this issue: #8400 (comment) , @commodo is right, but if it would be up to me, I'm interested to have just new Python3 packages as I don't need to debug something on Python2, fewer things to maintain and care. :)

[commodo]

@BKPepe
feel free to drop Python2 variants from your packages;
since they are new, it won't hurt;

i've been wanting to drop Python2 from the beginning, but it was mostly me against users;
but now if there is a more grouped interest, i am fine to do it, step-by-step;

[cotequeiroz]

All of my packages are dependencies of the Seafile suite. They have been slow in announcing migration to python 3: https://www.seafile.com/en/roadmap/ TLDR: They're in version 6 now; version 8 is expected for 2020/04, and Python 3 migration comes after that as "Long Term".
Should we start thinking now about what to do then, or hope they move migration up their schedule?

It would be nice to have a "deprecation warning" to the end users upon installation of python2 packages, perhaps starting at the next release, so they are not caught by surprise. I don't know how to handle it yet, unfortunately.

[jefferyto]

I'd like to see a list of applications/packages (like Seafile) in the repo that depend on Python 2; as long as we support those programs then we should continue to support the packages they depend on. (Whether we continue to support those programs is another question.)

For instance, obfsproxy (the reason why I started contributing Python packages) is a Python 2 program. It isn't in development anymore. (obfs4proxy is the unofficial successor but is megabytes larger in size.) Can it be ported to Python 3? Will someone port it? Should we drop it? I think these are the questions we need to ask for each Python 2-dependent package in the feed.

As for end users installing Python 2 directly, definitely some kind of deprecation notice 😂

[cotequeiroz]

I meant it for when end-users install them as dependencies. users of obfsproxy or seafile may not care much about what dependent packages get installed along, and are the ones probably hit hardest when we drop support. It will at least make them aware that something is not up-to-date.

[jefferyto]

If we are going to drop support for all Python 2 packages, can we even say we still support the programs that depend on those packages?

I don't want to be pedantic, but can we define what "dropping support" means? Do we remove all Python 2 packages from the feed? Leave them in the repo but not update them anymore? (Can we even not update a Python 2 package if the Python 2 and 3 packages are built from the same source?)

(All questions and no answers, sorry I'm not very useful right now lol)

[diizzyy]

New packages will be Python 3 only and only updates for 2.X will be allowed
Phase #1: Switch existing packages to 3 only where possible
Phase #2: Deprecate packages that only supports 2.X and perferably offer counterparts if deemed necessary
I don't think it needs to be more complicated than that.

[jefferyto]

So for end users, are we going to have any notice period / deprecation warning / etc.? Do we have some kind of schedule for this? Are we going to tell people about this?

Just because there are no packages that use a particular Python 2 package, does not necessarily mean no one is using it. People can install Python 2 packages and write their own programs without telling anyone else.

Python 2 is still being maintained upstream for more than 8 months. I would assume our users will expect Python 2 to be available in the packages feed for just as long.

[diizzyy]

We could do something like UPDATING in FreeBSD ports but I highly doubt the majority would read it and it's not really available for end users and neither does opkg have any(?) kind of facility for any kind of notification.

https://www.freebsd.org/doc/handbook/ports-using.html - 4.5.3. Upgrading Ports
https://github.com/freebsd/freebsd-ports/blob/master/UPDATING

The best thing looking at whats available (I guess) would to update https://github.com/openwrt/packages/blob/master/lang/python/README.md about the plans and by doing so notify users about the "new" policy.

It's impossible and unreasonable to account for all potential scenarios so the most logical solution would to make use of what we already know and leave it at that (ie users in the package tree).

As for schedule, my suggestion would to put phase #1 in motion as soon as possible and annouce deadline in 7 months or so as it will most likely slip. As we all have limited time a slow and steady transition is more preferably rather than a treewide change the last minute.

[jefferyto]

It's impossible and unreasonable to account for all potential scenarios so the most logical solution would to make use of what we already know and leave it at that (ie users in the package tree).

By this logic we can remove all packages from the feed since we have no information on what packages anyone may or may not install.

I'm not asking for us to account for all potential scenarios; I'm asking us to be forgiving and make the transition as painless as possible for end users.

[diizzyy]

Instead of opposing, feel free to suggest a viable solution.

[BKPepe]

We can make an announcement about it on the forum.

[champtar]

Introduce a @deprecated like @broken, and stop building these packages on the buildbots ?

[jefferyto]

A rough proposal for discussion:

• New Py2 programs and libraries
  • In general no new packages allowed starting now
  • Exceptions can be made given strong reasons
  • Absolutely no new packages after a deadline in May or June
• Existing Py2 libraries
  • Continue to get normal updates until October or November
  • After the normal update deadline, apply security updates until March 2020
  • Mass removal in March 2020
• Existing Py2 programs
  • Upgrade/update to use Py3, for all that can be updated
  • For those that cannot be updated, same schedule as for Py2 libraries

Mass removal may need to happen earlier, depending on when the 20.x release will be.

Announcing it on the forum as @BKPepe suggested would be great. I think adding a page to the wiki would also help, since that's the main source of reference info for users.

For a @deprecated option, I'm not sure how much value there would be in keeping things in-tree but not build them on buildbots.

[diizzyy]

Looks good to me

[commodo]

A rough proposal for discussion:

* New Py2 programs and libraries
  
  * In general no new packages allowed starting now
  * Exceptions can be made given strong reasons
  * Absolutely no new packages after a deadline in May or June

* Existing Py2 libraries
  
  * Continue to get normal updates until October or November
  * After the normal update deadline, apply security updates until March 2020
  * Mass removal in March 2020

* Existing Py2 programs
  
  * Upgrade/update to use Py3, for all that can be updated
  * For those that cannot be updated, same schedule as for Py2 libraries

Mass removal may need to happen earlier, depending on when the 20.x release will be.

Announcing it on the forum as @BKPepe suggested would be great. I think adding a page to the wiki would also help, since that's the main source of reference info for users.

For a @deprecated option, I'm not sure how much value there would be in keeping things in-tree but not build them on buildbots.

I'm also fine with this proposal.
Thanks :)
Alex

[cotequeiroz]

LGTM

[neheb]

Fine with me.

[neheb]

FWIW: https://www.phoronix.com/scan.php?page=news_item&px=Fedora-31-Drop-Py2-Packages

[diizzyy]

Which will be released in late October this year
https://fedoraproject.org/wiki/Releases/31/Schedule

[diizzyy]

FWIW, https://svnweb.freebsd.org/ports?view=revision&revision=498529

[jefferyto]

(Apologies for my delay on getting back to this. I've been trying to organize my thoughts, to explain my reasoning better. I've also been trying to research how other distros are handling this, but there is a surprising lack of easily-searchable information out there.)

This is probably obvious to everyone, but I'll say it anyway: While other distros expect most (or all) of their "normal" users to use officially released, stable versions of their OS, OpenWrt has a small group of users that use development snapshots on their day-to-day hardware (probably not for mission critical stuff but still "production" use). This is because support for new routers is added to trunk/master between stable releases, so if a user buys such a router and wants to use OpenWrt, they will need to install a snapshot (until the next stable release is available). (There are also people who build their own firmware to try things like fast path or flow offload, but I'm not worried about them; they should know the risks.)

I don't know how large this group of users is, but the fact remains, we have "normal" users using packages from the master branch. This means we should be a little more careful about breaking things, or at least try to minimize unnecessary breakage. This is why my inclination is to support Python 2 for as long as practical (while also recognizing that we can't support it forever).

[jefferyto]

Anyway, assuming folks are still ok with my proposal above, I suggest these dates:

• Hard deadline for new Py2 programs and libraries: 31 May 2019 (see note below)
• Normal updates deadline: 31 October 2019 (security updates continue afterwards)
• Mass removal: 31 March 2020 (or 2 weeks before freeze for 20.x release, whichever is sooner)

Thoughts? Any other team members / contributors that should be looped into this discussion?

Edit:
Note about the hard deadline: We are already not accepting new Python 2 packages anymore, but if someone has really strong reasons, we can make exceptions up to this hard deadline.

[commodo]

Anyway, assuming folks are still ok with my proposal above, I suggest these dates:

* Hard deadline for new Py2 programs and libraries: 31 May 2019

* Normal updates deadline: 31 October 2019 (security updates continue afterwards)

* Mass removal: 31 March 2020 (or 2 weeks before freeze for 20.x release, whichever is sooner)

Thoughts? Any other team members / contributors that should be looped into this discussion?

Fine from my side.
We would need to add this in the Python readme.
Once it's there, people will accept this easier, like it's in the bible :)

[diizzyy]

Go for it

[cotequeiroz]

LGTM.

[jefferyto]

I've made a post to the forum about this discussion: https://forum.openwrt.org/t/python-2-deprecation-policy-under-discussion/35826

Let's wait a few days to see if there are any major objections.

[jeffsf]

Major Linux distros have already moved to Python 3. It's been over a decade since Python 3 was released and if third-party applications haven't transitioned, or are not at least Python 3 compatible, it shouldn't hold back OpenWrt from moving out of the past. Dropping Python 2 sooner than later means less distractions for package maintainers. While the start of 2020 seems too far out, I concede that the v19 release is perhaps too close in.

I agree that "nobody" is going to read READMEs or even package messages, if we had them. I don't know if there is a way to look at package download stats to see just how many people are using Python 2 and its packages. My gut feeling is that as, at least in my experience, getting to fit Python on a 16 MB device is challenging, it probably isn't a large fraction of the non-expert users of OpenWrt.

• Hard deadline for new Py2 programs and libraries: 31 May 2019

Perhaps cynical, but why bother with accepting anything new that is going to be removed in a year? Just say "Done with Python 2, please submit on Python 3" just like ar71xx vs. ath79. This would just seem to encourage a flood of hasty submissions.

[hnyman]

My two cents, rather late in to the discussion:

Hard deadline for new Py2 programs and libraries: 31 May 2019

I would set the new Py2 package deadline even closer: 1 May 2019.
There really isn't any valid reason to introduce new 2.7 stuff any more. (But 31 May 2019 works, too.)

Mass removal: 31 March 2020

I would maybe also hasten that "mass removal" regarding those packages that have python3 variants (or can be converted). Once the 19.0x has been branched, there is a "stable" version of the current master as 19.0x branch, and the remaining actual master can again be more forward-looking. So, I would start dropping python2 variants from master after the 19.0x branching, but only/mainly of packages that have python3 variant and are no hard dependency of another package that only supports python2.

Current python2 variants could stay alive in the stable 19.0x branch.

Ps.
One critical thing is to ensure that the OpenWrt toolchain switches to use 3.x. I briefly looked into into it two years ago, after Ubuntu had dropped the default python2 installation, and you needed to install it manually. But there was not much interest on the topic at that time as the 2.7 EOL was so far away.
https://lists.openwrt.org/pipermail/openwrt-devel/2017-October/009438.html

(Python isn't actually used by many of the actual build tools., but there may be quite a bunch of feed packages that use python in the build process.)

[jefferyto]

I added a small note to the dates comment: We are already not accepting new Python 2 packages anymore, but if someone has really strong reasons, we can make exceptions up to the hard deadline.

[jefferyto]

@jeffsf @hnyman I've tried to explain my thinking earlier in the thread, but I'll try to summarize:

• We have normal users using master, in between releases (a very small group perhaps, but still non-zero)
• Once a library package is on the feed, users can write custom Python programs that depend on that library (yes, we have no way of knowing if anyone is actually doing this)
• I would prefer to be as forgiving to users as possible, especially since we are only starting this deprecation process so close to the EOL deadline

There seems to be some activity to move buildroot to Python 3 (openwrt/openwrt#1937) but TBH I haven't been paying close attention.

[diizzyy]

master is a moving target, breakage is to be expected even if it should be avoided as much as possible.
Unless someone is willing to commit and maintain legacy software just drop it, there's no win in keeping potentially unsecure and outdated code around that no one wants to touch. It may sound aggressive but the "unwillingness" to move forward usually results in more cons than pros.

[jefferyto]

I've compiled some data in a Google spreadsheet, currently based on the repo at aee8628 (not taking into account any open PRs). (see note 1)

I plan on opening meta/tracking issues on GitHub to track progress based on this data. If anyone would like edit access, let me know.

As a summary, here are some package counts:

• Programs that use Python / provide Python libraries or plugins (see note 2):
  • Python 2 only: 19
  • Python 2 and 3: 7
  • Python 3 only: 8
• Python libraries (see notes 3, 4):
  • Python 2 only: 33
  • Python 2 and 3: 55
  • Python 3 only: 28

---

Notes:

1. "Programs" were found by grepping for "python"; I may have missed packages that use Python but don't have this keyword in their Makefile.
2. libs/libimobiledevice and net/uwsgi-cgi are not counted. They include python-package.mk but don't appear to use Python in any other way; they should be investigated.
3. Excludes these packages in lang/python: python, python3, micropython, micropython-lib, python-packages, python-pip-conf.
4. FWIW updating a Python 2 & 3 library package takes about the same amount of effort as updating a Python 3 only package.

[jefferyto]

@diizzyy We are already maintaining what is in master. I'm only saying we keep maintaining it for a little bit longer.

Let me rephrase: Whatever is happening / what we are doing in master, I'm asking we keep doing it for a little longer. (Yes there is unmaintained software, let's see if it can be updated/maintained.)

[jefferyto]

I've think we've waited long enough - I have submitted a PR (#8894), updating the Python readme with details on this plan (as you can see from the multiple references above 😅). Please come take a look.

There are a few minor changes from the proposal I outlined before (as noted in the commit message / PR text). Worth noting are:

• Added a bit on how the Python 2 interpreter will be handled. I don't think this is very controversial.

• Allowed for normal updates of Python 2 libraries that share the same Makefile as their Python 3 version.

  This is because the effort involved in applying separate updates for the two variants (security updates for Python 2, normal updates for Python 3) would be too great for little-to-no benefit.

[diizzyy]

I think this thread has served its purpose by now