[WIP][python3] Upgrade to 3.8.6 #4287

smaarn · 2020-12-05T20:49:16Z

Motivation: Upgrade python version to 3.8 (since it's now the oldest "stable" version)
Linked issues: #4284

Checklist

Followups

Fix domoticz (see Build failure with python 3.8 domoticz/domoticz#3703)
Fix ffsync build
Fix mosquitto build
Fix homeassistant build
Fix znc build on hi3535
Fix rdiff-backup build on evansport
Fix homeassistant build on x64

hgy59 · 2020-12-16T19:36:09Z

@smaarn can you please rebase to head@master to include the update of cross/libffi to version 3.3?

hgy59 · 2020-12-16T21:53:27Z

@smaarn as a learned on the home assistant update with #4149 there is an issue with cffi 1.14.4, so older compilers fail to build it.
The original bug has been fixed already in cffi:
https://foss.heptapod.net/pypy/cffi/-/commit/4855f749ad397f79867d214b2b516664bec14019)
But this fix is not yet released, so we still use cffi 1.14.1.
We should use cffi 1.14.1 here too, until a fixed version is released (will fix one kind of homeassistant build failures).

affected files

native/python3/Makefile
cross/cffi/Makefile

smaarn · 2020-12-26T21:47:57Z

cross/PyNaCl/Makefile

@@ -1,12 +1,11 @@
 PKG_NAME = PyNaCl
-PKG_VERS = 1.3.0
+PKG_VERS = 1.4.0


PyNaCl version 1.4.0 introduces support for Python 3.8.

but homeassistant requires this in version 1.3.0
If we really need v1.4.0 for pythion 3.8 we must add cross/PyNaCl_1.3 and let homeassistant depend on it.

smaarn · 2020-12-26T21:48:27Z

cross/cffi/Makefile

@@ -13,14 +13,4 @@ LICENSE  = MIT

 PLIST_TRANSFORM = sed -e 's%@PYTHON_SITE_PACKAGES@%$(PYTHON_LIB_DIR)/site-packages%' -e 's%@PYTHON_VERSION@%$(SPK_SHORT_VERS)%'

-POST_INSTALL_TARGET = install_into_cross_env


This didn't seem to be really needed. So I removed it.

smaarn · 2020-12-26T21:49:14Z

cross/domoticz/patches/0001-DelayedLink.h-fix-build-with-python-3.8.patch

@@ -0,0 +1,67 @@
+From e9bd383ceb63db7cfe8a284014f0cdf8c2bfe4f0 Mon Sep 17 00:00:00 2001


When upgrading domoticz this shouldn't be necessary anymore.

smaarn · 2020-12-26T21:50:24Z

cross/domoticz/patches/0002-Fix-segfault-on-python-plugin-stop-3867.patch

@@ -0,0 +1,27 @@
+From b2752bc4a5e867e5fdfba7fab66ead49efe4de5a Mon Sep 17 00:00:00 2001


I found this in a forked repository of domoticz. It didn't seem to go through a patch or anything of that kind in the domoticz repository, not too sure about what to make of it.

smaarn · 2020-12-26T21:55:21Z

cross/gevent/Makefile

@@ -14,9 +14,11 @@ LICENSE  = MIT

 INSTALL_TARGET = gevent_install

+ENV += EMBED=false LIBEV_EMBED=false CARES_EMBED=false 


It turns out that it was always trying to build libenv and cares. Which was actually a terrible idea since

It's already "cross compiled"

Didn't work very well

smaarn · 2020-12-26T21:59:41Z

cross/python3/PLIST

@@ -6,13 +6,10 @@ lnk:bin/pydoc3
 rsc:bin/pydoc@PKG_SHORT_VERS@
 lnk:bin/python3
 bin:bin/python@PKG_SHORT_VERS@
-bin:bin/python@PKG_SHORT_VERS@m
-lnk:bin/pyvenv
-rsc:bin/pyvenv-@PKG_SHORT_VERS@


This was deprecated. It has been removed.

smaarn · 2020-12-26T22:00:58Z

mk/spksrc.wheel.mk

@@ -63,9 +63,9 @@ build_wheel_target: $(PRE_WHEEL_TARGET)
 		if [ ! -z "$(CROSS_COMPILE_WHEELS)" ] ; then \
 			$(MSG) "Force cross-compile" ; \
 			if [ -z "$(CROSSENV)" ]; then \
-				$(RUN) _PYTHON_HOST_PLATFORM="$(TC_TARGET)" CFLAGS="$(CFLAGS) -I$(STAGING_INSTALL_PREFIX)/$(PYTHON_INC_DIR) $(WHEELS_CFLAGS)" LDFLAGS="$(LDFLAGS) $(WHEELS_LDFLAGS)" $(PIP_WHEEL) ; \


It turnsout that using -I wasn't appreciated in some cases (more specifically gevent) where it was failing, indicating that it should go through CPPFLAGS instead.

smaarn · 2020-12-26T22:02:20Z

spk/ffsync/Makefile

@@ -40,6 +40,7 @@ include ../../mk/spksrc.spk.mk
 ffsync_extra_install:
 	install -m 755 -d $(STAGING_DIR)/var
 	install -m 644 src/ffsync.ini $(STAGING_DIR)/var/
+	install -m 644 src/requirements.txt $(STAGING_DIR)/share/wheelhouse/requirements.txt


Otherwise the sed operation below was failing. To be honest it's either we remove that line and the below line (the sed one). Or we keep that line or at least I need to know where that requirements.txt file would be supposed to be coming from.

smaarn · 2020-12-26T22:02:48Z

spk/ffsync/src/requirements.txt

@@ -92,11 +92,11 @@ repoze.lru==0.6
 #requests==2.4.3
 simplejson==3.6.5
 six==1.8
-testfixtures==4.1.0


This was failing building. But since it's related to testing I figured we could dismiss it, right ?

smaarn · 2020-12-26T22:02:57Z

spk/ffsync/src/requirements.txt

 tokenlib==0.3.1
 translationstring==1.1
 umemcache==1.6.3
-unittest2==0.6


This was failing building. But since it's related to testing I figured we could dismiss it, right ?

smaarn · 2020-12-27T12:47:49Z

Corresponding binaries have been published here: https://github.com/smaarn/spksrc/releases/tag/python-3.8.6-RC1

hgy59 · 2020-12-31T10:51:32Z

@smaarn please wait with the update of homeassistant package as I am currently creating v0.118.5-11 with fixes for some integrations (and with python 3.7.7), see #4345.

smaarn · 2020-12-31T10:55:27Z

@smaarn please wait with the update of homeassistant package as I am currently creating v0.118.5-11 with fixes for some integrations (and with python 3.7.7), see #4345.

Don't worry I still need to try all the updated packages before pretending that this is ready to be merged.

Fixed domoticz to support Python 3.8

…ries

…ration

…hanism

…ies)

removed unrequired test resources and fixed requirements packaging

smaarn · 2021-01-06T21:36:34Z

Updated binaries can now be found here: https://github.com/smaarn/spksrc/releases/tag/python-3.8.6-RC2

Squashed icu commits and reorganized commits.

hgy59 · 2021-01-07T06:18:12Z

Squashed icu commits and reorganized commits.

@smaarn please avoid unnecessairy force-pushes as it always breaks local clones from fast-forwarding.

hgy59 · 2021-01-07T06:31:55Z

@ymartin59 @smaarn and all py3 contirbutors here
As update of python 3.x to 3.(x+1) breaks all python3 packages installed with 3.x, we should consider to build different packages for 3.7 and 3.8.
If we start this versioning here we will leave python3 untouched and introduce python38 with it's own revision numbering (i.e. this would lead to python38-*-3.8.6-1.spk.

Benefits:

do not break packages that do not support python 3.8 yet
avoid the need of updating all py3 packages now
do not break installed python3 packages at installation of python 3.8
allow to create and publish python39 without retiring python38

I am not a python guru but all seems to depend on the virtual environments that are linked to installed Python-Major.Minor.

smaarn · 2021-01-07T06:35:04Z

@ymartin59 @smaarn and all py3 contirbutors here

As update of python 3.x to 3.(x+1) breaks all python3 packages installed with 3.x, we should consider to build different packages for 3.7 and 3.8.

If we start this versioning here we will leave python3 untouched and introduce python38 with it's own revision numbering (i.e. this would lead to python38-*-3.8.6-1.spk.

Benefits:

do not break packages that do not support python 3.8 yet

avoid the need of updating all py3 packages now

do not break installed python3 packages at installation of python 3.8

allow to create and publish python39 without retiring python38

I am not a python guru but all seems to depend on the virtual environments that are linked to installed Python-Major.Minor.

I would tend to agree with you here. Unfortunately I'm not an expert either here.
Another point to take into consideration is that other cross packages could need the same kind of approach (packages which no longer support python2 for example)

smaarn · 2021-01-07T11:40:34Z

@hgy59 with your previous point in my mind I would propose to:

Create separate PRs for fixes included in this PR (znc, homeassistant and so on)
Create in another PR a new python38 package and let package owners perform their migration when they see fit (I already have a pending package which I could use as guinea pig)

What do you think?

iandarbey · 2021-01-07T21:18:30Z

Any ideas when this will be merged? Some home assistant integrations rely on 3.8 or higher python (ikea in particular).

smaarn · 2021-01-29T18:39:46Z

Superseeded by #4413

smaarn force-pushed the feat/python3/upgrade-3.8 branch 2 times, most recently from 45f2212 to 2c45e2c Compare December 10, 2020 21:04

smaarn force-pushed the feat/python3/upgrade-3.8 branch 2 times, most recently from ebfff1c to 77ab8fb Compare December 13, 2020 23:42

smaarn force-pushed the feat/python3/upgrade-3.8 branch from 03e95a6 to 9226112 Compare December 16, 2020 19:44

smaarn force-pushed the feat/python3/upgrade-3.8 branch 3 times, most recently from 05ac415 to b617a70 Compare December 19, 2020 14:05

smaarn changed the title ~~[python3] Upgrade to 3.8.6~~ [python3][WIP] Upgrade to 3.8.6 Dec 19, 2020

smaarn changed the title ~~[python3][WIP] Upgrade to 3.8.6~~ [WIP][python3] Upgrade to 3.8.6 Dec 19, 2020

smaarn force-pushed the feat/python3/upgrade-3.8 branch 3 times, most recently from e98e83c to 8e1fbe8 Compare December 21, 2020 22:35

hgy59 mentioned this pull request Dec 23, 2020

Python 3.5 instead of Python 3.6 is found via DSM Package Center for DS918+ #3841

Closed

smaarn force-pushed the feat/python3/upgrade-3.8 branch from 73fe594 to 3664672 Compare December 26, 2020 21:31

smaarn commented Dec 26, 2020

View reviewed changes

smaarn marked this pull request as ready for review December 31, 2020 10:04

smaarn force-pushed the feat/python3/upgrade-3.8 branch from bbb775f to a3f0db8 Compare December 31, 2020 10:39

smaarn force-pushed the feat/python3/upgrade-3.8 branch 2 times, most recently from c931093 to a90e04d Compare January 3, 2021 20:16

smaarn added 5 commits January 6, 2021 22:33

cross/libicu: fixed build on hi3535 architecture

375f4d6

cross/numpy: fixed auto detection of avx-512f intrinsics

459d1e7

rsync-diff: fixed evansport build

64f74e3

[python3] Upgrade to 3.8.6

caba945

Fixed domoticz to support Python 3.8

wheel builds: prefer using CPP_FLAGS for specifying inclusion directo…

ff4df2b

…ries

smaarn added 15 commits January 6, 2021 22:33

cross/cffi: removed any crossenv custom logic

57211a7

cross/poetry: fixed cross/cffi dependency and reworked crossenv integ…

375449f

…ration

cross/PyNaCl: upgrade 1.3.0 -> 1.4.0 and fixed dependency on cross/cffi

4cd9373

cross/python3: ensure cffi is installed through standard crossenv mec…

ebe5b88

…hanism

cross/gevent: disable build of libev and cares (it's in the dependenc…

0b0c5dd

…ies)

spk/ffsync: python 3.8 support

6cfaad2

removed unrequired test resources and fixed requirements packaging

cross/greenlet: updated 0.4.5 -> 0.4.17

d719177

beets: added python 3.8 support

71da771

borgbackup: added python 3.8 support

60f6b09

flexget: added Python 3.8 support

26333cd

homeassistant: added python 3.8 support

b14e91e

rdiff-backup: added Python 3.8 support

e832c6d

sabnzbd: added Python 3.8 support

ecdc7d2

salt-minion: upgraded Python 3.8

9d0d27c

znc: added support for python 3.8

9c1abcd

smaarn force-pushed the feat/python3/upgrade-3.8 branch from 1485206 to 9c1abcd Compare January 6, 2021 21:34

smaarn closed this Jan 29, 2021

smaarn deleted the feat/python3/upgrade-3.8 branch January 29, 2021 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][python3] Upgrade to 3.8.6 #4287

[WIP][python3] Upgrade to 3.8.6 #4287

smaarn commented Dec 5, 2020 •

edited

hgy59 commented Dec 16, 2020

hgy59 commented Dec 16, 2020

smaarn Dec 26, 2020

hgy59 Jan 7, 2021

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn Dec 26, 2020

smaarn commented Dec 27, 2020

hgy59 commented Dec 31, 2020

smaarn commented Dec 31, 2020

smaarn commented Jan 6, 2021

hgy59 commented Jan 7, 2021

hgy59 commented Jan 7, 2021

smaarn commented Jan 7, 2021

smaarn commented Jan 7, 2021

iandarbey commented Jan 7, 2021

smaarn commented Jan 29, 2021

		@@ -13,14 +13,4 @@ LICENSE = MIT

		PLIST_TRANSFORM = sed -e 's%@PYTHON_SITE_PACKAGES@%$(PYTHON_LIB_DIR)/site-packages%' -e 's%@PYTHON_VERSION@%$(SPK_SHORT_VERS)%'

		POST_INSTALL_TARGET = install_into_cross_env

		@@ -0,0 +1,67 @@
		From e9bd383ceb63db7cfe8a284014f0cdf8c2bfe4f0 Mon Sep 17 00:00:00 2001

		@@ -0,0 +1,27 @@
		From b2752bc4a5e867e5fdfba7fab66ead49efe4de5a Mon Sep 17 00:00:00 2001

		@@ -14,9 +14,11 @@ LICENSE = MIT

		INSTALL_TARGET = gevent_install

		ENV += EMBED=false LIBEV_EMBED=false CARES_EMBED=false

[WIP][python3] Upgrade to 3.8.6 #4287

[WIP][python3] Upgrade to 3.8.6 #4287

Conversation

smaarn commented Dec 5, 2020 • edited

Checklist

Followups

hgy59 commented Dec 16, 2020

hgy59 commented Dec 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smaarn commented Dec 27, 2020

hgy59 commented Dec 31, 2020

smaarn commented Dec 31, 2020

smaarn commented Jan 6, 2021

hgy59 commented Jan 7, 2021

hgy59 commented Jan 7, 2021

smaarn commented Jan 7, 2021

smaarn commented Jan 7, 2021

iandarbey commented Jan 7, 2021

smaarn commented Jan 29, 2021

smaarn commented Dec 5, 2020 •

edited