Migrate all packages to openssl3 and python 3.11 #5820
Conversation
|
@th0ma7 octoprint needs to be updated to v1.9.0 at least to work with python311 (I have already a local build with octoprint 1.9.0, but latest is v1.9.2). |
Thnx for the heads-up, in the meantime I'll re-map it to py310 but enforce it to use newer |
th0ma7
changed the title
|
@th0ma7 found that crypography < 40.0 has static binding to openssl and prebuilt wheels of version 38.0.3 are linked to openssl 1.1. currently trying to build cryptography==38.0.3 with cross/cryptography_38 and cross/openssl3 for homeassistant 2023.1.7.
EDIT: oops - OpenSSL 1.1.0 has been dropped, but not OpenSSL 1.1.1 |
|
@th0ma7 headphones has a known issue with python 3.11 |
@hgy59 Reading through the thread is appears that the |
|
@th0ma7 sorry, I missed that you already fixed the codec2 URL... |
|
@th0ma7 fixed protobuf wheel for duplicity. This is strange since my local build successfully builds all archs (except OLD_PPC_ARCHS) like:
for OLD_PPC_ARCHS even Python 3.11 fails to build with configure warning: |
|
@hgy59 I've hit a really interesting case where It hapends that it had worked for DSM7 using newer gcc and resuming DSM6 build would re-use the pip cache, thus finding the already resulting wheel. In this specific case, greenlet also needed |
@hgy59 thnx catching that. Note that with that train of thought I've added a section in the top summary for EDIT: Honestly I'd remove |
That was my finding too! BTW I have updated flexget to 3.7.10 too but was delayed by validation whether the transmission integration works, and found that the wheel installation does not work as expected (cross env wheels are not taken from the package but downloaded from index, and pip cache is ignored on updates with DSM 6 due to wrong user [install runs as root but all folders under var are changed to sc-flexget]) |
Just popped up that it would be very useful if we could reuse a prevously built python (incl crossenv) when building packages with wheels. Otherwise you need to build python for each package and arch... |
|
ffmpeg et. al. fail to build as nasm.us is currently down (https://updownradar.com/status/nasm.us) EDIT: and we are not the only one (microsoft/vcpkg#32741) |
That is something i'd really really much like doind, sinilar to ffmpeg.... That would save so many build cycles. I'll tentatively give that a shot and see if i can make that to work. It's the perfect pr for doing that... |
Qoriq was working yesterday on my build. It's hi3535 that was not.
Crossenv are used locally but mainly on non x86_64 archs and for wheels not providing pre-built download option. Otherwise you are right, it fetches them all. For dsm6, wonder if a chown hook somewhere couldn't fix that? |
|
@hgy59 I got the basics to prepare the build environment working using I'll have more cycles tonight to further look into it, but feel free to dig-in a little in the meantime if you have spare cycles. |
mk/spksrc.python.mk
Outdated
| SPK_DEPENDS := "python$(PYTHON_VERSION)>=$(shell sed -n 's/^SPK_VERS = \(.*\)/\1/p' $(shell pwd)/../python$(PYTHON_VERSION)/Makefile)-$(shell sed -n 's/^SPK_REV = \(.*\)/\1/p' $(shell pwd)/../python$(PYTHON_VERSION)/Makefile)" | ||
|
|
There was a problem hiding this comment.
IMHO SPK_DEPENDS must be defined by the respective package.
This would not work for packages that depend not only on python (like SPK_DEPENDS = "python311:git")
And probably it is not wanted to include the python package revision in the dependency.
There was a problem hiding this comment.
I have a different view on this, at least food for thoughts... I'm doing something in tvheadend that sort of solves this and would automate the populating of the SPK_DEPENDS variable value.
Something like the following in spksrc.python.mk would ensure python is always included in the SPK_DEPENDS:
PYTHON_DEPENDS = python$(PYTHON_VERSION)>=$(shell sed -n 's/^SPK_VERS = \(.*\)/\1/p' $(shell pwd)/../python$(PYTHON_VERSION)/Makefile)-$(shell sed -n 's/^SPK_REV = \(.*\)/\1/p' $(shell pwd)/../python$(PYTHON_VERSION)/Makefile)
ifneq ($(wildcard $(SPK_DEPENDS)),)
SPK_DEPENDS := "$(SPK_DEPENDS):$(PYTHON_DEPENDS)"
else
SPK_DEPENDS = "$(PYTHON_DEPENDS)"
endif
|
@th0ma7 I have a working version of just needs some verifications.... |
|
verification done. needed some investigation as the *.so files in the cross compiled wheels have binary differences when reusing prebuilt python. The difference is caused by different
IMHO this is not an issue that must be fixed since the package folder has precedence over the python311 folder. |
There is a possible source of runtime errors for cross compiled wheels: The only solution is to disable the pip wheel cache (we can keep the cache for source files in @th0ma7 any other suggestion? |
|
@hgy59 so following-up on #5841, at first glance this looks like specific to Thus, situation is not DSM6 specific, and I'd beleive not python specific as it is the same version as prior ( Although it has somehow to be
... Therefore, before releasing anything else from this PR I'd rather figure out where the issue lies. So currently investigating... Out of curiosity, having a EDIT: Found a first culprit! Missing is the python armv5 python3 configure log: armv5 build error relatively to python _ssl extension: Error really is: This looks totally specific to Which in turns comes down to EDIT2: Only option seems to be by patching python311 code to enforce building |
|
or just drop python311 for ARMv5 (as we did for OLD_PPC_ARCHS) |
|
@hgy59 honestly, I'm thinking along the same line and just go ahead with deleting the armv5 packages online and mark it as such in python311. I'd push things even a bit further, declaring |
|
We should then also mark all Python packages (like SABnzbd) as not supporting those arch's anymore. |
IMO a segfault indicates that the bigendian arch is not considered by the source of the affected library. The only models with qoric are DS213+, DS413. I like to support old models, as it helps to keep those usable and to reduce resource wasting... Or we reinvest into chroot packages (at least for models without docker support) - but this might not be a solution for (32-bit) PPC archs (and ARMv5 was only supported until debian 9 aka stretch with EOL 2022.06) |
|
Available options for
As for
As for supporting newer gcc, I've tried that and came to the conclusion that to build our own cross-compiler we have to restart from scratch and go with the stage1 and stage2 builds. It's a hell of a road to get to something that works. crosstool-ng may be a hell lot more easier to use? |
|
@th0ma7 If I remove the following line in then cross build of the wheel does not work. It takes headers in the build system instead of the toolchain: Any hint? |
|
Yup, i've hit that same issue a few weeks ago while updating python packages. I had to stick with the previous 9.x version as it didn't work. My investigations led me to exactly the same point. I would argue either trying out from cross and/or opening a ticket on their github project page... |
Thanks, I prefer to stay on homeassistant 2023.7.3 for the moment. |
- update homeassistant to v2023.7.3 - update to python311 with openssl3 - this PR now depends on SynoCommunity#5820 to be merged first
* homeassistant: update to v2023.5.4 - update homeassistant to v2023.5.4 - update HACS integration to v1.32.1 - update DTLSSocket to v0.1.15 - update and adjust requirements * fix build of greenlet wheel for older compilers * fix build of protobuf wheel for older compilers * fix build of webrtcvad for arch-qoriq-6.1 - thanks to @th0ma7 for wiseman/py-webrtcvad#78 - thanks to @smaarn for wiseman/py-webrtcvad@3bd7613 * homeassistant: update to v2023.7.3 - update homeassistant to v2023.7.3 - update to python311 with openssl3 - this PR now depends on #5820 to be merged first * fix some integrations - add cross/libstdc++ to fix grpcio for DSM < 7 - add cross/opus to fix voice over ip integration - fix Google Generative AI Conversation integration (requires grpcio fix and google_generativeai) * migrate to PYTHON_PACKAGE
@hgy59 I've looked at this and may have found a few interesting bits... some of it pertains to next release of Building python wheels is a mix-bag of magic, vodoo and luck... it seems that starting with newer pillow and numpy we just got luckier :) |
I tried agaein to build The error message tells, that this option is not valid when building with I think we should fully rely on PEP 517 and remove the |
|
There will be some wheels that will refuse to build. I recall when i added it initially there where many failures. Although situation may have changed since whereas only a few may still fail for which we could then add the option (e.g. disabling pep517) on a per wheel basis. |
OK, one issue I found with PEP517: Cross builds need investigation. The error above shows that it takes include files from the system, instead of the toolchain. |
|
@hgy59 was that from pillow build or another one entirely? |
|
@hgy59 if I recall you had looked into Is this something you have cycles for to complete before I publish |
yes, it is when building wheel Pillow==10.0.0 for homeassistant with arch-armv7-6.2.4 without the The code in Python.h (from spk/python311/work-armv7-6.2.4/install/...) looks like this // stdlib.h, stdio.h, errno.h and string.h headers are not used by Python
// headers, but kept for backward compatibility. They are excluded from the
// limited C API of Python 3.11.
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
# include <stdlib.h>
# include <stdio.h> // FILE*
# include <errno.h> // errno
# include <string.h> // memcpy()
#endif |
|
@th0ma7, are there any other packages to publish from this? Just wondering if we should change the label to "published" or not. |
|
@mreid-tt all packages have been migrated and published. The only remaining items relates to :
So long-story-short:
|
|
@th0ma7 thanks for the update. I've never used salt or dug very deeply into python packages but I have touched on a few with web gui components. If you have a branch or a PR that I can look at I can try to help. For now based on your comments, I'll leave the labels on this PR as is. |
I don't have anything started on salt packages. They are already migrated to python 3.11 and openssl3 as-is. If you do have spare cycles you'll find the web portion being a dependency of salt-gui makefile if i recall. What's missing is enabling it and confirming that it actually "works" somehow. |
Description
Fixes #5800, #5818
Checklist
all-supportedcompleted successfullyType of change
Package testing (x86_64)
cross/libtorrent-rakshasaneeded migratingtvheadend--> Tested and works ok with py311 + openssl 3.1.1 (no cross-compiled python wheel dependency) - Migration toopenssl3to be completed through subsequent PR including ffmpegPackage migrated to
spksrc.python.mkpython310--> Not Applicablepython311--> Not Applicabletvheadend--> Nothing to doBuild rule
all-supportedcompleted successfullytvheadend--> To be tested through another PRPUBLISHING DAY 1
PUBLISHING DAY 3
PUBLISHING any day from DAY 5
saltguienablement PR ...saltguienablement PR ...Note to self
How to get the list of supported language standards:
https://stackoverflow.com/questions/34836775/compiler-standards-support-c11-c14-c17