Python 3.6.10 + fix cross-compilation issues #3833

smaarn · 2019-12-27T15:14:25Z

Motivation: I'm working on migrating bazarr (PR #3788 ) and it supports Python 3.6 ... But the one on spksrc doesn't work.
Linked issues: Optionally, add links to existing issues or other PR's

Checklist

Build rule all-supported completed successfully
Package upgrade completed successfully
New installation of package completed successfully

This is a raw revival of PR #3731

smaarn · 2019-12-27T15:32:10Z

It seems I need to actually use contents of #3739 (for the native files cleanups thing).
Updating it...

hgy59 · 2019-12-27T17:32:21Z

for mpfr fixes you can use #3811
@ymartin59 maybe its time to merge #3811?

smaarn · 2019-12-27T22:19:00Z

@hgy59 to be honest I could even remove everything which is not about python itself... But that's another topic and since it was part of the initial dev I thought it could be worth a shot.

hgy59 · 2019-12-27T22:56:41Z

What is the problem with python3 cross compile?
Is it the sysconfigdata.py file?
I worked on this more than a month ago and had to use cross compiled bcrypt library for homeassistant and made a fix for the name of sysconfigdata.py.

smaarn · 2019-12-28T11:21:37Z

What is the problem with python3 cross compile?
Is it the sysconfigdata_.py file? I worked on this more than a month ago and had to use cross compiled bcrypt library for homeassistant and made a fix for the name of sysconfigdata_.py.

OK that's weird.
I rebased at head of master and I'm still getting the following error: ModuleNotFoundError: No module named '_sysconfigdata_m_linux_i386-linux-gnu'

smaarn · 2019-12-28T11:22:00Z

Let me retry with a little bit more investigation.

hgy59 · 2019-12-28T16:21:16Z

@smaarn to fix this, I changed the file 002.xcompile.patch to look like this one
002-xcompile.patch

smaarn · 2019-12-30T05:33:00Z

After checking, fixup the patch file wasn't sufficient, there was also the copy instruction which was no longer valid so I updated the PR accordingly.

cross/python3/Makefile

smaarn · 2019-12-30T06:04:56Z

Generated pre-release: https://github.com/smaarn/spksrc/releases/tag/python-3.6.8-10

chickenandpork · 2019-12-30T06:26:27Z

In order to confirm the condition that was failing for this SPK, you need to test that a cross-build has the correct arch values (ie build arm on i386 an check the resulting arch on the python spk installed on the ARM NAS) then build a venv that cross-compiles dependencies. That cross-compile was breaking during the test case of home assistant building the modules it needs.

smaarn · 2020-01-01T17:41:09Z

In order to confirm the condition that was failing for this SPK, you need to test that a cross-build has the correct arch values (ie build arm on i386 an check the resulting arch on the python spk installed on the ARM NAS) then build a venv that cross-compiles dependencies. That cross-compile was breaking during the test case of home assistant building the modules it needs.

@chickenandpork Just to be sure I didn't misunderstand your comment: what you're saying is that the proper way to test it is to generate a package using cross compiled dependencies and a virtual env (using the "venv" module) ?

chickenandpork · 2020-01-01T18:28:40Z

@smaarn yes - specifically testing that the cross works, and that using the cross-built spk on it’s proper arch, that it can pull and build dependencies.

If I recall correctly, I got stuck on openzwave dependencies which were critical to my use of HA.

If you can nail this, it would be awesome.

HAPPY NEW YEAR :)

ymartin59 · 2020-01-02T07:14:48Z

@ymartin59 I confirm I also gave up openzwave support in HA because that module does not support cross-compilation options as far as I understood.
By the way, one step after the other, first Python 3.6, then HA update, then native modules for some HA plugins...

smaarn · 2020-01-02T18:40:12Z

Fair enough, I'll split this PR into two for the basic reason that I don't know how to use homeassistant (nor what would be relevant to test ;) ).
Besides: for the python3 sole part I have already a corresponding use case.

smaarn · 2020-01-02T22:28:19Z

All homeassistant logic was moved out of this PR. If anyone is interested in picking up from what was done so far (I can attach corresponding binaries for those interested) it's here: https://github.com/smaarn/spksrc/tree/build/homeassistant/fix-cross-compilation-issue

Safihre

Following this with interest, since SABnzbd is now finally getting a Python 3 package 📦.

Safihre · 2020-01-03T20:32:29Z

spk/python3/Makefile

@@ -1,7 +1,7 @@
 SPK_NAME = python3
 SPK_SHORT_VERS = 3.6
 SPK_VERS = $(SPK_SHORT_VERS).8
-SPK_REV = 9


Why the bump to 11? Since only 9 was published.

It's only to keep track of the various attempts and have changelogs in case some options turned out to work better than the others.

Safihre · 2020-01-03T20:33:35Z

spk/python3/Makefile

@@ -21,7 +21,7 @@ DESCRIPTION_SPN = Lenguaje de programación Python.
 RELOAD_UI = yes
 STARTABLE = no
 DISPLAY_NAME = Python3
-CHANGELOG = "Update to Python 3.6.8"
+CHANGELOG = "Fixed cross compile settings"


Since this will be the next published version (previous was retracted), I would suggest keeping both the bump and the cross compile fix in the changelog.

Safihre · 2020-01-03T20:37:44Z

I have an ARM Synology. Should I build all packages (still have the TravisCI pipeline) and test it on there? What should I test specifically?

ymartin59 · 2020-01-03T20:59:55Z

@Safihre Package revision has been increased because of successive beta deliveries to volunteers. Please test ARM first... no need to build for all arch if packages are already broken for a single one...

smaarn · 2020-01-04T08:22:55Z

I did another commit because of following "warnings" issued in the cross python3 build (using docker):

PYTHONPATH=/spksrc/spk/python3/work-x64-6.1/install/var/packages/python3/target/lib/python3.6   \
	/spksrc/spk/python3/work-x64-6.1/Python-3.6.8/hostpython -Wi /spksrc/spk/python3/work-x64-6.1/install/var/packages/python3/target/lib/python3.6/compileall.py \
	-d /spksrc/spk/python3/work-x64-6.1/install/var/packages/python3/target/lib/python3.6 -f \
	-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
	/spksrc/spk/python3/work-x64-6.1/install/var/packages/python3/target/lib/python3.6
Traceback (most recent call last):
  File "/spksrc/spk/python3/work-x64-6.1/install/var/packages/python3/target/lib/python3.6/compileall.py", line 17, in <module>
    import struct
  File "/spksrc/spk/python3/work-x64-6.1/install/var/packages/python3/target/lib/python3.6/struct.py", line 13, in <module>
    from _struct import *
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

To be honest, I didn't face any issue in my test cases, but then, I was probably not using the underlyingly generated resources...

ymartin59 · 2020-01-04T08:45:15Z

@smaarn That is a culprit when building and testing on Intel 64 bits... everything seems to work. Truth comes when testing cross-compiled packages like aarch64 with application depending on native cross-compiled wheels. I will have a look.

smaarn · 2020-01-04T11:30:47Z

I tested it anyway on bazarr on evansport which, as far as I can tell, does fit the requirement, right ? :)

smaarn · 2020-02-02T07:59:33Z

Just squashed all the commits into a single one.

smaarn · 2020-02-02T07:59:42Z

(And rebased, obviously)

tunisiano187 · 2020-02-05T19:23:57Z

Why aren't you approving this? That seems Good and many scripts working on sabnzbd doesn't work anymore... they need python 3.6...

chickenandpork · 2020-02-05T19:27:46Z

Why aren't you approving this? That seems Good and many scripts working on sabnzbd doesn't work anymore... they need python 3.6...

If you haven’t confirmed that the author’s builds work as required on your NAS, that would help...

tunisiano187 · 2020-02-05T19:30:44Z

I'm new with syno packages, how can i test, do you have some tutorials ? I've got a DS418play if it can help
And i'm happy to help

chickenandpork · 2020-02-05T22:26:30Z

@smaarn would need to publish privately a build matching your hardware. You’d need to Download it, then go to the Synology installer and install a custom Package from where you downloaded. Finally, ensure the SPK acts as you expect (or at least in line with the expectations if any in this PR discussion). Make a note of the commit hash if @smaarn notes which he/she built, and include it in your remarks. I know I’d be happy to see someone confirming from each architecture, but likely we are all aware how that’s difficult to get testing coverage.

This project does lean towards lint-free optimal PRs rather than merging and correcting post-merge, which can extend the time it takes from PR to accept and merge. It can be really frustrating.

smaarn · 2020-02-06T17:50:48Z

I'm new with syno packages, how can i test, do you have some tutorials ? I've got a DS418play if it can help
And i'm happy to help

Hello @tunisiano187

Basically I generated the spks which can be found over there: https://github.com/smaarn/spksrc/releases/tag/python-3.6.8-11

From what I've just read here your model seems to be an x64 model so you could use the following temporary binary: https://github.com/smaarn/spksrc/releases/download/python-3.6.8-11/python3_x64-6.1_3.6.8-11.spk

Just download it, install it manually and "Accept" the risk of installing an unsigned one.

For what it's worth I've been running this package on evansport fine for several weeks now, but since my usage is for a single project (bazarr namely) there may definitely be "holes" in the fix.

ymartin59 · 2020-02-22T14:37:20Z

Hello. I have planned to review, compile and update application packages depending on Python 3.
I propose to jump to 3.6.10 if there is any reason to take benefits of this minor update.
Any opinion about it?

smaarn · 2020-02-22T18:48:39Z

Hello. I have planned to review, compile and update application packages depending on Python 3.
I propose to jump to 3.6.10 if there is any reason to take benefits of this minor update.
Any opinion about it?

Hello @ymartin59
I don't know if you're expecting any input from me. Personally I'd tend to think that while we're at it we might as well proceed to upgrading to the latest patch version.
You want to handle it or should I proceed within this PR ?

smaarn · 2020-02-23T17:22:09Z

New binaries available: https://github.com/smaarn/spksrc/releases/tag/python-3.6.10

Testing them on evansport while we speak. So far so good.

ymartin59 · 2020-02-26T17:32:24Z

cross/python3/Makefile

+PATH := $(WORK_DIR)/../../../native/$(PKG_NAME)/work-native/install/usr/local/bin:$(PATH)
+export PATH
+
+LD_LIBRARY_PATH := $(WORK_DIR)/../../../native/$(PKG_NAME)/work-native/install/usr/local/lib:$(LD_LIBRARY_PATH)
+export LD_LIBRARY_PATH
+


May you please document in a comment why these environment variables are required, for which commands?

According to official cross compilation instructions natively built python binary directory should be prepended to the PATH environment variable. Set LD_LIBRARY_PATH variable to target the "just generated native python" Fix configure args and removed no longer required patch changes Co-authored-by: Smaarn <antoine.malliarakis@gmail.com>

Document reason for environment variables overriding

ymartin59 · 2020-02-26T19:52:50Z

Hello @smaarn
I have consolidated history for soon-to-come merge.
I have updated dependencies and requirements.
I am currently working on application package updates to deliver all of them in a single publish.
Regards. Yves

Safihre · 2020-02-29T18:32:21Z

Great! Now we can start on adding Python 3.8!

ekanters71 · 2020-06-17T16:51:16Z

Hi. I have a DS415+. Which package should I best install? I have seen references to avoton in the past but this Is not currently listed.

Please assist.

smaarn force-pushed the build/python3/fix-cross-compilation-issue branch from 8e055e5 to fab0ccd Compare December 27, 2019 15:50

smaarn mentioned this pull request Dec 27, 2019

bazarr: 0.9.0.8 #3788

Closed

3 tasks

smaarn force-pushed the build/python3/fix-cross-compilation-issue branch from fab0ccd to 7d24167 Compare December 27, 2019 22:24

smaarn force-pushed the build/python3/fix-cross-compilation-issue branch from 7d24167 to 1daa9cb Compare December 30, 2019 05:31

smaarn commented Dec 30, 2019

View reviewed changes

cross/python3/Makefile Outdated Show resolved Hide resolved

miklosandras mentioned this pull request Jan 2, 2020

HomeAssistant 0.87.1-3 won't start #3643

Closed

smaarn force-pushed the build/python3/fix-cross-compilation-issue branch from 1daa9cb to cf3da5e Compare January 2, 2020 18:56

smaarn changed the title ~~Homeassistant 0.87.1 and Python 3.6.8 - fix cross-compilation issues~~ Python 3.6.8 - fix cross-compilation issues Jan 2, 2020

ymartin59 mentioned this pull request Jan 3, 2020

Python 3.5 instead of Python 3.6 is found via DSM Package Center for DS918+ #3841

Closed

Safihre reviewed Jan 3, 2020

View reviewed changes

smaarn force-pushed the build/python3/fix-cross-compilation-issue branch from e24e3ae to ecb8279 Compare February 2, 2020 07:58

smaarn changed the title ~~Python 3.6.8 - fix cross-compilation issues~~ Python 3.6.10 - fix cross-compilation issues Feb 23, 2020

smaarn changed the title ~~Python 3.6.10 - fix cross-compilation issues~~ Python 3.6.10 + fix cross-compilation issues Feb 23, 2020

ymartin59 reviewed Feb 26, 2020

View reviewed changes

smaarn requested review from ymartin59 and th0ma7 February 26, 2020 19:05

roleoroleo and others added 3 commits February 26, 2020 20:13

python3: upgrade to 3.6.10

82677d5

Document reason for environment variables overriding

python3: update dependencies and requirements

9d1523e

ymartin59 added the status/needs-review label Feb 26, 2020

ymartin59 force-pushed the build/python3/fix-cross-compilation-issue branch from 25a4fd3 to 9d1523e Compare February 26, 2020 19:51

ymartin59 removed the status/needs-review label Feb 29, 2020

ymartin59 merged commit 2d23bd2 into SynoCommunity:master Feb 29, 2020

ymartin59 mentioned this pull request Feb 29, 2020

Upgrade Beets to v1.4.9 #3743 #3897

Closed

3 tasks

ymartin59 mentioned this pull request Apr 10, 2020

Fix Python 3 wheels cross-compilation #3944

Closed

smaarn deleted the build/python3/fix-cross-compilation-issue branch April 12, 2020 09:33

ymartin59 removed the status/to-publish label Oct 31, 2020

Python 3.6.10 + fix cross-compilation issues #3833

Python 3.6.10 + fix cross-compilation issues #3833

Conversation

smaarn commented Dec 27, 2019 • edited

Checklist

smaarn commented Dec 27, 2019

hgy59 commented Dec 27, 2019

smaarn commented Dec 27, 2019

hgy59 commented Dec 27, 2019

smaarn commented Dec 28, 2019

smaarn commented Dec 28, 2019

hgy59 commented Dec 28, 2019

smaarn commented Dec 30, 2019

smaarn commented Dec 30, 2019

chickenandpork commented Dec 30, 2019 • edited

smaarn commented Jan 1, 2020

chickenandpork commented Jan 1, 2020

ymartin59 commented Jan 2, 2020

smaarn commented Jan 2, 2020

smaarn commented Jan 2, 2020

Safihre left a comment

Choose a reason for hiding this comment

Safihre Jan 3, 2020

Choose a reason for hiding this comment

smaarn Jan 3, 2020

Choose a reason for hiding this comment

Safihre Jan 3, 2020

Choose a reason for hiding this comment

Safihre commented Jan 3, 2020

ymartin59 commented Jan 3, 2020

smaarn commented Jan 4, 2020

ymartin59 commented Jan 4, 2020

smaarn commented Jan 4, 2020

smaarn commented Feb 2, 2020

smaarn commented Feb 2, 2020

tunisiano187 commented Feb 5, 2020

chickenandpork commented Feb 5, 2020

tunisiano187 commented Feb 5, 2020

chickenandpork commented Feb 5, 2020

smaarn commented Feb 6, 2020

ymartin59 commented Feb 22, 2020

smaarn commented Feb 22, 2020

smaarn commented Feb 23, 2020 • edited

ymartin59 Feb 26, 2020

Choose a reason for hiding this comment

ymartin59 commented Feb 26, 2020

Safihre commented Feb 29, 2020

ekanters71 commented Jun 17, 2020

smaarn commented Dec 27, 2019 •

edited

chickenandpork commented Dec 30, 2019 •

edited

smaarn commented Feb 23, 2020 •

edited