Cross-compiling of Python modules with and without distutils

[chewi]

This probably won't be merged all in one go like this but it's mainly to get some initial feedback. I'm pretty happy with it now but I'm open to suggestions.

[chewi]

Here's the list of packages I've had success with. The Waf-based ones will fail for you as I have extra hacks to deal with that.

app-backup/rdiff-backup
dev-python/apsw
dev-python/bcrypt
dev-python/cffi
dev-python/coverage
dev-python/cryptography
dev-python/html5-parser
dev-python/lxml
dev-python/markupsafe
dev-python/msgpack
dev-python/netifaces
dev-python/numpy
dev-python/psutil
dev-python/pyalsa
dev-python/pyblake2
dev-python/pycryptodome
dev-python/pycurl
dev-python/pynacl
dev-python/pyrsistent
dev-python/pyxattr
dev-python/regex
dev-python/simplejson
dev-python/sip
dev-vcs/bzr
dev-vcs/mercurial
games-util/nml
sys-apps/portage
sys-devel/distcc
sys-libs/talloc
sys-libs/tdb
sys-libs/tevent

After making some tweaks and doing a full rebuild of the above, I have found that dev-python/sip now fails. I think it's a bad interaction between @mgorny's new patch and mine. I shall investigate. The rest is still good though.

[mgorny]

I think this will break stuff. One of the points of PYTHON being the real path is that people actually hardcode the path given. If you change it, then I think we'll end up with having scripts referencing #!/var/tmp/portage/dev-python/pyblake2-1.2.3/temp/python2.7/bin/python2.7 installed.

[chewi]

It's true that I did not consider this but I believe our python-exec stuff prevents this from happening with shebangs. I grepped my entire test cross system and did not find a single instance of a path like that in any non-binary file. I suppose it could hypothetically appear in other places but I'll build the whole of dev-python to find them if that's what it takes. Unless you have a better idea? Having PYTHON point to the wrapper is not necessary for most packages, generally just the ones that don't use distutils, so we could tackle those individually but it would still be quite tedious.

[mgorny]

It's not dev-python/ I'm worried about, it's everything outside dev-python/. Especially cheapo build systems.

[chewi]

I'll widen the search then. 😜 If cheapo build systems are installing scripts that are not handled by python-exec then that's probably a mistake in itself.

[mgorny]

python-exec doesn't affect the shebangs in scripts themselves. They're preserved as-is and respected, in /usr/lib/python-exec/*.

[mgorny]

(this makes me think of adding a QA check to Portage to make sure thingies don't install stuff with generic shebangs there)

[mgorny]

Oh, and please test it on regular system as well ;-). I'm not really opposed to those changes (though I haven't reviewed them completely) but I'm afraid this is going to cause silent retroactive breakage of stuff.

[chewi]

Fixed dev-python/sip. Still looking into the other issue.

[chewi]

After realising just how many Python packages there are, I tried to be smarter and grep'd for ebuilds where this might break. I did find some interesting cases, including several that are already broken because they use python_newexe on scripts with shebangs like #!/usr/bin/env python. Some used sed to replace ${PYTHON} with something else and one replaced the existing shebang with ${PYTHON}.

While the odd ebuild may need fixing with (or without!) this change, I tried to mitigate most cases by dealing with it in python_fix_shebang. This involved reworking the function to work in a slightly different way that I believe is an improvement. See for yourself. The interesting commits are d7170b0 and c7d3a01.

[stefson]

Would you mind adding cython to the list?

[chewi]

cython isn't in the _PYTHON_ALL_IMPLS list so we don't support it at all right now?

[stefson]

I meant to ask for dev-python/cython :-)

[chewi]

That is very much outside the scope of this pull request.

[stefson]

k, thanks for clarification.

[mgorny]

is GENTOO_LIBDIR the only issue here?

[chewi]

For those particular files, yes, I have just diff'd them to check. Does that mean we can avoid the wrapper issue? Not quite. You still have to override the file pointed at by _PYTHON_SYSCONFIGDATA_NAME. This variable cannot be a full path so PYTHONPATH is still needed. It only applies to Python 3 as 2 is hardcoded to load _sysconfigdata.py. It also needs to be set via a wrapper because as soon as you set it within the Portage environment, Portage itself keels over. Maybe we could patch in support for a full path but I'm not sure how we'd prevent it from screwing with Portage.

I need to rebase this because your patch was reverted in the end.

[mgorny]

Ok, so if we got rid of GENTOO_LIBDIR altogether will it reduce the needed patch or not at all?

[chewi]

Only the following call to ln.

[chewi]

I think this is good to go now. It hasn't changed much since my last big push a couple of months ago but I've made PYTHON only point to the wrapper when SYSROOT is not / to avoid unnecessary complexity. I cross-built the above list of packages again and also built them natively on a regular system to check.

I did have a very long think about whether there was another way to deal with the _sysconfigdata.py and PYTHONPATH issue to avoid having to rely on the wrapper. There really isn't. Even if you patched Python or did something clever with bind mounts, you need these overrides to only apply when Python is invoked in the context of the code being built, as opposed to being invoked by some external dependency or some Portage helper. Calling the wrapper via PYTHON or the PATH is the only effective way to do that.

[chewi]

It occurs to me that this is slightly broken for prefix users although the existing version is broken too. PYTHON is primarily used for building, which means it should be defined using BROOT rather than EPREFIX but this value will sometimes find its way into shebangs, where the opposite is true. You also have to consider prefix bootstrapping, where IIRC Python should be executed from the target prefix rather than the build prefix. I've forgotten exactly how this works but I think it relies on the PATH. More on this soon.

[chewi]

Apologies @stefson, I only now realise that I was totally confused about cython, thinking it was some kind of alternative Python implementation. No idea why. I'm pleased to report that it already cross-compiles with these changes.

[stefson]

I applied this pullrequest as a patch against the portage tree, which is at b6f034485fba47fc5e626da9fce75e80f4d3e8d0 , and got a few checksum warnings from portage:

Calculating dependencies / * Digest verification failed:
 * /usr/portage/dev-libs/boost/boost-1.65.0.ebuild
 * Reason: Filesize does not match recorded size
 * Got: 12239
 * Expected: 12440
* Digest verification failed:
 * /usr/portage/sys-libs/tdb/tdb-1.3.16.ebuild
 * Reason: Filesize does not match recorded size
 * Got: 1381
 * Expected: 1332
 - * Digest verification failed:
 * /usr/portage/sys-libs/tevent/tevent-0.9.37.ebuild
 * Reason: Filesize does not match recorded size
 * Got: 1625
 * Expected: 1576
 \ * Digest verification failed:
 * /usr/portage/sys-libs/talloc/talloc-2.1.14.ebuild
 * Reason: Filesize does not match recorded size
 * Got: 2422
 * Expected: 2373

[chewi]

Ebuild manifests are not included in the git repository but get added to the rsync repository later so patching like that would lead to this problem. A quick workaround is to use FEATURES="-strict".

[gentoo-repo-qa-bot]

Pull request CI report

Report generated at: 2019-04-21 11:38 UTC
Newest commit scanned: c928b99
Status: ✅ good

No issues found

[gaboroszkar]

@chewi Do these changes also fix wrong includes for python module builds?
I have had problems with some dev-python/* packages include the wrong version of /usr/include/python<VERSION>/* files when cross-compiling, and I had to make changes to these manually. (It tried to use the build system includes, for packages like sys-apps/portage, dev-python/lxml, dev-python/pyblake2 and dev-python/pyxattr.)

I really appreciate, that this Pull Request would fix the problem, that cross-emerge installs some site-packages into /usr/lib64/* instead of /usr/lib/* when cross-compiling from a 64 bit system to a 32 bit one! I'm all for these changes, very useful!

[chewi]

I don't recall seeing the version problem specifically but it's a good idea to keep Python versions, among other things, in sync. I imagine these changes would fix that problem though.

I'm still going to try and get these changes in but there's something I need to do first.

[gaboroszkar]

I don't recall seeing the version problem specifically but it's a good idea to keep Python versions, among other things, in sync.

I was talking about the problem that includes were used from the build system, instead of the built system. There were no Python version problems. Sorry if I was misleading.

[deorder]

When I run /usr/armv7a-rpi2hs-linux-musleabihf/usr/bin/python2.7-config --includes it returns:
-I/usr/include/python2.7 -I/usr/include/python2.7
which is not correct

When I run /usr/armv7a-rpi2hs-linux-musleabihf/usr/bin/python3.7-config --includes it returns:
-I/usr/armv7a-rpi2hs-linux-musleabihf/usr/include/python3.7m -I/usr/armv7a-rpi2hs-linux-musleabihf/usr/include/python3.7m
as expected, apparently resolved using a shell script and using PWD

libxml2, vim and more ebuilds that are using python2.7 with USE="python" fail with:
error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?).

An issue which is also mentioned here:
https://bugs.gentoo.org/582130

I tried reverting the libxml2 fix so that it will use the wrapper one instead:
From ${ROOT%/}${PYTHON} to ${PYTHON}, but I still get the same issue

This happens when cross compiling a 32bit environment on a 64bit environment.
Also mentioned on here at: #9822 (comment)
Any idea how to fix this?

Update:

I got python-config --includes to return the right includes paths by adding the following to python-utils-r1.eclass:

            cat > "${workdir}/lib/sitecustomize.py" <<-_EOF_ || die
                import sys
                import site

                sys.path, sys_path = sys.path[:1], sys.path[1:]
                site.addsitedir('${pysysroot}/usr/lib/${EPYTHON}/site-packages')
                sys.path.extend(sys_path)

                sys.base_prefix = '${pysysroot}/usr'
                sys.prefix = '${pysysroot}/usr'
            _EOF_

and the following on top of ${workdir}/bin/python-config so it will use the sitecustomize.py:

export PYTHONPATH="${workdir}/lib\${PYTHONPATH:+:}\${PYTHONPATH}"

It still does not work for vim with python support. It seems that this is because it uses python-single-r1.eclass which only creates a single wrapper (for one version of python), but the build uses multiple versions of python. I do not know enough about this any/single logic to be able to fix it.

[chewi]

@deorder, I get the impression you haven't actually tried my work here. To be clear, despite the age of this pull request, it hasn't been merged yet. Changes have been requested and related parts of PMS need to be clarified first.

Although this work tackles many of the common issues, the python2.7-config --includes point you've raised may still be a problem but I haven't seen it cause any issues in practise.

[deorder]

@deorder, I get the impression you haven't actually tried my work here. To be clear, despite the age of this pull request, it hasn't been merged yet. Changes have been requested and related parts of PMS need to be clarified first.

Although this work tackles many of the common issues, the python2.7-config --includes point you've raised may still be a problem but I haven't seen it cause any issues in practise.

I am using everything that is included in this pull request. Using your modifications I was able to cross-compile a RPi3 64bit environment without issues where before it did not. When I build a RPi2 32bit environment however I get the error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?) during the build of some packages that use Python or are Python modules. By adding the modifications to sitecustomize.py (which makes python*-config --includes --ldflags return the right paths) I was able to build half of the packages that first did not, but still not everything. I am not criticizing your work here as it actually helps, but there is still something missing that cause 32bit packages to sometimes fail when building on a 64bit machine.

[deorder]

Okay I found another method that works without modifying any eclass file. You can find it at:
https://github.com/deorder/gentoo-crossdev-tools
It creates a wrapper environment in /tmp

You can use it with my overlay at:
https://github.com/deorder/gentoo-overlay

Also have a look at the example profiles.

Using this I was able to cross-compile a stage3 for the following targets: aarch64-rpi3hs-linux-musleabi, aarch64-rpi3s-linux-gnueabi, armv7a-rpi2hs-linux-musleabihf, armv7a-rpi2s-linux-gnueabih

[andrew-aladev]

If you have an AMD threadripper - you can just rebuild everything using qemu. I am working on these images here. But I have to build dev-python/pyblake2 and sys-apps/portage somehow before chrooting inside.

I am going just to ignore new python cross compile eclass feature (for now) and return back to monkey patching of python itself. I am building cross image inside "host" native image, so I can do anything with this "host" python.

Please do not forget about this pull request, people are watching. Thank you.

[chewi]

Don't worry, I haven't. I was looking at it last week. @mgorny said it was too complicated and he was right because trying to fix the bits that have broken in the meantime is making my head spin. I have a rough idea for a simpler approach.

[andrew-aladev]

I've just created more less reliable workaround for now.

Python 3.6 already includes right workaround: prefix_real=$(installed_prefix "$0"). It provides like /usr/arm-unknown-linux-gnueabi for any directory used by python. Python 2.7 should be fixed by monkey patching in the same way.

+-------------------------+
|                         |
|           +-----------+ |              +-----------+
|           |           | |              |           |
| native    | cross     | |    export    | cross     |
| container | container | | +----------> | container |
|           |           | |              | (+ qemu)  |
|           +-----------+ |              |           |
|                         |              +-----------+
+-------------------------+

I am patching this native container, it will be removed.

arm-unknown-linux-gnueabi-emerge -v1 dev-lang/python

find /usr/arm-unknown-linux-gnueabi/lib -maxdepth 1 -name ld* \
  -exec cp "{}" /lib/ \;
sed -i "s/export PYTHON=\${EPREFIX}\/usr\/bin\/\${impl}/export PYTHON=arm-unknown-linux-gnueabi-\${impl}/g" \
  /usr/portage/eclass/python-utils-r1.eclass
sed -i "s/\${EPYTHON:-python}/arm-unknown-linux-gnueabi-\${EPYTHON:-python}/g" \
  /usr/portage/eclass/distutils-r1.eclass
sed -i "s/esetup.py install \(.*\)/esetup.py install \1 --prefix=\"\/usr\"/g" \
  /usr/portage/eclass/distutils-r1.eclass
find /usr/portage/sys-apps/portage -maxdepth 1 -name portage-*.ebuild \
  -exec sed -i "s/\${D%\/}\${PYTHON_SITEDIR}/\${D%\/}\${PYTHON_SITEDIR#\${EROOT%\/}}/g" "{}" \; \
  -exec ebuild "{}" manifest \;

cp arm-unknown-linux-gnueabi-python3.6 /usr/bin/
ln -s /usr/bin/arm-unknown-linux-gnueabi-python3.6 /usr/bin/arm-unknown-linux-gnueabi-python3
ln -s /usr/bin/arm-unknown-linux-gnueabi-python3.6 /usr/bin/arm-unknown-linux-gnueabi-python

arm-unknown-linux-gnueabi-python3.6 just runs installed python (with all required config for target platform) using qemu inside native container:

#!/bin/bash

quoted_args=""

for arg in "$@"; do
  quoted_args+=$(printf " %q" "$arg")
done

LD_LIBRARY_PATH="/usr/arm-unknown-linux-gnueabi/lib:/usr/arm-unknown-linux-gnueabi/usr/lib:/usr/arm-unknown-linux-gnueabi/usr/lib/gcc/arm-unknown-linux-gnueabi/9.2.0/" \
  eval /usr/arm-unknown-linux-gnueabi/usr/bin/python3.6 "$quoted_args"

After that arm-unknown-linux-gnueabi-emerge -v1 sys-apps/portage works perfect. You can emerge many python extensions using this workaround. The only problem is the usage of PYTHON_ variables from python_export (eclass/python-utils-r1.eclass). For example pkg_preinst from sys-apps/portage:

[[ -d ${D%/}${PYTHON_SITEDIR} ]] || die "${D%/}${PYTHON_SITEDIR}: No such directory"

$D is a local image path, $PYTHON_SITEDIR is the global python path including /usr/arm-unknown-linux-gnueabi prefix. ${D%/}${PYTHON_SITEDIR} provides the right path only when EROOT equals /. So any mix of $D and $PYTHON_ need to be fixed for this workaround.

${D%/}${PYTHON_SITEDIR#${EROOT%/}}

See full example here.

[uis246]

Any news?

[chewi]

Not forgotten but no news, sorry.

[chewi]

This is obsolete now, mainly thanks to the wonderful gpep517. Meson currently still builds native extensions with the wrong filename, but I've almost fixed that.