byte-compile fails for cross-builds

[BenediktMorbach]

BPO	22724
Nosy	@doko42, @ncoghlan, @xdegaye, @tpetazzoni, @andrew-aladev
Dependencies	bpo-23968: rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)
Files	python_for_build_flag.patch py_build_environ.patch py_build_environ_2.patch: rebased to current tip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-10-24.23:52:39.521>
labels = ['build']
title = 'byte-compile fails for cross-builds'
updated_at = <Date 2020-01-24.20:12:01.692>
user = 'https://bugs.python.org/BenediktMorbach'

bugs.python.org fields:

activity = <Date 2020-01-24.20:12:01.692>
actor = 'xdegaye'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Cross-Build']
creation = <Date 2014-10-24.23:52:39.521>
creator = 'Benedikt.Morbach'
dependencies = ['23968']
files = ['43559', '43603', '43953']
hgrepos = []
issue_num = 22724
keywords = ['patch']
message_count = 11.0
messages = ['229969', '229970', '267350', '269359', '269617', '269675', '271698', '276300', '360366', '360379', '360651']
nosy_count = 7.0
nosy_names = ['doko', 'ncoghlan', 'rpetrov', 'xdegaye', 'Benedikt.Morbach', 'thomas-petazzoni', 'puchenyaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue22724'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[BenediktMorbach]

For cross-builds the byte-compiling and 2to3 Grammar generation fail.

This is because PYTHONPATH/sys.path includes 'build/lib.$PLATFORM', in which the compiled .so modules reside.
The host python obviously barfs on those, as they are compiled for a different architecture.

However, this directory also contains _sysconfigdata.py, which is needed at all times.

A possible fix would be to separate those two.

[BenediktMorbach]

alternatively, introduce a $(PYTHON_FOR_INSTALL), which doesn't set PYTHONPATH.

Then run those steps using that, with the Makefile setting PYTHONPATH=$(DESTDIR)$(LIBDEST), which doesn't contain the compiled modules at that point.

The first solution looks cleaner to me, but might be more tricky to implement, as e.g. Modules/getpath.c makes some assumptions about the location of the build dir.

[ncoghlan]

bpo-23968 moves _sysconfigdata.py to an arch dependendent directory

[xdegaye]

The byte-compiling for cross-builds fails because PYTHON_FOR_BUILD defined in configure, sets PYTHONPATH with the directory of the newly built extension modules shared libraries and because PYTHON_FOR_BUILD is used to run compileall.py in the Makefile. PYTHON_FOR_BUILD, i.e. the native python running on the build system, attempts to load the newly compiled libraries for the foreign host system and fails.

The problem is more difficult to reproduce after changeset 11c4f3f936e7 in bpo-22980 since the shared libraries names include now a PLATFORM_TRIPLET that prevents the wrong library to be loaded in most cases. On Android cross-builds, it should be possible to reproduce the failure when the build system and the host system have the same PLATFORM_TRIPLET. In all cases, the interpreter used to run compileall.py imports the wrong _sysconfigdata module.

To fix this problem, this patch adds the -E option to PYTHON_FOR_BUILD when running compileall in the Makefile and 'cross_compiling' is set. The patch does not fix the fact that, when 'cross_compiling' is not set, PYTHONPATH is set to '$(DESTDIR)$(LIBDEST)' in the Makefile at the beginning of the statement that runs compileall.py while PYTHON_FOR_BUILD is set to './$(BUILDPYTHON) -E', something is obviously wrong here. So the patch would be simpler if '-E' was used in both cases, but I don't know what would be lost by removing this setting of PYTHONPATH at the beginning of those compileall statements, IOW what was their initial purpose.

[xdegaye]

On Android cross-builds, it should be possible to reproduce the failure when the build system and the host system have the same PLATFORM_TRIPLET

I confirm that this is the case. The following backtrace is produced when cross-building and installing Android x86_64 on a linux x86_64:

PYTHONPATH=/sdcard/org.bitbucket.pyona/lib/python3.6   \
	_PYTHON_PROJECT_BASE=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64/build/lib.linux-x86_64-3.6:/home/xavier/src/packages/android/cpython/Lib:/home/xavier/src/packages/android/cpython/Lib/plat-x86_64-linux-gnu python3.6 -Wi /sdcard/org.bitbucket.pyona/lib/python3.6/compileall.py \
	-d /sdcard/org.bitbucket.pyona/lib/python3.6 -f \
	-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
	/sdcard/org.bitbucket.pyona/lib/python3.6
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/compileall.py", line 17, in <module>
    import struct
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/struct.py", line 13, in <module>
    from _struct import *
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
make[3]: [Makefile:1270: libinstall] Error 1 (ignored)

'python3.6' is the native interpreter built from the same source tree as the source tree used for the cross-build. PATH has been set so that this native interpreter is run. Running this same command interactively with the native interpreter gives the same result:

[xavier@bilboquet python-native]$   PYTHONPATH=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64/build/lib.linux-x86_64-3.6:/home/xavier/src/packages/android/cpython/Lib:/home/xavier/src/packages/android/cpython/Lib/plat-x86_64-linux-gnu ./python3.6
Python 3.6.0a2+ (default:64ed56fbc5e7+, Jun 27 2016, 10:36:45) 
[GCC 6.1.1 20160602] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "/home/xavier/config/.pystartup", line 14, in <module>
    import readline
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
>>> 

Cross-building and installing Android i686 on a linux x86_64 with the same Makefile does not cause the problem.

Adding -E to PYTHON_FOR_BUILD fixes the x86_64 install.

[xdegaye]

Adding -E to PYTHON_FOR_BUILD fixes the x86_64 install because in my Android cross-build setup, the native interpreter is built from the same source tree as the cross-build. However, currently the interpreter used for the cross-build may be any interpreter found in PATH and can import an incorrect importlib.util.MAGIC_NUMBER if PYTHONPATH is not set properly. So this new patch does not use -E and splits PYTHON_FOR_BUILD into two variables, PY_BUILD_ENVIRON that sets the environment variables and PYTHON_FOR_BUILD, the interpreter. PY_BUILD_ENVIRON is not used when running compileall.

Tested by running the test suite on the Android i686 and x86_64 emulators (excluding test_ctypes that segfaults python on x86_64). The tests cannot be run with a cross-build where PYTHON_FOR_BUILD is python 3.5.1 because of bpo-27434.

[xdegaye]

Patch rebased to current tip.

To summarize:

• The root cause of the problem reported by Benedikt still exists.
• The problem is hidden now that the shared libraries names include the PLATFORM_TRIPLET.
• But byte-compilation still fails for cross-builds when the 'host' and 'build' PLATFORM_TRIPLET are the same and this occurs for the x86_64 Android build on a x86_64 'build' system.

Matthias, do you have any comment on this patch?

[xdegaye]

Cross-compilation of Android x86_64 on a linux x86_64 host followed by 'make install' does not fail anymore after the changes made in bpo-27917. As expected since the triplets are different now when the target is an Android platform.

[andrew-aladev]

Hello. I've provided a double recompilation workaround in the following gentoo bug https://bugs.gentoo.org/705970. You can use it for now if you don't want to wait until this issue will be fixed.

I will try patches provided by people in this issue today's evening. Thank you.

[andrew-aladev]

I've made a quick test of all patches provided. All patches have failed to fix it. The problem is more heavy than it looks like.

Please look at the following code:

PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '${EPYTHON_FOR_BUILD:-${interp}}

It means that we will use "interp" to load cross compiled modules, env, etc from "_PYTHON_PROJECT_BASE". This code is broken by design. Loading of cross compiled modules, env, etc is not possible in general. For example you have different libc: glibc, musl, uclibc.

glibc -> cross compile -> glibc = loads fine.
musl -> cross compile -> musl = loads fine.
uclibc -> cross compile -> uclibc = loads fine.

glibc -> cross compile -> musl = failed to load.
glibc -> cross compile -> uclibc = failed to load.
musl -> cross compile -> glibc = failed to load.
musl -> cross compile -> uclibc = failed to load.
uclibc -> cross compile -> glibc = failed to load.
uclibc -> cross compile -> musl = failed to load.

Cross compilation doesn't mean only another arch, it means different libc, kernel headers, binutils and compiler. You will receive same issue while using clang toolchains.

Please assign this issue to python core developers. This issue can't be fixed by regular users without experience in python development. It requires complete change of cross compilation design.

[xdegaye]

PYTHON_FOR_BUILD does not use PYTHONPATH in the implementation of (closed) PR 17420 and should fix the current issue as a side effect.

FWIW, PR 17420 fixes cross-compilation of third-party extension modules by replacing the complex PYTHON_FOR_BUILD command line set in configure.ac by the more simple command:

"PYTHON_FOR_BUILD='PYTHON_PROJECT_BASE=$(abs_builddir) '$interp"

In this PR finding the path name of the target sysconfigdata module when needed is done by the sysconfig module and by distutils using the Misc/python-config.sh.in script instead of using PYTHONPATH.

Since PYTHONPATH is not used in PR 17420, the current issue should also be fixed by the PR. See also Benedikt OP that provided a hint for the fix.