python3-lxml: trying to link host python3 library #22330

egorenar · 2023-10-07T15:32:08Z

Maintainer: @commodo
Environment: ARMv7 Processor rev 4 (v7l), OpenWrt SNAPSHOT r24099-5dfd8dc7d4f5

python3-lxml fails to build because it tries to link host's Python3 library.
The compiler is using -L/usr/lib.

make V=sc -j$(($(nproc)+1)) package/feeds/packages/python-lxml/{clean,compile}
...
arm-openwrt-linux-muslgnueabi-gcc -shared -L/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/usr/lib -L/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/lib -fuse-ld=bfd -znow -zrelro -lpython3.11 -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -mfloat-abi=hard -fmacro-prefix-map=/home/egorenar/Repositories/openwrt-rel/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/pypi/lxml-4.9.3=lxml-4.9.3 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/usr/include -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/include/fortify -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/include -I/home/egorenar/Repositories/openwrt-rel/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/include/python3.11 build/temp.linux-arm-cpython-311/src/lxml/builder.o -L/usr/lib -o build/lib.linux-arm-cpython-311/lxml/builder.cpython-311-arm-linux-musleabihf.so
/usr/lib/libpython3.11.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status

The text was updated successfully, but these errors were encountered:

egorenar · 2023-10-08T11:00:06Z

For some reason, OpenWrt is using staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py instead of build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/command/build_ext.py to build lxml.builder extension module.

egorenar · 2023-10-08T11:07:15Z

Another hint:

$ rg "LIBDIR'.*/usr/lib" staging_dir
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
708: 'LIBDIR': '/usr/lib',

Very similar to this #9275.

egorenar · 2023-10-08T11:16:49Z

For some reason build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/sysconfig.py change from patch lang/python/python3/patches/008-distutils-use-python-sysroot.patch has either been reverted or never applied.

jefferyto · 2023-10-08T14:36:34Z

I'm able to build python-lxml with no issues. Can you try building with a clean buildroot/sdk?

egorenar · 2023-10-08T17:31:25Z

I did a clean build with master and LIBDIR is still /usr/lib in staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py.

If i change LIBDIR by hand to ... staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib then everything compiles.

jefferyto · 2023-10-08T23:30:53Z

Can you post a complete build log?

Edit: also what OS/version is your build host?

Edit 2: please include the build log for python as well as python-lxml

egorenar · 2023-10-09T16:52:59Z

Can you post a complete build log?

Edit: also what OS/version is your build host?

Edit 2: please include the build log for python as well as python-lxml

Linux distro: Void Linux (native Python 3.11 installed in /usr/lib !!!)
Build log for python3 and python-lxml attached:

make V=sc -j$(($(nproc)+1)) package/feeds/packages/python3/{clean,compile} 2>&1 | tee build.log
make V=sc -j$(($(nproc)+1)) package/python-lxml/{clean,compile} 2>&1 | tee -a build.log

build.log.gz

thanks for looking into it.

jefferyto · 2023-10-12T10:45:15Z

Do you get the same error if you compile for an existing arch/target instead of your custom target?

egorenar · 2023-10-14T09:18:47Z

Built the target ipq806x model Nighthawk X4S R7800 from master 359a6e36e972 ("mac80211: ath11k: sync with ath-next"). Enabled python3-lxml as the only extra packaged. The problem persists.

Build instructions:

$ git clone https://github.com/openwrt/openwrt.git openwrt-test
$ cd openwrt-test
$ make menuconfig
# Target: ipq806x
# Model: Nighthawk X4S R7800
$ ./scripts/feeds update -a && ./scripts/feeds install -a
$ make menuconfig
# select python3-lxml
$ grep lxml .config
CONFIG_PACKAGE_python3-lxml=y
$ make V=s -j$(($(nproc)+1)) 2>&1 | tee build.log

$ grep '\bLIBDIR\b' staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
 'LIBDIR': '/usr/lib',

$ rg "sysconfig.get_config_var.*LIBDIR" staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/distutils/command/build_ext.py
237:                libdir = sysconfig.get_config_var('LIBDIR')

build.log.gz

egorenar · 2023-10-14T09:23:15Z

Oh forgot about another interesting fact :)
Void Linux updated Python 3 to version 3.12 recently and since then everything builds again, lxml as well, no failures.
I think the reason is because OpenWrt builds Python 3.11 which is no longer present on Void Linux, and, therefore, OpenWrt fails to link the native Void Linux library.

But still, the problem is that LIBDIR is pointing to /usr/lib.

egorenar · 2023-10-14T12:32:56Z

I found something out.
python3-lxml uses staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py to compile C extensions.

Made the following change:

--- staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py.orig       2023-10-14 14:30:47.564996239 +0200
+++ staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py    2023-10-14 14:28:43.620002109 +0200
@@ -240,6 +240,9 @@
     def build_extension(self, ext):
         ext._convert_pyx_sources_to_lang()
         _compiler = self.compiler
+
+        log.warn("--- DEBUG1 --: {}".format(self.library_dirs))
+
         try:
             if isinstance(ext, Library):
                 self.compiler = self.shlib_compiler

And rebuilding the package produces this output:

...
--- DEBUG1 --: ['/usr/lib']
...

I still do not understand it properly, but there is no "_python_sysroot" support in this build_ext.py as described here: #9288

egorenar · 2023-10-14T12:41:13Z

Next finding :)
When lxml is being built then staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py inherits from staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py.
And the latter also doesn't support _python_sysroot.

        if sysconfig.get_config_var('Py_ENABLE_SHARED'):
            if not sysconfig.python_build:
                # building third party extensions
                self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
            else:
                # building python standard extensions
                self.library_dirs.append('.')

No "_python_sysroot" :/

egorenar · 2023-10-14T12:48:42Z

Do we need to patch setuptool's build_ext.py ?
As it is done here: https://github.com/openwrt/packages/blob/master/lang/python/python3/patches/008-distutils-use-python-sysroot.patch

egorenar · 2023-10-14T12:56:00Z

I applied this patch https://github.com/openwrt/packages/blob/53dc7146f4623a99b62da5918b3f4ce002697a32/lang/python/python3/patches/008-distutils-use-python-sysroot.patch#L30C18-L33C50
to staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py
and i no longer see -L/usr/lib. Instead i see -L/home/egorenar/Repositories/openwrt-test/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib

jefferyto · 2023-10-14T12:58:55Z

The buildbots have no problem building this package. I am using Ubuntu 23.04 with Python 3.11.4 installed and I have no problem building this package.

I'd like to understand what is different about your setup that is causing you problems before applying any solutions.

jefferyto · 2023-10-14T13:05:55Z

Are you using Void Linux with glibc or musl?

egorenar · 2023-10-14T13:06:11Z

Are you using Void Linux with glibc or musl?

glibc

setuptools provides a local copy of distutils and when building a C extension, this distutils will add the target LIBDIR (/usr/lib) to the list of library paths. If the build system has a libpython3.11.so in /usr/lib, then the linker will try to link to this shared library and fail. This adapts 008-distutils-use-python-sysroot.patch for host setuptools to add the correct library directory. Fixes: openwrt#22330 Signed-off-by: Jeffery To <jeffery.to@gmail.com>

jefferyto · 2023-10-30T17:21:11Z

So the reason why the build succeeds on Debian/Ubuntu (and I believe Fedora) is because:

There is no libpython3.11.so in /usr/lib (Debian/Ubuntu has multiarch directories and libpython3.11.so is in /usr/lib/x86_64-linux-gnu, I believe on Fedora the shared library is in /usr/lib64)
The linker still finds the correct shared library because STAGING_DIR is patched into the GCC specs

It is failing for you because libpython3.11.so is in /usr/lib on Void and the -L/usr/lib is ahead of the path patched into the GCC specs.

The fix is in #22551 - thanks for your help in diagnosing the issue and for your patience 🙏

setuptools provides a local copy of distutils and when building a C extension, this distutils will add the target LIBDIR (/usr/lib) to the list of library paths. If the build system has a libpython3.11.so in /usr/lib, then the linker will try to link to this shared library and fail. This adapts 008-distutils-use-python-sysroot.patch for host setuptools to add the correct library directory. Fixes: #22330 Signed-off-by: Jeffery To <jeffery.to@gmail.com>

setuptools provides a local copy of distutils and when building a C extension, this distutils will add the target LIBDIR (/usr/lib) to the list of library paths. If the build system has a libpython3.11.so in /usr/lib, then the linker will try to link to this shared library and fail. This adapts 008-distutils-use-python-sysroot.patch for host setuptools to add the correct library directory. Fixes: openwrt#22330 Signed-off-by: Jeffery To <jeffery.to@gmail.com> (cherry picked from commit 624fb95)

egorenar · 2023-10-31T19:15:51Z

thanks for looking into this!

setuptools provides a local copy of distutils and when building a C extension, this distutils will add the target LIBDIR (/usr/lib) to the list of library paths. If the build system has a libpython3.11.so in /usr/lib, then the linker will try to link to this shared library and fail. This adapts 008-distutils-use-python-sysroot.patch for host setuptools to add the correct library directory. Fixes: #22330 Signed-off-by: Jeffery To <jeffery.to@gmail.com> (cherry picked from commit 624fb95)

BKPepe assigned jefferyto Oct 8, 2023

jefferyto mentioned this issue Oct 30, 2023

python3: Fix building C extensions with setuptools #22551

Merged

neheb closed this as completed in #22551 Oct 30, 2023

jefferyto mentioned this issue Oct 31, 2023

[openwrt-23.05] python3: Fix building C extensions with setuptools #22560

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3-lxml: trying to link host python3 library #22330

python3-lxml: trying to link host python3 library #22330

egorenar commented Oct 7, 2023 •

edited by BKPepe

Loading

egorenar commented Oct 8, 2023

egorenar commented Oct 8, 2023

egorenar commented Oct 8, 2023

jefferyto commented Oct 8, 2023

egorenar commented Oct 8, 2023

jefferyto commented Oct 8, 2023 •

edited

Loading

egorenar commented Oct 9, 2023 •

edited

Loading

jefferyto commented Oct 12, 2023

egorenar commented Oct 14, 2023 •

edited

Loading

egorenar commented Oct 14, 2023

egorenar commented Oct 14, 2023

egorenar commented Oct 14, 2023 •

edited

Loading

egorenar commented Oct 14, 2023

egorenar commented Oct 14, 2023

jefferyto commented Oct 14, 2023

jefferyto commented Oct 14, 2023

egorenar commented Oct 14, 2023

jefferyto commented Oct 30, 2023

egorenar commented Oct 31, 2023

python3-lxml: trying to link host python3 library #22330

python3-lxml: trying to link host python3 library #22330

Comments

egorenar commented Oct 7, 2023 • edited by BKPepe Loading

egorenar commented Oct 8, 2023

egorenar commented Oct 8, 2023

egorenar commented Oct 8, 2023

jefferyto commented Oct 8, 2023

egorenar commented Oct 8, 2023

jefferyto commented Oct 8, 2023 • edited Loading

egorenar commented Oct 9, 2023 • edited Loading

jefferyto commented Oct 12, 2023

egorenar commented Oct 14, 2023 • edited Loading

egorenar commented Oct 14, 2023

egorenar commented Oct 14, 2023

egorenar commented Oct 14, 2023 • edited Loading

egorenar commented Oct 14, 2023

egorenar commented Oct 14, 2023

jefferyto commented Oct 14, 2023

jefferyto commented Oct 14, 2023

egorenar commented Oct 14, 2023

jefferyto commented Oct 30, 2023

egorenar commented Oct 31, 2023

egorenar commented Oct 7, 2023 •

edited by BKPepe

Loading

jefferyto commented Oct 8, 2023 •

edited

Loading

egorenar commented Oct 9, 2023 •

edited

Loading

egorenar commented Oct 14, 2023 •

edited

Loading

egorenar commented Oct 14, 2023 •

edited

Loading