Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3-lxml: trying to link host python3 library #22330

Closed
egorenar opened this issue Oct 7, 2023 · 19 comments · Fixed by #22551
Closed

python3-lxml: trying to link host python3 library #22330

egorenar opened this issue Oct 7, 2023 · 19 comments · Fixed by #22551
Assignees

Comments

@egorenar
Copy link
Contributor

egorenar commented Oct 7, 2023

Maintainer: @commodo
Environment: ARMv7 Processor rev 4 (v7l), OpenWrt SNAPSHOT r24099-5dfd8dc7d4f5

python3-lxml fails to build because it tries to link host's Python3 library.
The compiler is using -L/usr/lib.

make V=sc -j$(($(nproc)+1)) package/feeds/packages/python-lxml/{clean,compile}
...
arm-openwrt-linux-muslgnueabi-gcc -shared -L/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/usr/lib -L/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/lib -fuse-ld=bfd -znow -zrelro -lpython3.11 -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -mfloat-abi=hard -fmacro-prefix-map=/home/egorenar/Repositories/openwrt-rel/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/pypi/lxml-4.9.3=lxml-4.9.3 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/usr/include -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/include/fortify -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/include -I/home/egorenar/Repositories/openwrt-rel/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/include/python3.11 build/temp.linux-arm-cpython-311/src/lxml/builder.o -L/usr/lib -o build/lib.linux-arm-cpython-311/lxml/builder.cpython-311-arm-linux-musleabihf.so
/usr/lib/libpython3.11.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
@egorenar
Copy link
Contributor Author

egorenar commented Oct 8, 2023

For some reason, OpenWrt is using staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py instead of build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/command/build_ext.py to build lxml.builder extension module.

@egorenar
Copy link
Contributor Author

egorenar commented Oct 8, 2023

Another hint:

$ rg "LIBDIR'.*/usr/lib" staging_dir
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
708: 'LIBDIR': '/usr/lib',

Very similar to this #9275.

@egorenar
Copy link
Contributor Author

egorenar commented Oct 8, 2023

For some reason build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/sysconfig.py change from patch lang/python/python3/patches/008-distutils-use-python-sysroot.patch has either been reverted or never applied.

@jefferyto
Copy link
Member

I'm able to build python-lxml with no issues. Can you try building with a clean buildroot/sdk?

@egorenar
Copy link
Contributor Author

egorenar commented Oct 8, 2023

I did a clean build with master and LIBDIR is still /usr/lib in staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py.

If i change LIBDIR by hand to ... staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib then everything compiles.

@jefferyto
Copy link
Member

jefferyto commented Oct 8, 2023

Can you post a complete build log?

Edit: also what OS/version is your build host?

Edit 2: please include the build log for python as well as python-lxml

@egorenar
Copy link
Contributor Author

egorenar commented Oct 9, 2023

Can you post a complete build log?

Edit: also what OS/version is your build host?

Edit 2: please include the build log for python as well as python-lxml

Linux distro: Void Linux (native Python 3.11 installed in /usr/lib !!!)
Build log for python3 and python-lxml attached:

make V=sc -j$(($(nproc)+1)) package/feeds/packages/python3/{clean,compile} 2>&1 | tee build.log
make V=sc -j$(($(nproc)+1)) package/python-lxml/{clean,compile} 2>&1 | tee -a build.log

build.log.gz

thanks for looking into it.

@jefferyto
Copy link
Member

Do you get the same error if you compile for an existing arch/target instead of your custom target?

@egorenar
Copy link
Contributor Author

egorenar commented Oct 14, 2023

Built the target ipq806x model Nighthawk X4S R7800 from master 359a6e36e972 ("mac80211: ath11k: sync with ath-next"). Enabled python3-lxml as the only extra packaged. The problem persists.

Build instructions:

$ git clone https://github.com/openwrt/openwrt.git openwrt-test
$ cd openwrt-test
$ make menuconfig
# Target: ipq806x
# Model: Nighthawk X4S R7800
$ ./scripts/feeds update -a && ./scripts/feeds install -a
$ make menuconfig
# select python3-lxml
$ grep lxml .config
CONFIG_PACKAGE_python3-lxml=y
$ make V=s -j$(($(nproc)+1)) 2>&1 | tee build.log

$ grep '\bLIBDIR\b' staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
 'LIBDIR': '/usr/lib',

$ rg "sysconfig.get_config_var.*LIBDIR" staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/distutils/command/build_ext.py
237:                libdir = sysconfig.get_config_var('LIBDIR')

build.log.gz

@egorenar
Copy link
Contributor Author

Oh forgot about another interesting fact :)
Void Linux updated Python 3 to version 3.12 recently and since then everything builds again, lxml as well, no failures.
I think the reason is because OpenWrt builds Python 3.11 which is no longer present on Void Linux, and, therefore, OpenWrt fails to link the native Void Linux library.

But still, the problem is that LIBDIR is pointing to /usr/lib.

@egorenar
Copy link
Contributor Author

I found something out.
python3-lxml uses staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py to compile C extensions.

Made the following change:

--- staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py.orig       2023-10-14 14:30:47.564996239 +0200
+++ staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py    2023-10-14 14:28:43.620002109 +0200
@@ -240,6 +240,9 @@
     def build_extension(self, ext):
         ext._convert_pyx_sources_to_lang()
         _compiler = self.compiler
+
+        log.warn("--- DEBUG1 --: {}".format(self.library_dirs))
+
         try:
             if isinstance(ext, Library):
                 self.compiler = self.shlib_compiler

And rebuilding the package produces this output:

...
--- DEBUG1 --: ['/usr/lib']
...

I still do not understand it properly, but there is no "_python_sysroot" support in this build_ext.py as described here: #9288

@egorenar
Copy link
Contributor Author

egorenar commented Oct 14, 2023

Next finding :)
When lxml is being built then staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py inherits from staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py.
And the latter also doesn't support _python_sysroot.

        if sysconfig.get_config_var('Py_ENABLE_SHARED'):
            if not sysconfig.python_build:
                # building third party extensions
                self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
            else:
                # building python standard extensions
                self.library_dirs.append('.')

No "_python_sysroot" :/

@egorenar
Copy link
Contributor Author

Do we need to patch setuptool's build_ext.py ?
As it is done here: https://github.com/openwrt/packages/blob/master/lang/python/python3/patches/008-distutils-use-python-sysroot.patch

@egorenar
Copy link
Contributor Author

I applied this patch https://github.com/openwrt/packages/blob/53dc7146f4623a99b62da5918b3f4ce002697a32/lang/python/python3/patches/008-distutils-use-python-sysroot.patch#L30C18-L33C50
to staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py
and i no longer see -L/usr/lib. Instead i see -L/home/egorenar/Repositories/openwrt-test/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib

@jefferyto
Copy link
Member

The buildbots have no problem building this package. I am using Ubuntu 23.04 with Python 3.11.4 installed and I have no problem building this package.

I'd like to understand what is different about your setup that is causing you problems before applying any solutions.

@jefferyto
Copy link
Member

Are you using Void Linux with glibc or musl?

@egorenar
Copy link
Contributor Author

Are you using Void Linux with glibc or musl?

glibc

jefferyto added a commit to jefferyto/openwrt-packages that referenced this issue Oct 30, 2023
setuptools provides a local copy of distutils and when building a C
extension, this distutils will add the target LIBDIR (/usr/lib) to the
list of library paths.

If the build system has a libpython3.11.so in /usr/lib, then the linker
will try to link to this shared library and fail.

This adapts 008-distutils-use-python-sysroot.patch for host setuptools
to add the correct library directory.

Fixes: openwrt#22330

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
@jefferyto
Copy link
Member

So the reason why the build succeeds on Debian/Ubuntu (and I believe Fedora) is because:

  • There is no libpython3.11.so in /usr/lib (Debian/Ubuntu has multiarch directories and libpython3.11.so is in /usr/lib/x86_64-linux-gnu, I believe on Fedora the shared library is in /usr/lib64)
  • The linker still finds the correct shared library because STAGING_DIR is patched into the GCC specs

It is failing for you because libpython3.11.so is in /usr/lib on Void and the -L/usr/lib is ahead of the path patched into the GCC specs.

The fix is in #22551 - thanks for your help in diagnosing the issue and for your patience 🙏

neheb pushed a commit that referenced this issue Oct 30, 2023
setuptools provides a local copy of distutils and when building a C
extension, this distutils will add the target LIBDIR (/usr/lib) to the
list of library paths.

If the build system has a libpython3.11.so in /usr/lib, then the linker
will try to link to this shared library and fail.

This adapts 008-distutils-use-python-sysroot.patch for host setuptools
to add the correct library directory.

Fixes: #22330

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
jefferyto added a commit to jefferyto/openwrt-packages that referenced this issue Oct 31, 2023
setuptools provides a local copy of distutils and when building a C
extension, this distutils will add the target LIBDIR (/usr/lib) to the
list of library paths.

If the build system has a libpython3.11.so in /usr/lib, then the linker
will try to link to this shared library and fail.

This adapts 008-distutils-use-python-sysroot.patch for host setuptools
to add the correct library directory.

Fixes: openwrt#22330

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
(cherry picked from commit 624fb95)
@egorenar
Copy link
Contributor Author

thanks for looking into this!

1715173329 pushed a commit that referenced this issue Nov 1, 2023
setuptools provides a local copy of distutils and when building a C
extension, this distutils will add the target LIBDIR (/usr/lib) to the
list of library paths.

If the build system has a libpython3.11.so in /usr/lib, then the linker
will try to link to this shared library and fail.

This adapts 008-distutils-use-python-sysroot.patch for host setuptools
to add the correct library directory.

Fixes: #22330

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
(cherry picked from commit 624fb95)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants