Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libgpiod: does not compile on macOS 11.2-x86_64 host #14652

Closed
k-ronny opened this issue Feb 4, 2021 · 27 comments · Fixed by #14767
Closed

libgpiod: does not compile on macOS 11.2-x86_64 host #14652

k-ronny opened this issue Feb 4, 2021 · 27 comments · Fixed by #14767
Assignees

Comments

@k-ronny
Copy link
Contributor

k-ronny commented Feb 4, 2021

Maintainer: @mhei
Environment: all?

Description: cross compiling of libgpiod on a macOS 11.2-x86_64 host is failing

...
gpiodmodule.c:8:10: fatal error: Python.h: No such file or directory
 #include <Python.h>
          ^~~~~~~~~~
compilation terminated.
...

I think it is related to this definition in the Makefile:

CONFIGURE_VARS += \
        PYTHON="$(STAGING_DIR_HOSTPKG)/bin/$(PYTHON3)" \
        PYTHON_CPPFLAGS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --includes)" \
        PYTHON_LIBS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --libs)"
endif

If I apply this patch it compiles on macOS and Debian Buster and it runs on a Onion Omega2+:

diff --git a/libs/libgpiod/Makefile b/libs/libgpiod/Makefile
index 68996c831..b944dbcfa 100644
--- a/libs/libgpiod/Makefile
+++ b/libs/libgpiod/Makefile
@@ -33,9 +33,8 @@ endif
 ifneq ($(CONFIG_PACKAGE_python3-gpiod),)
 CONFIGURE_ARGS += --enable-bindings-python
 CONFIGURE_VARS += \
-       PYTHON="$(STAGING_DIR_HOSTPKG)/bin/$(PYTHON3)" \
-       PYTHON_CPPFLAGS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --includes)" \
-       PYTHON_LIBS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --libs)"
+       PYTHON_CPPFLAGS="-I$(PYTHON3_INC_DIR)" \
+       PYTHON_LIBS="-lpython$(PYTHON3_VERSION)"
 endif
 
 define Package/libgpiod

But I do not know if this is the right approach.

@mhei mhei self-assigned this Feb 4, 2021
@neheb
Copy link
Contributor

neheb commented Feb 8, 2021

ping @jefferyto

@neheb
Copy link
Contributor

neheb commented Feb 11, 2021

@k-ronny what does $(STAGING_DIR)/host/bin/$(PYTHON3)-config --includes actually return?

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 11, 2021

@neheb

k_ronny@pollux openwrt/ > staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
zsh: staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config: bad interpreter: /usr/bin/python3.9: no such file or directory

this is because

k_ronny@pollux openwrt/ > head -1 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config 
#!/usr/bin/python3.9

and in my build environment python3 is

k_ronny@pollux openwrt/ > type python3
python3 is /usr/local/bin/python3

this results in

<...> PYTHON="/Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9" PYTHON_CPPFLAGS="" PYTHON_LIBS="" <...>

and later to

gmake[7]: Entering directory '/Volumes/work/openwrt/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python'
/bin/sh ../../libtool <...> -I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9 <...>
In file included from /Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9/Python.h:63,
                 from gpiodmodule.c:8:
/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9/pyport.h:741:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
gmake[7]: *** [Makefile:525: gpiod_la-gpiodmodule.lo] Error 1 

with the Makefile changes I get

PYTHON_CPPFLAGS="-I/Volumes/work/openwrt/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9" PYTHON_LIBS="-lpython3.9"

and later

gmake[7]: Entering directory '/Volumes/work/openwrt/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python'
/bin/sh ../../libtool <...> -I/Volumes/work/openwrt/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9 <...>
libtool: link: mipsel-openwrt-linux-musl-gcc-ar cr .libs/gpiod.a  gpiod_la-gpiodmodule.o
libtool: link: mipsel-openwrt-linux-musl-gcc-ranlib .libs/gpiod.a
libtool: link: ( cd ".libs" && rm -f "gpiod.la" && ln -s "../gpiod.la" "gpiod.la" )
gmake[7]: Leaving directory '/Volumes/work/openwrt/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python'

and a working module

some additional info:

k_ronny@pollux openwrt/ > /usr/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes               
-I/Library/Python/3.8/include -I/Library/Python/3.8/include

k_ronny@pollux openwrt/ > /usr/local/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
-I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9

k_ronny@pollux openwrt/ > staging_dir/hostpkg/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
-I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9 -I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9

k_ronny@pollux openwrt/ > staging_dir/host/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
-I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9

@mhei
Copy link
Member

mhei commented Feb 11, 2021

So there is definitely something wrong with your Python: python3-config is a simple shell script and should start with "#!/bin/sh" - it's not a Python script itself.

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 11, 2021

@mhei
what can I say...
on macOS

k_ronny@pollux openwrt/ > python3 
Python 3.9.1 (default, Feb  3 2021, 07:38:02) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

k_ronny@pollux openwrt/ > ls -l $(find staging_dir -name "pyth*config*")                                                                                                      
lrwxr-xr-x 1 k_ronny staff   16 Feb 11 16:07 staging_dir/hostpkg/bin/python3-config -> python3.9-config
-rwxr-xr-x 1 k_ronny staff 2,1K Feb 11 16:07 staging_dir/hostpkg/bin/python3.9-config
-rwxr-xr-x 1 k_ronny staff 2,1K Feb 11 16:07 staging_dir/hostpkg/lib/python3.9/config-3.9/python-config.py
-rwxr-xr-x 1 k_ronny staff 2,0K Jan 29 03:22 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
-rwxr-xr-x 1 k_ronny staff 2,0K Jan 29 03:22 staging_dir/target-mipsel_24kc_musl/usr/lib/python3.9/config-3.9/python-config.py

k_ronny@pollux openwrt/ > cat staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/usr/bin/python3.9
# -*- python -*-

# Keep this script in sync with python-config.sh.in

import getopt
import os
import sys
import sysconfig
...

the shell scripts are gone???
bin/python3.9-config and lib/python3.9/config-3.9/python-config.py are the same

on Debian Buster it looks right

k_ronny@openwrt-build openwrt/ > python3       
Python 3.9.1+ (default, Feb  5 2021, 13:46:56) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

k_ronny@openwrt-build openwrt/ > ls -l $(find staging_dir -name "pyth*config*")                                       
lrwxrwxrwx 1 k_ronny k_ronny   16  6. Jan 14:13 staging_dir/hostpkg/bin/python3-config -> python3.9-config
-rwxr-xr-x 1 k_ronny k_ronny 3,3K  6. Jan 14:13 staging_dir/hostpkg/bin/python3.9-config
-rwxr-xr-x 1 k_ronny k_ronny 2,1K  6. Jan 14:13 staging_dir/hostpkg/lib/python3.9/config-3.9/python-config.py
-rwxr-xr-x 1 k_ronny k_ronny 3,8K  6. Jan 14:19 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
-rwxr-xr-x 1 k_ronny k_ronny 2,0K  6. Jan 14:19 staging_dir/target-mipsel_24kc_musl/usr/lib/python3.9/config-3.9/python-config.py

k_ronny@openwrt-build openwrt/ > cat staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/bin/sh

# Keep this script in sync with python-config.in

exit_with_usage ()
...

now I am cleaning up and rebuilding the whole thing, maybe my problem will go away

@neheb
Copy link
Contributor

neheb commented Feb 11, 2021

ping @ynezz @httpstorm Any idea here?

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 11, 2021

@mhei @neheb

in build_dir/hostpkg/Python-3.9.1/Makefile you can find this

...
python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
        @ # Substitution happens here, as the completely-expanded BINDIR
        @ # is not available in configure
        sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
        @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
        LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
        @ # On Darwin, always use the python version of the script, the shell
        @ # version doesn't use the compiler customizations that are provided
        @ # in python (_osx_support.py).
        @if test `uname -s` = Darwin; then \
                cp python-config.py python-config; \
        fi
...

@httpstorm
Copy link
Contributor

@neheb
For me it builds both with or without changing the Makefile for libgpiod.
make package/feeds/packages/libgpiod/{clean,compile} -j 16

@k-ronny
Would you be able to provide me with your .config, so I can test the same configuration? I have tried my existing setup for WRT3200ACM, as well as a blank build for TL-1043ND v4 with only libgpiod added. Both build successfully.

At this point I have the following suspects:

  • Building from an empty directory often helps.
  • I seem to have python3 installed in both brew and MacPorts.
  • You might be missing some build tools, or have them installed in both brew and MacPorts. Usually this should not matter, except for a recent change which only affects pkg-config, and requires the first occurrence in $PATH to be under /usr. They switched from which -a to command -pv which returns only the first occurrence in $PATH, but forgot to remove the filter. I should probably fix this since the author of that PR is trying to revert the change completely, instead of trying to correct it. Revert "build: replace which with Bash command built-in" openwrt#3820 (comment)
  • As a test you might create a symbolic link /usr/bin/python3.9 -> /usr/local/bin/python3.9 though I don't have it so it should not be necessary for the build.
  • What actually surprised me is that you are not getting build errors related to perl, and I cannot compile it without this patch perl: fix build errors on macOS #14676

Here is some data from my environment:

./staging_dir/hostpkg/bin/python3-config --includes
-I/Volumes/aaa/openwrt/staging_dir/hostpkg/include/python3.9 -I/Volumes/aaa/openwrt/staging_dir/hostpkg/include/python3.9

./staging_dir/target-mips_24kc_musl/host/bin/python3.9-config --includes
-bash: ./staging_dir/target-mips_24kc_musl/host/bin/python3.9-config: /usr/bin/python3.9: bad interpreter: No such file or directory

head -1 ./staging_dir/hostpkg/bin/python3-config 
#!/Volumes/aaa/openwrt/staging_dir/hostpkg/bin/python3.9

head -1 staging_dir/target-mips_24kc_musl/host/bin/python3.9-config 
#!/usr/bin/python3.9

ls /usr/bin/python3.9
ls: /usr/bin/python3.9: No such file or directory

which -a python3
/usr/local/bin/python3
/usr/bin/python3

which -a python3.9
/usr/local/bin/python3.9

python3
Python 3.9.1 (default, Jan  8 2021, 17:17:43) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Good luck!

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 12, 2021

@httpstorm
this is my environment (I use only Homebrew, there is and never was a MacPorts installation)

k_ronny@pollux openwrt/ > echo $path
/usr/local/bin /usr/bin /bin

k_ronny@pollux openwrt/ > ./staging_dir/hostpkg/bin/python3-config --includes
-I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9 -I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9

k_ronny@pollux openwrt/ > ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
zsh: ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config: bad interpreter: /usr/bin/python3.9: no such file or directory

k_ronny@pollux openwrt/ > head -1 ./staging_dir/hostpkg/bin/python3-config
#!/Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9

k_ronny@pollux openwrt/ > head -1 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/usr/bin/python3.9

k_ronny@pollux openwrt/ > ls /usr/bin/python3.9
ls: /usr/bin/python3.9: No such file or directory

k_ronny@pollux openwrt/ > which -a python3
/usr/local/bin/python3
/usr/bin/python3

k_ronny@pollux openwrt/ > which -a python3.9
/usr/local/bin/python3.9

k_ronny@pollux openwrt/ > python3 -VV
Python 3.9.1 (default, Feb  3 2021, 07:38:02) 
[Clang 12.0.0 (clang-1200.0.32.29)]

k_ronny@pollux openwrt/ > perl -v
This is perl 5, version 28, subversion 2 (v5.28.2) built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)

Copyright 1987-2019, Larry Wall
...

???
I can not get it to use the right include dir, it should use

<...>/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9

but it uses

<...>/staging_dir/hostpkg/include/python3.9

here is my .config

CONFIG_TARGET_ramips=y
CONFIG_TARGET_ramips_mt76x8=y
CONFIG_TARGET_ramips_mt76x8_DEVICE_mediatek_linkit-smart-7688=y
CONFIG_BUSYBOX_CUSTOM=y
CONFIG_BUSYBOX_CONFIG_STTY=y
# CONFIG_PACKAGE_dnsmasq is not set
# CONFIG_PACKAGE_firewall is not set
CONFIG_PACKAGE_gpiod-tools=y
CONFIG_PACKAGE_i2c-tools=y
# CONFIG_PACKAGE_ip6tables is not set
# CONFIG_PACKAGE_iptables is not set
CONFIG_PACKAGE_kmod-i2c-core=y
CONFIG_PACKAGE_kmod-i2c-mt7628=y
# CONFIG_PACKAGE_kmod-ip6tables is not set
# CONFIG_PACKAGE_kmod-ipt-conntrack is not set
# CONFIG_PACKAGE_kmod-ipt-core is not set
# CONFIG_PACKAGE_kmod-ipt-nat is not set
# CONFIG_PACKAGE_kmod-ipt-offload is not set
CONFIG_PACKAGE_kmod-ledtrig-netdev=y
# CONFIG_PACKAGE_kmod-lib-crc-ccitt is not set
CONFIG_PACKAGE_kmod-mtd-rw=m
# CONFIG_PACKAGE_kmod-nf-conntrack is not set
# CONFIG_PACKAGE_kmod-nf-conntrack6 is not set
# CONFIG_PACKAGE_kmod-nf-flow is not set
# CONFIG_PACKAGE_kmod-nf-ipt is not set
# CONFIG_PACKAGE_kmod-nf-ipt6 is not set
# CONFIG_PACKAGE_kmod-nf-nat is not set
# CONFIG_PACKAGE_kmod-nf-reject is not set
# CONFIG_PACKAGE_kmod-nf-reject6 is not set
# CONFIG_PACKAGE_kmod-ppp is not set
CONFIG_PACKAGE_kmod-usb-serial=y
CONFIG_PACKAGE_kmod-usb-serial-ftdi=y
CONFIG_PACKAGE_libbz2=y
CONFIG_PACKAGE_libffi=y
CONFIG_PACKAGE_libgpiod=y
CONFIG_PACKAGE_libi2c=y
# CONFIG_PACKAGE_libip4tc is not set
# CONFIG_PACKAGE_libip6tc is not set
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblua5.3=y
CONFIG_PACKAGE_liboniondebug=y
CONFIG_PACKAGE_libonioni2c=y
CONFIG_PACKAGE_libonionoledexp=y
CONFIG_PACKAGE_librt=y
CONFIG_PACKAGE_libubox-lua=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libuci-lua=y
CONFIG_PACKAGE_libuuid=y
# CONFIG_PACKAGE_libxtables is not set
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_lua-periphery=y
CONFIG_PACKAGE_lua-periphery5.3=y
CONFIG_PACKAGE_lua5.3=y
CONFIG_PACKAGE_luasocket=y
CONFIG_PACKAGE_luasocket5.3=y
CONFIG_PACKAGE_netcat=y
# CONFIG_PACKAGE_odhcp6c is not set
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
CONFIG_PACKAGE_oled-exp=y
# CONFIG_PACKAGE_ppp is not set
CONFIG_PACKAGE_procd-seccomp=y
CONFIG_PACKAGE_python3-base=y
CONFIG_PACKAGE_python3-ctypes=y
CONFIG_PACKAGE_python3-gpiod=y
CONFIG_PACKAGE_python3-light=y
CONFIG_PACKAGE_python3-oled-exp=y
CONFIG_PACKAGE_python3-omega-expansion=y
CONFIG_PACKAGE_python3-onion-i2c=y
CONFIG_PACKAGE_python3-periphery=y
CONFIG_PACKAGE_python3-smbus=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-rpcsys=y
CONFIG_PACKAGE_umdns=y
CONFIG_PACKAGE_zlib=y

@httpstorm
Copy link
Contributor

@k-ronny
Can you please provide the entire .config? Perhaps post it on pastern.com
One thing we can test is changing ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config, to use perl from /Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9.
Do you know which code is used to generate this script?

My brand new MacBook Pro 16 is totally borked at the moment, and I need to show it to the Apple support in about two hours before I can restart it. It's been making kernel panics whenever I restart it. I might be able to do some testing after that, and once I have the entire .config.

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 12, 2021

@httpstorm
I think there is no way to get the right include dir with this python version of the config script. It will always print out options for the build host python and not for the target. And I do not know how perl is involved here I see only some sed magic in <...>/build_dir/hostpkg/Python-3.9.1/Makefile.pre.in line 1578ff.

# bpo-21536: Misc/python-config.sh is generated in the build directory
# from $(srcdir)Misc/python-config.sh.in.
python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
        @ # Substitution happens here, as the completely-expanded BINDIR
        @ # is not available in configure
        sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
        @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
        LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config

If I enable the installation of the shell script version all is good.

e.g. <...>/feeds/packages/lang/python/python3/patches/666-macos-config.patch

--- a/Makefile.pre.in	2021-02-12 00:16:20.000000000 +0100
+++ b/Makefile.pre.in	2021-02-12 17:44:08.000000000 +0100
@@ -1583,12 +1583,6 @@
 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
-	@ # On Darwin, always use the python version of the script, the shell
-	@ # version doesn't use the compiler customizations that are provided
-	@ # in python (_osx_support.py).
-	@if test `uname -s` = Darwin; then \
-		cp python-config.py python-config; \
-	fi
 
 
 # Install the include files

But this means it will not use macOS specific options. Is this a problem?

@httpstorm
Copy link
Contributor

Hello @k-ronny
Please provide me with the full .config file, so I can try to build the same configuration!
You might also try building from my fork:

git clone https://github.com/httpstorm/openwrt.git
git checkout gvalkov

Before building, add the patches from this PR to feeds/packages/libs/perl/patches https://github.com/openwrt/packages/pull/14676/files

I don't think perl is involved in the python build. I mentioned it, because it itself fails to build on macOS. Accidentally the perl build fails because it fails to find the current directory (cwd).

Here is my list of installed brew and MacPorts packages:
https://pastebin.com/MvWrMpAP

Good luck!

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 13, 2021

@httpstorm
o.k. you can download my .config from here: config.txt but I am afraid there are some packages I have added that are not available to you.

My brew installed packages are

  • coreutils
  • findutils
  • gawk
  • gnu-getopt
  • gnu-tar
  • grep
  • make
  • wget

and it's dependencies.

I have just started a build from your tree.

@httpstorm
Copy link
Contributor

Hello @k-ronny !
I reproduced your issue using the .config you provided. Then I tried to compile the bare minimum:

  • Target System (MediaTek Ralink MIPS)
  • Subtarget (MT76x8 based boards)
  • Target Profile (MediaTek LinkIt Smart 7688)
  • libgpiod

The build succeeded. Then I added:

  • python3-gpiod
  • gpiod-tools

The build failed reproducing your issue. I will look into this soon later today or tomorrow.
Good luck!

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 13, 2021

@httpstorm
oh I already know the reason for this, see #14652 (comment).
<...>/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python/gpiodmodule.c includes
<...>/staging_dir/hostpkg/include/python3.9/Python.h which includes
<...>/staging_dir/hostpkg/include/python3.9/pyport.h where you can find this

...
#ifndef LONG_BIT
#define LONG_BIT (8 * SIZEOF_LONG)
#endif

#if LONG_BIT != 8 * SIZEOF_LONG
/* 04-Oct-2000 LONG_BIT is apparently (mis)defined as 64 on some recent
 * 32-bit platforms using gcc.  We try to catch that here at compile-time
 * rather than waiting for integer multiplication to trigger bogus
 * overflows.
 */
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
#endif
...

and pyport.h includes
<...>/staging_dir/hostpkg/include/python3.9/pyconfig.h where you can find this

/* The size of `long', as computed by sizeof. */
#define SIZEOF_LONG 8

I am compiling on a platform where LONG is 8 (macOS) for a platform where LONG is 4 (mipsel)

so the compiler should use
<...>/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9/pyport.h and than
<...>/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9/pyconfig.h where LONG is defined as

/* The size of `long', as computed by sizeof. */
#define SIZEOF_LONG 4

@jefferyto
Copy link
Member

Thanks @k-ronny for debugging this - I think if we install the shell script version of python-config to $(STAGING_DIR)/host/bin on macOS that would solve your build error, is this correct?

I think it makes sense to install the shell script version for target Python, since it is used to help cross-compile other packages and so no Mac-specific customizations should be used. The shell script version should also be packaged in python3-dev, to be consistent with Linux builds. For host Python (installed to $(STAGING_DIR_HOSTPKG)/bin) I think it should stay as the Python version.

@commodo any thoughts?

@commodo
Copy link
Contributor

commodo commented Feb 15, 2021

Thanks @k-ronny for debugging this - I think if we install the shell script version of python-config to $(STAGING_DIR)/host/bin on macOS that would solve your build error, is this correct?

I think it makes sense to install the shell script version for target Python, since it is used to help cross-compile other packages and so no Mac-specific customizations should be used. The shell script version should also be packaged in python3-dev, to be consistent with Linux builds. For host Python (installed to $(STAGING_DIR_HOSTPKG)/bin) I think it should stay as the Python version.

@commodo any thoughts?

works for me;
i have nothing to add

@httpstorm
Copy link
Contributor

httpstorm commented Feb 15, 2021

Hello @k-ronny
Your fix is very elegant, and we should implement it. I can confirm, that it works correctly on both macOS and Ubuntu Linux. -I$(PYTHON3_INC_DIR) is already in use by a few other packages. There is also a version of this macro for the host: HOST_PYTHON3_INC_DIR. The make variables are defined respectively in python3-package.mk, and python3-host.mk. Can you make a pull-request with your patch or would you like me to do it?

Your observations are right on the point! The reason why the following check fails #if LONG_BIT != 8 * SIZEOF_LONG, is because LONG_BIT = 32 is defined in limits.h for the target architecture of your router, while SIZEOF_LONG = 8 comes from the version of python compiled for the host. The build for the target is using headers from the host package which is incorrect.
./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes should return
-I/home/g/test/ccc/openwrt/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9
but there are multiple issues in the script:

  1. The shebang #!/usr/bin/python3.9 points to a read only file system on macOS, so python cannot be installed there. The only alternative is generating the script with a valid python3.9 path.
  2. Even if the shebang is corrected, the script lacks prefix and prefix_real, which are defined when building on Linux. As a result, the script returns different include paths specific to the instance of python in the shebang.
  3. On Linux, python3.9-config is using a shell script, while on macOS it is a python script.

We should fix this script to include a proper shebang, and a valid prefix. Using a shell script is preferred to be consistent on both platforms.

@httpstorm
Copy link
Contributor

Thanks @k-ronny for debugging this - I think if we install the shell script version of python-config to $(STAGING_DIR)/host/bin on macOS that would solve your build error, is this correct?

I think it makes sense to install the shell script version for target Python, since it is used to help cross-compile other packages and so no Mac-specific customizations should be used. The shell script version should also be packaged in python3-dev, to be consistent with Linux builds. For host Python (installed to $(STAGING_DIR_HOSTPKG)/bin) I think it should stay as the Python version.

@jefferyto @commodo
Sounds good to me. Can you please implement this? It would otherwise take me some time to get familiar with the python build.

@mhei
Copy link
Member

mhei commented Feb 15, 2021

@httpstorm: I'm confused with your earlier comment

-I$(PYTHON3_INC_DIR) is already in use by a few other packages

and where you propose to implement this as a fix? At the moment, I'd NAK this since in my eyes the problem here is that python development environment is mis-configured on macOS. So I'd agree with @jefferyto proposal too, i.e. to fixup the python-config script. But maybe I got you wrong, then sorry...

@httpstorm
Copy link
Contributor

httpstorm commented Feb 15, 2021

@mhei
We have two issues:

  1. python-config is misconfigured on macOS.
  2. libgpiod not building on macOS because of 1.

The patch suggested by @k-ronny in his initial post #14652 (comment) is a valid solution to point 2. -I$(PYTHON3_INC_DIR) is already in use by other projects. Of course we must address point 1 with priority. I will look into the python3.9-config script now.

grep -r "PYTHON3_INC_DIR" .
./feeds/packages/net/uwsgi/Makefile:		CFLAGS="$(TARGET_CPPFLAGS) -I$(PYTHON3_INC_DIR) $(TARGET_CFLAGS)" \
./feeds/packages/lang/python/python3-libsemanage/Makefile:	PYINC="-I $(PYTHON3_INC_DIR)" \
./feeds/packages/lang/python/python3-package.mk:PYTHON3_INC_DIR:=$(PYTHON3_DIR)/include/python$(PYTHON3_VERSION)
./feeds/packages/lang/python/python3-package.mk:	CPPFLAGS="$(TARGET_CPPFLAGS) -I$(PYTHON3_INC_DIR)" \
./feeds/packages/lang/python/python3-libselinux/Makefile:	PYINC="-I $(PYTHON3_INC_DIR)" \
./feeds/packages/lang/python/python3-host.mk:HOST_PYTHON3_INC_DIR:=$(HOST_PYTHON3_DIR)/include/python$(PYTHON3_VERSION)
./feeds/packages/lang/python/python3-host.mk:	CPPFLAGS="$(HOST_CPPFLAGS) -I$(HOST_PYTHON3_INC_DIR)" \
./package/feeds/packages/python3-libsemanage/Makefile:	PYINC="-I $(PYTHON3_INC_DIR)" \
./package/feeds/packages/python3-libselinux/Makefile:	PYINC="-I $(PYTHON3_INC_DIR)" \
./package/feeds/packages/uwsgi/Makefile:		CFLAGS="$(TARGET_CPPFLAGS) -I$(PYTHON3_INC_DIR) $(TARGET_CFLAGS)" \

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 15, 2021

Hi,
based on this discussion I now use this patch
<...>feeds/packages/lang/python/python3/patches/666-macos-config.patch

--- a/Makefile.pre.in	2021-02-15 15:58:26.000000000 +0100
+++ b/Makefile.pre.in	2021-02-15 16:16:10.000000000 +0100
@@ -1586,7 +1586,7 @@
 	@ # On Darwin, always use the python version of the script, the shell
 	@ # version doesn't use the compiler customizations that are provided
 	@ # in python (_osx_support.py).
-	@if test `uname -s` = Darwin; then \
+	@if test `uname -s` = Darwin -a "$(STAGING_DIR_HOSTPKG)" = "$(STAGING_PREFIX)"; then \
 		cp python-config.py python-config; \
 	fi
 

this patch results in

k_ronny@pollux openwrt/ > head -1 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/bin/sh

k_ronny@pollux openwrt/ > head -1 staging_dir/hostpkg/bin/python3.9-config
#!/Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9

and solves my problem without patching libgpiod.

@httpstorm
Copy link
Contributor

@k-ronny
I have tested your last patch #14652 (comment). It compiles correctly on both Linux and macOS. When building python3/host on macOS, the python version of the config script is used, which enables compiler optimisations, since the python script returns configuration for the running python interpreter. This configuration however does not apply to the target and results in build errors. Your last patch resolves the errors by using the shell script for the target build, which returns the correct configuration.

Please name your patch according to these rules, and create a pull-request:
https://openwrt.org/docs/guide-developer/build-system/use-patches-with-buildsystem#naming_patches

jefferyto added a commit to jefferyto/openwrt-packages that referenced this issue Feb 16, 2021
The Python version of python-config is installed when building on macOS,
rather than the shell script version when building on Linux. When run on
macOS, the Python version of python-config will return values with
Mac-specific customizations.

This patches the python-config install recipe so that which version is
installed can be controlled by the package makefile. When building on
macOS, this installs the Python version for host Python and the shell
script version for target Python.

This also updates Host/Compile and Host/Install to use the default host
build recipes, so that the various HOST_* variables are taken into
account automatically.

Fixes openwrt#14652

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
@jefferyto
Copy link
Member

Thanks again @k-ronny for your work on this. Your patch is functionally the same as #14767 but I think #14767 is slightly more readable/intentional.

@k-ronny
Copy link
Contributor Author

k-ronny commented Feb 16, 2021

Hi,
of course, many thanks to all. I am happy that this issue has been solved.

@httpstorm
Copy link
Contributor

Thank you, @k-ronny for the hard work and research, and @jefferyto for the fix! Good luck!

@mhei
Copy link
Member

mhei commented Feb 16, 2021

From me, too: thank you all for your hard work and time on this issue.

1715173329 pushed a commit to immortalwrt/packages that referenced this issue Feb 18, 2021
The Python version of python-config is installed when building on macOS,
rather than the shell script version when building on Linux. When run on
macOS, the Python version of python-config will return values with
Mac-specific customizations.

This patches the python-config install recipe so that which version is
installed can be controlled by the package makefile. When building on
macOS, this installs the Python version for host Python and the shell
script version for target Python.

This also updates Host/Compile and Host/Install to use the default host
build recipes, so that the various HOST_* variables are taken into
account automatically.

Fixes openwrt/packages#14652

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Tianling Shen <cnsztl@project-openwrt.eu.org>
jefferyto added a commit to jefferyto/openwrt-packages that referenced this issue Feb 22, 2021
The Python version of python-config is installed when building on macOS,
rather than the shell script version when building on Linux. When run on
macOS, the Python version of python-config will return values with
Mac-specific customizations.

This patches the python-config install recipe so that which version is
installed can be controlled by the package makefile. When building on
macOS, this installs the Python version for host Python and the shell
script version for target Python.

This also updates Host/Compile and Host/Install to use the default host
build recipes, so that the various HOST_* variables are taken into
account automatically.

Fixes openwrt#14652

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
(cherry picked from commit 1833090)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants