Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pkg-config autoconf macros to detect flags for Modules/Setup #89736

Closed
tiran opened this issue Oct 22, 2021 · 15 comments
Closed

Use pkg-config autoconf macros to detect flags for Modules/Setup #89736

tiran opened this issue Oct 22, 2021 · 15 comments
Assignees
Labels
3.11 only security fixes build The build process and cross-build type-feature A feature request or enhancement

Comments

@tiran
Copy link
Member

tiran commented Oct 22, 2021

BPO 45573
Nosy @Yhg1s, @brettcannon, @tiran, @ned-deily, @erlend-aasland
PRs
  • bpo-45573: Use pkg-config for Modules/Setup #29164
  • bpo-45573: Detect stdlib extension modules in configure #29534
  • bpo-45573: Use Makefile's dependencies in setup.py (GH-29559) #29559
  • bpo-45573: Introduce extension module flags in Makefile (GH-29594) #29594
  • bpo-45573: check for ossaudiodev in configure (GH-29614) #29614
  • bpo-45573: Add Modules/Setup.stdlib with conditional modules (GH-29615) #29615
  • bpo-45573: Move mandatory core modules to Modules/Setup.bootstrap (GH-29616) #29616
  • Superseder
  • bpo-45774: Detect SQLite in configure.ac
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
    closed_at = None
    created_at = <Date 2021-10-22.15:22:43.638>
    labels = ['type-feature', 'build', '3.11']
    title = 'Use pkg-config autoconf macros to detect flags for Modules/Setup'
    updated_at = <Date 2021-11-19.15:41:03.253>
    user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2021-11-19.15:41:03.253>
    actor = 'christian.heimes'
    assignee = 'christian.heimes'
    closed = False
    closed_date = None
    closer = None
    components = ['Build']
    creation = <Date 2021-10-22.15:22:43.638>
    creator = 'christian.heimes'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45573
    keywords = ['patch']
    message_count = 14.0
    messages = ['404781', '404783', '404784', '406084', '406135', '406138', '406139', '406331', '406375', '406520', '406527', '406535', '406586', '406591']
    nosy_count = 5.0
    nosy_names = ['twouters', 'brett.cannon', 'christian.heimes', 'ned.deily', 'erlendaasland']
    pr_nums = ['29164', '29534', '29559', '29594', '29614', '29615', '29616']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = '45774'
    type = 'enhancement'
    url = 'https://bugs.python.org/issue45573'
    versions = ['Python 3.11']

    @tiran
    Copy link
    Member Author

    tiran commented Oct 22, 2021

    pkg-config [1] is a standard tool on Linux and other platforms to detect presence of dependencies as well as to figure out which compiler and linker flags they require. Development packages provide a .pc file, e.g. ncurses provides a ncursesw.pc file.

    $ pkg-config --libs --cflags ncursesw
    -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -lncursesw -ltinfo

    I propose to use a modified version of pkg-config's PKG_HAVE_DEFINE_WITH_MODULES macro in our configure script. On succss the modified macro defines:

    • HAVE_FOO=1
    • FOO_CFLAGS="some compile flags"
    • FOO_LIBS="some library flags for linker"

    On error, it sets nothing and does not cause configure to stop with an error.

    The macro also allows users to override flags by setting FOO_CFLAGS and FOO_LIBS env vars. HAVE_FOO is added to pyconfig.h. The FOO_CFLAGS/LIBS are added to Makefile, from where it can be consumed by Modules/Setup.

    Eventually Python could use the flags in setup.py, too. For now I would like to start with Modules/Setup. It is only used by some power users and has less risk of breaking the setup of beginners.

    [1] https://www.freedesktop.org/wiki/Software/pkg-config/

    @tiran tiran added the 3.11 only security fixes label Oct 22, 2021
    @tiran tiran self-assigned this Oct 22, 2021
    @tiran tiran added build The build process and cross-build type-feature A feature request or enhancement 3.11 only security fixes labels Oct 22, 2021
    @tiran tiran self-assigned this Oct 22, 2021
    @tiran tiran added build The build process and cross-build type-feature A feature request or enhancement labels Oct 22, 2021
    @tiran
    Copy link
    Member Author

    tiran commented Oct 22, 2021

    I added some whitespace to Modules/Setup. All lines starting with #([a-z_]) compile cleanly on Fedora 34 with all dependencies available. dbm, tkinter, and sqlite are missing or untested.

    $ sed -E -i 's/^#([a-z_])/\1/g' Modules/Setup
    $ make
    ...
    The following modules found by detect_modules() in setup.py, have been
    built by the Makefile instead, as configured by the Setup files:
    _abc               _asyncio           _bisect         
    _blake2            _bz2               _codecs_cn      
    _codecs_hk         _codecs_iso2022    _codecs_jp      
    _codecs_kr         _codecs_tw         _contextvars    
    _crypt             _csv               _curses         
    _curses_panel      _datetime          _elementtree    
    _hashlib           _heapq             _json           
    _lsprof            _lzma              _md5            
    _multibytecodec    _opcode            _pickle         
    _posixsubprocess   _queue             _random         
    _sha1              _sha256            _sha3           
    _sha512            _socket            _ssl            
    _statistics        _struct            _testbuffer     
    _testimportmultiple   _testinternalcapi   _testmultiphase 
    _typing            _xxsubinterpreters   _xxtestfuzz     
    _zoneinfo          array              audioop         
    binascii           cmath              fcntl           
    grp                math               mmap            
    nis                ossaudiodev        pwd             
    pyexpat            readline           resource        
    select             spwd               syslog          
    termios            time               unicodedata     
    xxlimited          xxlimited_35       zlib   
    
    $  ./python -c "import sys; print(sys.builtin_module_names)"
    ('_abc', '_ast', '_asyncio', '_bisect', '_blake2', '_bz2', '_codecs', '_codecs_cn', '_codecs_hk', '_codecs_iso2022', '_codecs_jp', '_codecs_kr', '_codecs_tw', '_collections', '_contextvars', '_crypt', '_csv', '_curses', '_curses_panel', '_datetime', '_elementtree', '_functools', '_hashlib', '_heapq', '_imp', '_io', '_json', '_locale', '_lsprof', '_lzma', '_md5', '_multibytecodec', '_opcode', '_operator', '_pickle', '_posixsubprocess', '_queue', '_random', '_sha1', '_sha256', '_sha3', '_sha512', '_signal', '_socket', '_sre', '_ssl', '_stat', '_statistics', '_string', '_struct', '_symtable', '_testbuffer', '_testimportmultiple', '_testinternalcapi', '_testmultiphase', '_thread', '_tokenize', '_tracemalloc', '_typing', '_warnings', '_weakref', '_xxsubinterpreters', '_xxtestfuzz', '_zoneinfo', 'array', 'atexit', 'audioop', 'binascii', 'builtins', 'cmath', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'mmap', 'nis', 'ossaudiodev', 'posix', 'pwd', 'pyexpat', 'readline', 'resource', 'select', 'spwd', 'sys', 'syslog', 'termios', 'time', 'unicodedata', 'xx', 'xxlimited', 'xxlimited_35', 'xxsubtype', 'zlib')
    $ ldd python
            linux-vdso.so.1 (0x00007fffa1f40000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6032fe2000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007f6032fdb000)
            libutil.so.1 => /lib64/libutil.so.1 (0x00007f6032fd6000)
            libm.so.6 => /lib64/libm.so.6 (0x00007f6032e92000)
            libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f6032e58000)
            libnsl.so.2 => /lib64/libnsl.so.2 (0x00007f6032e3c000)
            libtirpc.so.3 => /lib64/libtirpc.so.3 (0x00007f6032e0a000)
            libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f6032df7000)
            liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f6032dcb000)
            libz.so.1 => /lib64/libz.so.1 (0x00007f6032db1000)
            libreadline.so.8 => /lib64/libreadline.so.8 (0x00007f6032d59000)
            libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f6032cbc000)
            libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f60329cc000)
            libncursesw.so.6 => /lib64/libncursesw.so.6 (0x00007f603298d000)
            libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f603295e000)
            libpanel.so.6 => /lib64/libpanel.so.6 (0x00007f6032958000)
            libc.so.6 => /lib64/libc.so.6 (0x00007f6032789000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f6033029000)
            libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6032732000)
            libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6032652000)
            libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f603263a000)
            libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6032633000)
            libncurses.so.6 => /lib64/libncurses.so.6 (0x00007f6032606000)
            libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f60325f5000)
            libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f60325ec000)
            libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f60325d2000)
            libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f60325a6000)
            libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f603250f000)

    @tiran
    Copy link
    Member Author

    tiran commented Oct 22, 2021

    $ find build/lib.linux-x86_64-3.11/ -name '*.so' | sort
    build/lib.linux-x86_64-3.11/_ctypes.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_ctypes_test.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_dbm.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_decimal.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_gdbm.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_multiprocessing.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_posixshmem.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_sqlite3.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_testcapi.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_tkinter.cpython-311-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.11/_uuid.cpython-311-x86_64-linux-gnu.so

    @tiran
    Copy link
    Member Author

    tiran commented Nov 10, 2021

    We can detect majority of our dependencies with pkg-config. The use of pkg-config has some benefits:

    • Distro's provide the .pc files in their -dev / -devel packages. The presence of a .pc file indicates that all development dependencies are available.

    • pkg-config does the right thing for non-standard include and libraries directories as well as multiarch builds. In case a library or header is not on the default search path, pkg-config returns necessary -I and -L flags.

    • At least the pkgconf implementation of pkg-config standard search /usr/local and ~/.local/ directories for .pc files. Cases like bpo-45774: Autoconfiscate SQLite detection (GH-29507) #29507 are handled correctly. On FreeBSD "pkgconf sqlite3 --cflags --libs" returns "-I/usr/local/include -L/usr/local/lib -lsqlite3".

    • pkg-config understands dependencies. For example "pkg-config --libs tk" returns linker flags for TK *and* TCL.

    • pkg-config can check for module version, e.g. "pkg-config sqlite3 --atleast-version=3.7.15"

    pkg-config modules:

    readline, libedit
    ncursesw, ncurses, panel, tinfo
    sqlite3
    zlib
    bzip2
    liblzma
    expat
    uuid (Linux's util-linux uuid)
    libffi
    libnsl, libtirpc
    libcrypt
    tcl, tk
    openssl, libssl, libcrypto

    modules / libraries without pkg-config modules:

    decimal: libmpdec
    gdbm: gdbm
    dbm: gdbm_compat, ndbm, libdb (bdb)

    To simplify use of flags in Modules/Setup, I propose to add two make variables for each module that needs cflags and ldflags:

    f"MODULE_{ext.name.upper()}CFLAGS"
    f"MODULE
    {ext.name.upper()}_LDFLAGS"

    e.g. for the _ssl module:

      MODULE__SSL_CFLAGS=
      MODULE__SSL_LDFLAGS=-lssl -lcrypto

    Then use the flags from Makefile in setup.py:

        def update_extension_flags(self, ext):
            name = ext.name.upper()
            cflags = sysconfig.get_config_var(f"MODULE_{name}_CFLAGS")
            if cflags:
                ext.extra_compile_args.extend(shlex.split(cflags))
            ldflags = sysconfig.get_config_var(f"MODULE_{name}_LDFLAGS")
            if ldflags:
                ext.extra_link_args.extend(shlex.split(ldflags))
            return ext

    Finally update Modules/makesetup to use the new variables, too.

    @brettcannon
    Copy link
    Member

    SGTM!

    @ned-deily
    Copy link
    Member

    SGTM,2

    This all sounds great. I think a goal here should be to remove all header and lib file searching from setup.py as that has always been a bug magnet. Perhaps one workaround for those libs that don't (yet) provide .pc files would be for us to supply reasonable default .pc for them, if possible, so that most builders wouldn't have to set the MODULE_xxx_*FLAGS variables while still removing the header/lib file searching from setup.py? Also it would be great to document how using pkg-config works in generic cross-compiling cases; we should strive to make at least the most common cases just work with minimal tweaking of configure arguments.

    @tiran
    Copy link
    Member Author

    tiran commented Nov 10, 2021

    gdbmmodule and dbmmodule need special treatment anyway. macOS has dbm-API build into libc. Linux has either libgdbm_compat, libndbm, or libdb. The --with-dbmliborder makes it even more interesting. Users can override in which order they want to probe for gdbm, ndbm, and libdb. We have to keep the "manual" library and header checks.

    Cross-compilation with pkg-config uses a trivial wrapper script, https://autotools.io/pkgconfig/cross-compiling.html . The build system has to create and provide a script with correct sysroot setting.

    Is traditional cross-compiling work still useful these days anyway? Emulation has made big leaps in the last decade. Emulated cross compiling with qemu has become widespread. It also has the big advantage that you can run the test suite on the emulated hardware and verify that your binaries work. AFAIK Fedora's build system uses qemu for a bunch of hardware targets.

    @tiran
    Copy link
    Member Author

    tiran commented Nov 14, 2021

    New changeset c399786 by Christian Heimes in branch 'main':
    bpo-45573: Use Makefile's dependencies in setup.py (GH-29559)
    c399786

    @brettcannon
    Copy link
    Member

    Is traditional cross-compiling work still useful these days anyway?

    Yes for things like WebAssembly where there is no equivalent of an emulated CPU under QEMU.

    @tiran
    Copy link
    Member Author

    tiran commented Nov 18, 2021

    New changeset 25ecc04 by Christian Heimes in branch 'main':
    bpo-45573: Introduce extension module flags in Makefile (GH-29594)
    25ecc04

    @tiran
    Copy link
    Member Author

    tiran commented Nov 18, 2021

    New changeset 5275e59 by Christian Heimes in branch 'main':
    bpo-45573: check for ossaudiodev in configure (GH-29614)
    5275e59

    @tiran
    Copy link
    Member Author

    tiran commented Nov 18, 2021

    New changeset e4bb22f by Christian Heimes in branch 'main':
    bpo-45573: Add Modules/Setup.stdlib with conditional modules (GH-29615)
    e4bb22f

    @tiran
    Copy link
    Member Author

    tiran commented Nov 19, 2021

    Erlend implemented pkg-config for sqlite3 in bpo-45774. Thanks to his tireless effort we now have a blue print how to port other checks to optional pkg-config lookup.

    @tiran
    Copy link
    Member Author

    tiran commented Nov 19, 2021

    New changeset 7e44dc0 by Christian Heimes in branch 'main':
    bpo-45573: Move mandatory core modules to Modules/Setup.bootstrap (GH-29616)
    7e44dc0

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @erlend-aasland
    Copy link
    Contributor

    AFAICS, this is implemented. If you disagree, please re-open or create a new issue describing the problem.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 only security fixes build The build process and cross-build type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants