Python 3.8 and python3-config --libs

[bentiss]

Quoting https://bugzilla.redhat.com/show_bug.cgi?id=1718290:

The build no longer links to libpython, resulting in undefined references to Python API.
Code that embeds Python (rather than building an extension module) needs to pass --embed to any python3-config --libs invocation to build with Python 3.8.
More information:
https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

In libratbag, we are using swig to generate a Python library that we link with a dependencies of dependency('python3'). I wasn't able to find out how the pkg-config was invoced, so I can not really submit a patch.

[lazka]

The python module handles the first part this since #4187

edit: oh, it's the other way around in that pkg-config no longer makes it link against libpython.. never mind then :/ I guess we need to an an option to the python module somehow for this new case.

[lazka]

There is a new "python-embed.pc" in 3.8, have you tried that?

[whot]

Yep, changing to dependency('python3-embed') fixes the issue, thanks!

[bentiss]

Thanks everybody. I guess we can close this issue.

[nirbheek]

I'm not sure this is the correct fix; the fix should work on all python versions. If it is, we should document it somewhere?

[jpakkane]

The bigger issue is how should Python dependencies looked up via the Python module work? That one probably needs a new kwarg because you can not specify which dependency name to use.

[ignatenkobrain]

I've also got one bug https://bugzilla.redhat.com/show_bug.cgi?id=1718283.

I think proper way would be to use some kwarg which would choose between -embed and normal python.

[jpakkane]

Seems reasonable.

[sdroege]

Checking for python3-embed.pc is not sufficient as it might not even exist (it does not on Debian until the default Python version becomes 3.8). The correct way would be to call python3.8-config --ldflags --embed.

Also related bug: https://gitlab.freedesktop.org/gstreamer/gst-python/issues/28

[MathieuDuponchelle]

The bigger issue is how should Python dependencies looked up via the Python module work? That one probably needs a new kwarg because you can not specify which dependency name to use.

yes, precisely, I reckon an embed argument to the dependency method would be the desirable API

[eli-schwartz]

Checking for python3-embed.pc is not sufficient as it might not even exist (it does not on Debian until the default Python version becomes 3.8).

If python3-embed.pc does not exist, and python3.pc is a symlink to python-3.8.pc, then your distribution is flawed and MUST be fixed.

If python3-embed.pc does not exist, and python3.pc is a symlink to python-3.7.pc, then your meson.build project will build using the default version of python on your distribution, which is python 3.7, so this issue is not topical to your use case.

The correct way would be to call python3.8-config --ldflags --embed.

Incorrect, projects which expect to build against some version of python 3.x must not hardcode python3.8-config but rather use python3-config, which shall be a symlink to the default version of python3, which on your distro is python 3.7.

If it were appropriate to look up python3.8-config, then it would likewise be appropriate to declare a dependency('python3.8-embed'), since python3.8-embed.pc MUST exist (even if python3-embed.pc does not), for any installation of python 3.8 that includes development headers, regardless of whether it is the default, unversioned 3.x namespace.

[sdroege]

@eli-schwartz You're missing the point.

There could be multiple Python versions installed on a system, which is the case on Debian for example (right now: 2.7, 3.7 and 3.8). One Python 3 version is going to be the default (and have the corresponding python3.pc etc), but you can select different Python versions during the build and that also always worked with both autotools and meson until Python 3.8 appeared.

See https://gitlab.freedesktop.org/gstreamer/gst-python/ for an example meson build system that allows to select the Python version. Using python3.pc unconditionally is not going to work there, you'd have to use either python-3.X.pc (and the -embed version if X > 7) or the corresponding python3.X-config (with --embed if needed for > 3.7).

[eli-schwartz]

If you're not using pkg-config in the first place, then why are you suggesting that "checking python3-embed.pc is insufficient"?

Moreover, why are you suggesting that the correct way is to call python3.8-config, which is a false equivalence, since if you're going to hardcode or detect the version in the python*-config tool, you can hardcode or detect the version in the python*-embed.pc file too.

Using meson's python module via find_installation().dependency() already checks the major.minor version of the detected python binary, then looks for 'python-{}.pc'.format(pkg_version) explicitly, never python3.pc, so that's fine.

The recommendation in previous comments "have you tried python3-embed.pc" was strictly a recommendation for users who are currently using dependency('python3'), rather than the python module, so I assumed that was happening. In such cases, it makes perfect sense to first try dependency('python3-embed').

The same logic would always apply to autotools, which doesn't automatically resolve PKG_CHECK_MODULES(PYTHON, [python3], , ) to the major.minor version, and you get to do all the magic yourself.

[eli-schwartz]

A bit hacky, but you could do arbitrary version selection yourself (no need for the python module) like this:

python = find_program('python3')
pythonver = run_command(python, '-c', 'import sys; print("%s.%s" % sys.version_info[:2], end="")').stdout()

# or via language_version()
python = import('python')
pythonver = python.find_installation('python3').language_version()

python_dep = dependency('python-@0@-embed'.format(pythonver), version: '>=3', required: false)
if not python_dep.found()
    python_dep = dependency('python-@0@'.format(pythonver))
endif

Alternatively (this prints up to 3 missing unrequired dependencies first):

pythonver = get_option('pythonver')
python = dependency('', required: false)
foreach base : ['python@0@-embed', 'python@0@', 'python-@0@-embed', 'python-@0@']
    if not python.found()
        python = dependency(base.format(pythonver), version: '>=3', required: false)
    endif
endforeach

if not python.found()
    error('no suitable version of python@0@ found'.format(pythonver))
endif

[sdroege]

If you're not using pkg-config in the first place, then why are you suggesting that "checking python3-embed.pc is insufficient"?

Checking for a generic python3*.pc is insufficient. My comment was about having this as the "official" workaround for this problem. It won't allow selecting a specific Python version.

python-3.8-embed.pc if Python 3.8 is selected would work too for example, or using python-3.8-config.

Your workaround to parse the Python version and then look for the corresponding .pc file would work though, thanks for that!

[mensinda]

This should be fixed with #6341. It explicitly checks for python-3.x-embed.pc, so there should not be any issues as far as I can tell.