[BUG] No module named '_sysconfigdata_d_linux_x86_64-linux-gnu'

[henryiii]

setuptools version

setuptools==60.0.0

Python version

Python 3.9

OS

Ubuntu 20.04

Additional environment information

Can reproduce in Docker or with deadsnakes action.

Description

Today, 3.9 deadsnakes debug builds on GitHub Actions for pybind11 started failing with ModuleNotFoundError: No module named '_sysconfigdata_d_linux_x86_64-linux-gnu'. Setting SETUPTOOLS_USE_DISTUTILS=stdlib` fixes it.

Logs: https://github.com/pybind/pybind11/runs/4583094186?check_suite_focus=true

Original report: deadsnakes/issues#188

Expected behavior

It should be possible to use setuptools and sysconfig on Python 3.9 debug builds! :)

How to Reproduce

deadsnakes/action@v2.1.1 on ubuntu-latest, GHA, followed by the final line below. Or it can be reproduced in Docker. Here is a MWE:

$ docker run --rm -it ubuntu:20.04
# apt update && apt install software-properties-common -y
# add-apt-repository ppa:deadsnakes/ppa && apt update && apt install -y python3.9-dev python3.9-dbg python3.9-venv python3.9-distutils
# python3.9-dbg -m venv .venv
# ./.venv/bin/python -m pip install -U pip setuptools wheel
# ./.venv/bin/python -c "import setuptools; from distutils import sysconfig as s; s.get_config_var('LDVERSION')"

Output

Traceback (most recent call last):
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 488, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_d_linux_x86_64-linux-gnu'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 532, in get_config_vars
    func()
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 491, in _init_posix
    _temp = __import__(
ModuleNotFoundError: No module named '_sysconfigdata'

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

Looks like that code has been around a while, but it's not present in cpython's copy of distutils. Maybe that's an indication that there are some changes to the stdlib distutils that didn't make it into the standalone repo.

[jaraco]

Hmm. The relevant change is python/cpython#24549 python/cpython#23142. Someone's going to need to figure out how to apply those changes to standalone distutils.

[henryiii]

I'm guessing the problem with a direct port is that it would need to be Python version dependent?

[henryiii]

The latest version here works with Python 3.6+.

No, so was it just never applied?

[henryiii]

Hmm, this is a PR from April, how could it be breaking Python 3.9?

[jaraco]

Thanks Henry for the help. And thanks for creating the PR against distutils! Correct, it was never applied.

Hmm, this is a PR from April, how could it be breaking Python 3.9?

I believe the change was never backported and was only applied to Python 3.10, so may not have received any thorough testing on various Python versions.

[jaraco]

Corrected above, I somehow got confused when bouncing between PRs. The PR I believe to be needed by distutils is python/cpython#23142.

[henryiii]

That PR removes most of distutils.sysconfig and replaces it with sysconfig. Would that be safe for older Pythons? (3.6+). I checked a few, and some of them (like _findvar1_rx expand_makefile_vars) are clearly missing.

[FFY00]

Looks like that code has been around a while, but it's not present in cpython's copy of distutils. Maybe that's an indication that there are some changes to the stdlib distutils that didn't make it into the standalone repo.

I don't think so. I don't have much time to dive into this, but my gut is telling me that the failure is because of the unusual loading that distutils makes of sysconfig. The failure is related to the loading of config variables, the _sysconfig* module is an automatically generated module that includes the Makefile variables, for sysconfig to then load, it is not available on Windows. This is a very private bit of CPython and distutils should not be poking anywhere near it. distutils must be misusing private sysconfig APIs, or there is a bug sysconfig due to the way distutils loads stuff. I am guessing that the answer might actually be something in between.

[jaraco]

It may be the case that the local distutils simply needs to fall back to Python 3.9 behaviors on Python 3.9 and earlier.

[jaraco]

I'm able to replicate the issue with this xonsh command (adapted from the repro above):

docker run -it @$(echo '''FROM ubuntu:20.04
RUN apt update && apt install software-properties-common -y
RUN add-apt-repository ppa:deadsnakes/ppa && apt update && apt install -y python3.9-dev python3.9-dbg python3.9-venv python3.9-distutils
RUN python3.9-dbg -m venv .venv
RUN ./.venv/bin/python -m pip install -U pip setuptools wheel
CMD ./.venv/bin/python -c "import setuptools; from distutils import sysconfig as s; s.get_config_var('LDVERSION')"
''' | docker build -q -)

[jaraco]

The import setuptools; can be omitted and the issue still occurs.

[jaraco]

I observe these sysconfig modules exist in that environment:

root@eda1978d024a:/# ls /usr/lib/python3.9/_sysconfigdata_*
/usr/lib/python3.9/_sysconfigdata__linux_x86_64-linux-gnu.py
/usr/lib/python3.9/_sysconfigdata_d_x86_64-linux-gnu.py
/usr/lib/python3.9/_sysconfigdata__x86_64-linux-gnu.py

[jaraco]

It looks like something has created a symlink for the non-debug version but not for the debug version:

root@eda1978d024a:/# ls -l /usr/lib/python3.9/_sysconfigdata_*
lrwxrwxrwx 1 root root    35 Nov 16 03:08 /usr/lib/python3.9/_sysconfigdata__linux_x86_64-linux-gnu.py -> _sysconfigdata__x86_64-linux-gnu.py
-rw-r--r-- 1 root root 25879 Nov 16 03:08 /usr/lib/python3.9/_sysconfigdata__x86_64-linux-gnu.py
-rw-r--r-- 1 root root 26563 Nov 16 03:08 /usr/lib/python3.9/_sysconfigdata_d_x86_64-linux-gnu.py

I wonder what creates that symlink.

[jaraco]

Creating a debug symlink seems to work around the issue.

root@eda1978d024a:/# ln -s _sysconfigdata_d_x86_64-linux-gnu.py /usr/lib/python3.9/_sysconfigdata_d_linux_x86_64-linux-gnu.py
root@eda1978d024a:/# ./.venv/bin/python -c "from distutils import sysconfig as s; s.get_config_var('LDVERSION')" && echo worked
worked

[jaraco]

But the question is - why doesn't the issue manifest on stdlib distutils?

Aha. It's because Debian patches distutils.sysconfig:

root@eda1978d024a:/# grep -A 10 'def _init_posix' /usr/lib/python3.9/distutils/sysconfig.py
def _init_posix():
    """Initialize the module as appropriate for POSIX systems."""
    # _sysconfigdata is generated at build time, see the sysconfig module
    name = os.environ.get('_PYTHON_SYSCONFIGDATA_NAME',
        '_sysconfigdata_{abi}_{multiarch}'.format(
        abi=sys.abiflags,
        multiarch=getattr(sys.implementation, '_multiarch', ''),
    ))
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    build_time_vars = _temp.build_time_vars
    global _config_vars

(the _sysconfigdata_{abi}_{multiarch} template doesn't have {platform} as found in distutils)

pypa/distutils#68 may be the answer.

[jaraco]

I looked into applying the patch suggested here, but it doesn't do the trick as it doesn't include any support for patching _sysconfig_name_tmpl. But since pypa/distutils#69, distutils does provide a hook for that, which I was able to apply:

root@eda1978d024a:/# cat > .venv/lib/python3.9/site-packages/_distutils_system_mod.py
import distutils.sysconfig
distutils.sysconfig._sysconfig_name_tmpl = '_sysconfigdata_{abi}_{multiarch}'
root@eda1978d024a:/# ./.venv/bin/python -c "from distutils import sysconfig as s; s.get_config_var('LDVERSION')" && echo worked
worked

I'm unsure what distutils should do here. My instinct is that whatever creates that symlink for non-debug builds should do the same for debug builds. That will fix the issue without any extra patching of distutils.

I'm not sure what distutils should do here, if anything.

[jaraco]

I note that the workaround also works if applied at the system lib level:

root@eda1978d024a:/# rm .venv/lib/python3.9/site-packages/_distutils_system_mod.py 
root@eda1978d024a:/# cat > /usr/lib/python3.9/_distutils_system_mod.py
import distutils.sysconfig
distutils.sysconfig._sysconfig_name_tmpl = '_sysconfigdata_{abi}_{multiarch}'
root@eda1978d024a:/# ./.venv/bin/python -c "from distutils import sysconfig as s; s.get_config_var('LDVERSION')" && echo worked
worked

[henryiii]

We could report this missing symlink to Debian or deadsnakes, then? @asottile, do you know where this should be raised?

[lazka]

pypa/distutils#111 might help here, by delegating to sysconfig

[henryiii]

Ah, that sounds promising.

[ryancheley]

I had this same issue and this StackOverflow answer was able to resolve it for me

[jaraco]

This issue no longer exists (available since Setuptools 61):

 draft $ cat Dockerfile
FROM ubuntu:20.04
RUN apt update && apt install software-properties-common -y
RUN add-apt-repository ppa:deadsnakes/ppa && apt update && apt install -y python3.9-dev python3.9-dbg python3.9-venv python3.9-distutils
RUN python3.9-dbg -m venv .venv
RUN ./.venv/bin/python -m pip install -U pip setuptools wheel
CMD ./.venv/bin/python -c "import setuptools; from distutils import sysconfig as s; s.get_config_var('LDVERSION')"
 draft $ docker run -it @$(docker build -q .)
 draft $

[stefan-zh]

For me this StackOverflow comment solved the issue. Then I was able to install the latest setuptools running python -m pip install -U pip setuptools wheel