[Python 3.12] ModuleNotFoundError: No module named 'setuptools' as setuptools not in the venv, only installed by apt!

[LebedevRI]

Failing CI job: https://github.com/darktable-org/rawspeed/actions/runs/9698273466/job/26764443826
python3-setuptools package is installed: https://github.com/darktable-org/rawspeed/actions/runs/9698273466/job/26764443826#step:5:765

make venv does not seem to remove it:
https://github.com/darktable-org/rawspeed/actions/runs/9698273466/job/26764443826#step:10:17

Yet make package ultimately fails:
https://github.com/darktable-org/rawspeed/actions/runs/9698273466/job/26764443826#step:10:250

+ BUILD_LOGGER_64_BIT_ONLY=YES BUILD_UI_DIST=NO make package
mkdir -p /__w/rawspeed/rawspeed/codechecker/build && \
mkdir -p /__w/rawspeed/rawspeed/codechecker/build/CodeChecker/bin && \
mkdir -p /__w/rawspeed/rawspeed/codechecker/build/CodeChecker/lib/python3
if [ -d "/__w/rawspeed/rawspeed/codechecker/.git" ]; then git config --local commit.template .gitmessage; fi
cp -p scripts/gerrit_changed_files_to_skipfile.py /__w/rawspeed/rawspeed/codechecker/build/CodeChecker/bin
BUILD_DIR=/__w/rawspeed/rawspeed/codechecker/build BUILD_LOGGER_64_BIT_ONLY=YES make -C /__w/rawspeed/rawspeed/codechecker/analyzer package_analyzer
make[1]: Entering directory '/__w/rawspeed/rawspeed/codechecker/analyzer'
mkdir -p /__w/rawspeed/rawspeed/codechecker/build && \
mkdir -p /__w/rawspeed/rawspeed/codechecker/build/CodeChecker/bin && \
mkdir -p /__w/rawspeed/rawspeed/codechecker/build/CodeChecker/lib/python3
make -C /__w/rawspeed/rawspeed/codechecker/analyzer/../tools/tu_collector build
make[2]: Entering directory '/__w/rawspeed/rawspeed/codechecker/tools/tu_collector'
python3 setup.py build --build-purelib build/tu_collector
Traceback (most recent call last):
  File "/__w/rawspeed/rawspeed/codechecker/tools/tu_collector/setup.py", line 4, in <module>
    import setuptools
ModuleNotFoundError: No module named 'setuptools'
make[2]: *** [Makefile:44: build] Error 1
make[2]: Leaving directory '/__w/rawspeed/rawspeed/codechecker/tools/tu_collector'
make[1]: *** [Makefile:54: build_tu_collector] Error 2
make[1]: Leaving directory '/__w/rawspeed/rawspeed/codechecker/analyzer'
make: *** [Makefile:45: package] Error 2
Error: Process completed with exit code 2.

Google suggests that setuptools should not be in requirements.txt and certainly should not be in a lock file.
I'm a little bit lost as to what is going on.

X-Ref: darktable-org/rawspeed#738 (comment)

[whisperity]

First off, which operating system and environment is the problematic? It would be crucial to know what falls apart.

[LebedevRI]

First off, which operating system and environment is the problematic? It would be crucial to know what falls apart.

Sorry, i thought linking the actual log would be sufficient to answer any such question.
As it can be seen in the log i linked, that is amd64 linux, debian sid.

[whisperity]

Okay, I was able to reproduce it. Unfortunately, sid isn't a stable distribution, so all I can say is that I used the docker image debian:sid@5d3ebeda1613. It is likely that Python's behaviour with regards to falling back to system-wide packages when also being in a virtualenv has changed in the newer versions; and it looks like even though setuptools is installed globally (via apt), it will not be visible to a Python interpreter running in a virtualenv:

root@7684ab69eef9:/CodeChecker# which python3
/usr/bin/python3
root@7684ab69eef9:/CodeChecker# python3 -V
Python 3.12.4
root@7684ab69eef9:/CodeChecker# python3 -c "import setuptools"; echo $?
0

root@7684ab69eef9:/CodeChecker# source venv/bin/activate
(CodeChecker venv) root@7684ab69eef9:/CodeChecker# which python3
/CodeChecker/venv/bin/python3
(CodeChecker venv) root@7684ab69eef9:/CodeChecker# python3 -V
Python 3.12.4
(CodeChecker venv) root@7684ab69eef9:/CodeChecker# python3 -c "import setuptools"; echo $?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
1

As a workaround, doing pip install setuptools when in the venv will result in no downloads, but putting the setuptools library files into the virtualenv, and it will work:

(CodeChecker venv) root@7684ab69eef9:/CodeChecker# pip install setuptools
Collecting setuptools
  Using cached setuptools-70.2.0-py3-none-any.whl.metadata (5.8 kB)
Using cached setuptools-70.2.0-py3-none-any.whl (930 kB)
Installing collected packages: setuptools
Successfully installed setuptools-70.2.0

(CodeChecker venv) root@7684ab69eef9:/CodeChecker# python3 -c "import setuptools"; echo $?
0

It seems like once we reach the point that we can be sure to support such new Python versions, we will have to update the documentation.

---

N.B. this is a global Python-level issue or behavioural quirk, as it can be reproduced with packages other than CodeChecker's structure:

root@7684ab69eef9:/# python3 -c "import tabulate"; echo $?
[...]
ModuleNotFoundError: No module named 'tabulate'
1

root@7684ab69eef9:/# apt -yqq install python3-tabulate
[...]

root@7684ab69eef9:/# python3 -c "import tabulate"; echo $?
0

root@7684ab69eef9:/# python3 -m venv my_test_venv
root@7684ab69eef9:/# source my_test_venv/bin/activate
(my_test_venv) root@7684ab69eef9:/# python3 -c "import tabulate"; echo $?
[...]
ModuleNotFoundError: No module named 'tabulate'
1

[LebedevRI]

@whisperity thank you for taking a look!

[whisperity]

Some more information and potential "venv-level" or "global" workarounds available here:

• https://stackoverflow.com/a/65371265
• https://docs.python.org/3.8/library/venv.html

Oddly enough, --system-site-packages has been a thing for a considerable time, even back in version 3.3...