Bootstrapping on ubuntu:18.04, python3-distutils not installed by default

[haampie]

Steps to reproduce the issue

$ docker run -it -w /root ubuntu:18.04
# apt-get update -qq && apt-get install -yqq python3 libpython3-dev git gcc g++ gfortran make file curl wget bison
# git clone --depth 1 -b develop https://github.com/spack/spack.git
# export PATH="/root/spack/bin:$PATH"
# wget -qO- "https://cmake.org/files/v3.20/cmake-3.20.0-linux-x86_64.tar.gz" | tar --strip-components=1 -xz -C /usr/local
# spack external find --not-buildable cmake bison
# spack --debug --stacktrace solve zlib

lib/spack/spack/installer.py:281 ==> [2021-03-29-13:36:58.006190] No binary for clingo-bootstrap-spack-suq7ubei4oyla73cmpnsnph5akp4omif found: installing from source
lib/spack/spack/package.py:1522 ==> [2021-03-29-13:36:58.028352] Warning: Missing a source id for python@3.6
lib/spack/spack/compilers/__init__.py:741 ==> [2021-03-29-13:36:58.324252] [TOOLCHAIN] MATCH Gcc
lib/spack/spack/util/executable.py:182 ==> [2021-03-29-13:36:58.338027] '/usr/bin/python3.6' '-c' 'from distutils.sysconfig import get_config_var; print(get_config_var('"'"'prefix'"'"'))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils.sysconfig'
lib/spack/spack/util/executable.py:182 ==> [2021-03-29-13:36:58.365391] '/usr/bin/python3.6' '-c' 'from distutils.sysconfig import get_config_var; print(get_config_var('"'"'CC'"'"'))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils.sysconfig'
lib/spack/spack/stage.py:313 ==> [2021-03-29-13:36:58.400220] Creating stage lock spack-stage-clingo-bootstrap-spack-suq7ubei4oyla73cmpnsnph5akp4omif
lib/spack/spack/error.py:55 ==> [2021-03-29-13:36:58.409111] Error: ProcessError: Command exited with status 1:
    '/usr/bin/python3.6' '-c' 'from distutils.sysconfig import get_config_var; print(get_config_var('"'"'CC'"'"'))'
Traceback (most recent call last):
  File "/root/spack/lib/spack/spack/build_environment.py", line 855, in _setup_pkg_and_run
    context=context)
  File "/root/spack/lib/spack/spack/build_environment.py", line 744, in setup_package
    modifications_from_dependencies(pkg.spec, context=context)
  File "/root/spack/lib/spack/spack/build_environment.py", line 831, in modifications_from_dependencies
    getattr(dpkg, method)(env, spec)
  File "/root/spack/var/spack/repos/builtin/packages/python/package.py", line 884, in setup_dependent_build_environment
    config_link = self.get_config_var(link_var)
  File "/root/spack/var/spack/repos/builtin/packages/python/package.py", line 680, in get_config_var
  File "/root/spack/lib/spack/spack/util/executable.py", line 218, in __call__
    proc.returncode, long_msg)
spack.util.executable.ProcessError: Command exited with status 1:
    '/usr/bin/python3.6' '-c' 'from distutils.sysconfig import get_config_var; print(get_config_var('"'"'CC'"'"'))'
lib/spack/spack/installer.py:1301 ==> [2021-03-29-13:36:58.409507] Flagging clingo-bootstrap-spack-suq7ubei4oyla73cmpnsnph5akp4omif as failed: ProcessError: Command exited with status 1:
    '/usr/bin/python3.6' '-c' 'from distutils.sysconfig import get_config_var; print(get_config_var('"'"'CC'"'"'))'
lib/spack/spack/stage.py:313 ==> [2021-03-29-13:36:58.412691] Creating stage lock spack-stage-clingo-bootstrap-spack-suq7ubei4oyla73cmpnsnph5akp4omif
lib/spack/spack/main.py:771 ==> [2021-03-29-13:36:58.412935] ChildError: ProcessError: Command exited with status 1:
    '/usr/bin/python3.6' '-c' 'from distutils.sysconfig import get_config_var; print(get_config_var('"'"'CC'"'"'))'

Error Message

ModuleNotFoundError: No module named 'distutils.sysconfig'

Information on your system

• Spack: 0.16.1
• Python: 3.6.9
• Platform: linux-ubuntu18.04-zen2
• Concretizer: original

I have to install python3-distutils with apt to make this work:

apt-get install python3-distutils

[haampie]

Ping @alalazo / @adamjstewart

[alalazo]

This will be solved by #22720 If we bootstrap from a binary we don't need distutils:

[KineticTheory]

I ran into a similar issue on a fresh Ubuntu 20.04 image because I forgot to install libpython3-dev. Without this package, Clingo-bootstrap failed to configure with the error "python was not found."

[healther]

I'm running into the same error, would it be possible to force spack to build its own python interpreter as part of the bootstrap process?

[fangohr]

This issue appears to persist (and the above looks like it is not scheduled to be solved for 0.16.4, or any upcoming release?)

I have put together this little repository (https://github.com/fangohr/tmp-spack-clingo-issue) which uses github workflows and Docker containers to illustrate the problem (on a Debian image). In essence:

• installing zlib works fine with spack 0.16.3.
• installing zlib fails with the spack develop branch
• unless one installs (on a Debian system) the packages libpython3-dev and python3-distutils as additional requirements.

All details, including the failed build attempt, can be taken from the repository.

[alalazo]

@fangohr From the failed build attempt you linked it seems bootstrapping from binaries failed and Spack is falling back to bootstrap from sources.

[alalazo]

I think you may be missing some system requirements and therefore you are not able to bootstrap from binaries. For instance, if I reproduced your environment correctly:

root@acf433e953b5:/# which file
root@acf433e953b5:/#

you don't have file installed, and that is required to use binary packages. Without binary packages you need libpython3-dev etc. because you are building clingo from sources.

Also, I don't know if it is of interest, but we provide a few docker images with clingo already bootstrapped from binaries.

[fangohr]

@alalazo - thank you for the quick and useful feedback. As you said: one of the dependencies was missing (the file command). Having added that to the image, all works as it should.

Thanks for the link to the Docker images. What would be interesting is to see the source of the Dockerfiles that are used to build these containers: that's what I got wrong. (I ran this in a container to be sure I got all dependencies right in spack package development, not because I need a container.)

[haampie]

There's also spack install spack now :)

[alalazo]

What would be interesting is to see the source of the Dockerfiles that are used to build these containers: that's what I got wrong.

@fangohr You can find the Dockerfiles under share/spack. For instance this is the Dockerfile used for Ubuntu 18.04

[fangohr]

@alalazo That's really useful - thank you!