Unwanted rebuilds with sequential environment builds

[greenc-FNAL]

Steps to reproduce

$ docker run -it docker.io/greenc/spack-mu2e-concretizer:init
# ./demonstrate 2>&1 | tee build.log

or

$ docker run -it docker.io/greenc/spack-mu2e-concretizer:in-progress
# . spack_env/setup-env.sh
# spack env list
⋮

Error message

The build error (in elfutils) is irrelevant. What matters here is that the mubase... environment in particular is forcing the rebuild of several packages that the naive user would expect to be used as already installed.

Notes:

1. The Spack installation obtained here is a Fermilab-specific fork based on 776ab13. There are some recipe-specific tweaks, but the only non-mainline item affecting the concretizer is the application of the still-in-flight PR reuse concretization: allow externals from remote when locally configured #35975.
2. I provide Docker images here because:
   • The above-mentioned Spack installation.
   • Use of a recipe repo.
   • Use of externals in packages.yaml.
   • Use of a script to manage the setup of the Spack installation and sequential build of specified environments from spack.yaml files.
3. The images were prepared with podman; podman may be used to obtain and run the image as a drop-in replacement for Docker.
4. There are two images, tagged :init and :in-progress. In both, the command ./demonstrate -- has been executed to install and configure the Spack installation and recipe repo; in the latter, the sequential environment build has been run to failure (via ./demonstrate).

I appreciate that this is by no means a "minimal reproducer." However, I'm hoping that someone taking an initial look at these containers will be able to give pointers on how to produce one, given the external recipe repo.

Any guidance on solving this (or on constructing a reproducer that renders it solvable by someone else) would be appreciated.

Information on your system

• Spack: 0.21.0.dev0
• Python: 3.6.8
• Platform: linux-scientific7-cascadelake
• Concretizer: clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[greenc-FNAL]

Including Slack threads relevant to the problem. Note that while the setup is different to that described in this issue and demonstrated in the containers (the second environment includes a matrix) the resulting undesired behavior is the same, and may be a little more tractable in the second scenario.


spack.yaml.txt
checkit.log.txt



spack.yaml.txt
spack.lock.txt




spack.yaml.txt
spack.lock.txt