vendor libstdc++ and clingo for bootstrapping from source in arcane environments

[cosmicexplorer]

One alternative to #20068 (possibly complementary?) is to attempt to hack away at the build system for clingo (CMake) so that it will avoid introducing a requirement on a C++14 compiler and we can reliably bootstrap it from source in arcane environments.

Rationale

Removing the futuristic and unnecessary C++14 requirement doesn't break anything (yet)! It currently doesn't require more than a small patch:
cosmicexplorer/clingo@a1db68e

Description

I found the above clingo commit built and was successfully importable with python 2.6.7, and editing the CMakeLists.txt as in the HEAD commit avoids producing any command lines requiring a C++14 compiler!

So only two changes are necessary to keep this clingo patchset up to date and buildable on e.g. RHEL/CentOS 7 as per #20069! If we hooked that up into spack, it would likely be a lot more stable / a lot less tech debt than a fork (which I was previously worried about for some version).

Additional information

I think #20068 seems 100% orthogonal to this approach -- rather, developing a reliable (reproducible?) method of bootstrapping from sources seems like the key to both of these issues. If we can trust our robust script, we can easily re-ship updated binaries for multiple architectures as in #20068, while ensuring via CI that our bootstrap script avoids correctness (or performance?) regressions as we try to self-bootstrap.

I think from any self-bootstrapping process for a tool like spack that runs itself from source, I as a user might (eventually) have these requirements:

1. Users will clearly understand when a bootstrap occurs (can possibly cancel it if performed as a transaction?).
2. Users will be able to point spack to their own internal toolchain binary distribution server (servers?) without any configuration.
   • Spack could (like pants does) assume such a server's endpoints are laid out corresponding to a hierarchical filesystem tree.
   • The docs clearly and unambiguously state how to do this.
3. There is a tag created specifically to track issues with binary resolution.

General information

It looks like spack already exposes the entry point for cron jobs/etc to clean up space with spack gc, and if we can make the bootstrap "look like a normal spack install", we may be able to get control of that like this for free.

Relatedly, here's a really interesting discussion thread from the pants repo on invalidating external binaries: pantsbuild/pants#10768 (comment)

Aaaaand ... this is still broken in realistic ways. Namely, upgrading or downgrading a binary found here previously will result in the same output from this rule and thus not trigger a re-run of rules depending on this data, which is wrong. The only cases currently covered by this fix from the list below are 1 and 2:

An applicable binary is added to the search path.
An applicable binary is removed from the search path.
An applicable binary on the search path is modified (upgraded or downgraded most likely).

To handle 3 the contents of the discovered binaries need to be hashed (And even that is not enough! Dynamically linked libraries could change the binaries output ... but we'll need to whistle past the graveyard - that level of detail can really only be addressed by always running in an image / using a local docker / podman / runc / crun type solution). This could be done by storing it in the CAS as a hack, but we won't / can't generally use the binary stored in the CAS so that just adds more (see make_process_volatile comment) storage bloat. Ideally we'd ask an engine intrinsic to get us a fingerprint without storing the object. It could also be done by just hashing the resulting absolute path in python rule code at the expense of potentially heavy IO inhibiting parallelism. On my machine, hashing my biggest binary (/usr/bin/packer @ 154M) takes ~400ms in pure python code and hashing the currently more typical python binary (@ 19M) takes ~50ms.

I'd like to defer handling case 3 to a follow-up if folks are amenable.

[cosmicexplorer]

Going to make a PR of this today/tomorrow!

[cosmicexplorer]

I think the gnu autotools are going to be a great place to learn from. The automake/INSTALL file is a very good description of how things can go wrong when building your project: https://git.savannah.gnu.org/gitweb/?p=automake.git;a=blob_plain;f=INSTALL;hb=HEAD.

The more subtle distinction between autoconf, automake, and m4's responsibilities, vs something like gcc's, stems from the really stark distinction in where each of them except gcc are required strictly before the build (not at all even during what most people consider to be "the build") -- it demonstrates I think how the gnu build system has this really well-developed concept of lowering itself down from the full build system, into just traces of it, suitable e.g. for distribution (so, ./configure shell scripts instead of .ac or .in source files to autoconf). That would be a really nice way to be able to talk about spack packaging concepts in terms that are familiar to those who know how to run ./configure.

I would also really like to be able to serialize a "fully resolved" spack environment (i.e. right after a successful concretization) into some format (JSON) and then be able to isolate spack's "how to build all the packages" logic into a subset of the codebase which doesn't touch clingo. That leads to one quick success criteria for this phase of the concretizer investigation being "can we remove the old concretizer?" (without breaking anyone) as @alalazo has described. I also think that this particular kind of separation (along process boundaries) is more likely to be a good atomic unit than python threads for parallelism.

[tgamblin]

Commenting to get notifications

[cosmicexplorer]

Current progress:

1. bootstrapping: bootstrap spack dependencies (executable and python module) #20207 covers lazily building clingo with the old concretizer.
2. bootstrap a python 2.6 installation #20218 covers building a python2.6 install which we can test against in CI.
3. clean up how spack builds clingo #20159 covers building clingo in a py2.6, c++11-only environment.
4. add patches to bootstrap z3 with python 2.6 #20219 covers building z3 in a py2.6-only environment.
   • This may be unnecessary -- see consider z3 implementation of ASP concretizer #20221.

[cosmicexplorer]

#20159 has been very close for weeks....going to try to put that to bed today.

[cosmicexplorer]

Adventures from #20159 include:

# TODO(#20123): there are multiple alternatives to this script:
#  - TODO(#20430): package spack as a PEX file to avoid mixing up spack-installed packages
#                  and others!
#  - TODO(#20260): sandbox all builds with fakechroot to avoid this double bootstrapping!
# - Add your own! #20207 makes python configuration a lot smarter, and yet still fails CentOS 6.

Referencing #20430, #20260, and #20207 here. @tgamblin, @becker33: I am 100% aware that neither pex nor sandboxing are important goals for us right now. Regardless, I currently have two solutions for the clingo bootstrap problem, which I'm pretty sure isn't going to go away. I could figure out another one, but I think I might also just want to refresh my memory with @becker33 which PRs he may be waiting on from me (or none).

[cosmicexplorer]

Like one alternative is to start patching CMake python support, and that got very difficult very fast.