enable spack core code to depend on the output of a spack build

[cosmicexplorer]

Spack doesn't seem to have a concept of "detecting" or "installing" binaries (like xz) that it assumes are available on the PATH in the interstitial code that runs before and after package build steps. This means that it fails with very opaque error messages in situations that Spack itself can fix!

This ticket would likely be a good prequel to #20523, because this ticket focuses more on fixing a weird inconsistency than trying to develop an alternative model for types of dependencies as in #20523.

Rationale

Spack isn't currently able to detect that it needs to build xz when pulling down a .tar.xz package if it's being run on e.g. a bare centos6. On our spack/centos6 container, I can get the following error message (for a different missing executable: bzip2), which is remarkably opaque if you haven't seen it before:

> spack --debug --stacktrace install -y -n --fail-fast -j 12 --verbose nim@0.20.0
...
lib/spack/spack/fetch_strategy.py:336 ==> [2020-12-22-02:28:58.024604] Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/19/19108658b23b3ec5058edc9f66ac545ea19f9537234be1ec62b714c84399366d.tar.bz2
lib/spack/spack/util/executable.py:162 ==> [2020-12-22-02:28:58.025649] '/usr/bin/curl' '-C' '-' '-o' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2.part' '-f' '-D' '-' '-L' 'https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/19/19108658b23b3ec5058edc9f66ac545ea19f9537234be1ec62b714c84399366d.tar.bz2' '-#' '--connect-timeout' '10'
######################################################################## 100.0%
lib/spack/spack/fetch_strategy.py:445 ==> [2020-12-22-02:28:58.665538] Staging archive: /tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2
lib/spack/spack/util/executable.py:162 ==> [2020-12-22-02:28:58.668826] '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'
tar (child): lbzip2: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
/usr/bin/tar: Child returned status 2
/usr/bin/tar: Error is not recoverable: exiting now
lib/spack/spack/error.py:55 ==> [2020-12-22-02:28:58.688234] Error: ProcessError: Command exited with status 2:
    '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'
Traceback (most recent call last):
  File "/h/tools/spack/lib/spack/spack/build_environment.py", line 851, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/h/tools/spack/lib/spack/spack/installer.py", line 1641, in build_process
    pkg.do_patch()
  File "/h/tools/spack/lib/spack/spack/package.py", line 1368, in do_patch
    self.do_stage()
  File "/h/tools/spack/lib/spack/spack/package.py", line 1354, in do_stage
    self.stage.expand_archive()
  File "/h/tools/spack/lib/spack/spack/util/pattern.py", line 18, in __call__
    for item in self.container]
  File "/h/tools/spack/lib/spack/spack/stage.py", line 590, in expand_archive
    self.fetcher.expand()
  File "/h/tools/spack/lib/spack/spack/fetch_strategy.py", line 72, in wrapper
    return fun(self, *args, **kwargs)
  File "/h/tools/spack/lib/spack/spack/fetch_strategy.py", line 468, in expand
    decompress(self.archive_file)
  File "/h/tools/spack/lib/spack/spack/util/executable.py", line 198, in __call__
    proc.returncode, long_msg)
ProcessError: Command exited with status 2:
    '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'
lib/spack/spack/installer.py:1301 ==> [2020-12-22-02:28:58.689396] Flagging pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff as failed: ProcessError: Command exited with status 2:
    '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'
lib/spack/spack/installer.py:1311 ==> [2020-12-22-02:28:58.694483] Warning: Skipping build of nim-0.20.0-wb5pmjyo7qxcdcqcisch2d3hd6tjgt3u since pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff failed
lib/spack/spack/installer.py:1301 ==> [2020-12-22-02:28:58.694691] Flagging nim-0.20.0-wb5pmjyo7qxcdcqcisch2d3hd6tjgt3u as failed
lib/spack/spack/main.py:765 ==> [2020-12-22-02:28:58.708811] InstallError: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'
lib/spack/spack/error.py:55 ==> [2020-12-22-02:28:58.709064] Error: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'
Traceback (most recent call last):
  File "/h/tools/spack/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/h/tools/spack/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/h/tools/spack/lib/spack/spack/cmd/install.py", line 371, in install
    install_specs(args, kwargs, zip(abstract_specs, specs))
  File "/h/tools/spack/lib/spack/spack/cmd/install.py", line 213, in install_specs
    builder.install()
  File "/h/tools/spack/lib/spack/spack/installer.py", line 1582, in install
    .format(fail_fast_err, str(exc)))
spack.installer.InstallError: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/usr/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-pcre-8.44-aek3wkf4dpptjoxzc5x456w4naiaggff/pcre-8.44.tar.bz2'

Description

Chronologically, this is how I approached the issue:

1. problem: spack seems to require several unannounced dependencies that screwed me up when i had a working empty centos6, most notably git, xz, bz2, gcc{,-c++}, python{,-devel}, a couple more? -- i am pretty sure it failed on make because make wasn't a high enough version.
2. partial solution: in that case, i was able to install git and then use spack to install the rest.
3. proposal: i would like to say "why can't we make spack automagically install each dependency it uses itself?"

Implementation

Implementing this might take the form of:

1. Determine all of spack's actual required dependencies from a centos 6 machine by running it until it succeeds.
   • Use a special wrapper method when invoking a subprocess anywhere in spack core code.
2. Determine whether to:
   • add a check to setup-env.sh which may run another spack instance?
   • add those tools to some sort of bootstrap image as per vendor libstdc++ and clingo for bootstrapping from source in arcane environments #20123 and/or consider bootstrapping spack as a standalone PEX to avoid interpreter clashes #20430?

[cosmicexplorer]

I would love for any advice on which of the two entries in the Implementation section to investigate further!! I would strongly prefer the first method so we can say for sure "Spack has exactly these requirements!" and then users who cannot meet those requirements know they can make an issue to ask for it. Here is an example I love from Pants: https://www.pantsbuild.org/docs/installation

The "special wrapper method" is intentionally vague because I am not confident I know the correct wrapper method at the start.

[cosmicexplorer]

#20123 describes (some of) the new workstream to smooth the spack bootstrap process for new users. When testing clingo on CentOS6, I ran into the problem in this issue and it was super frustrating because the error message (subprocess execution couldn't find the tool on the PATH) was very difficult to decipher and it can be difficult to know how many times you may have to keep rerunning spack. in extreme cases, even installing the necessary tools could fail, because of a separate instance of this same issue!

I think we want Spack to not only be able to bootstrap everywhere, it should also be easy to follow the directions on how to bootstrap anything super arcane.

[cosmicexplorer]

Running spack external find would probably be a good first step to advise in any docs. [EDIT]: Although, I don't think that would change the PATH at all.