Create the ability for Spack to recognize Spack installs for externals

[psakievich]

Summary

Let's make Spack be able to recognize spack installs for externals. We parse a lot of the information for the builds/package into a json file that gets included in every install (environments, views, opt, etc). As a user I would like to be able to use spack external to get externals from other users spack instances by pointing to a spack environment or view they've created.

@tgamblin @becker33 @scheibelp @haampie @alalazo

Rationale

It's no secret that people want to re-use binaries and not keep re-building them. Spack has several elegant ways of addressing this:

• Upstreams
• --reuse
• Binary Caches
• Externals

The issue I have with the first 3 is they all rely on the concretizer to choose the "best" version of the software for me.
However, in the development workflow I often want explicit control over which binaries I am using.

Some examples:

• developer a is developing an upstream dependency and wants developer b to test it. Sure developer b can re-build, but why not just link against their installation in the first place?
• I want to store time stamped releases of my software and link against them. The spack package is not changing so the hash isn't likely to change. Should I need to create a new instance of spack every time I want to upstream this, and then swap out my upstreams? Same question for binary caches?

I think there is a fundamental weakness in the first 3 ways of including previous builds: they all rely on the concretizer to pick you binaries for you.

This is where externals seems like a perfect fit, but they have long been a dark horse for Spack. This is because they are typicaly used for system installed packages which Spack has no idea about.

However, I propose that spack external --path [foo] when pointed to:

1. A valid Spack environment
2. A valid Spack view

should be able to generate externals that the concretizer can handle, and spack can use as efficiently as an upstream or binary cache.

Description

I would like to be able to run spack external --path /path/to/spack/view-or-env with options to blacklist or whitelist packages. A bonus feature would be the ability to dump to a file instead of automatically including them in my spack.yaml or packages.yaml.

We've prototyped this as part of Spack-Manager with the following interface:

spack manager external --help
usage: spack manager external [-hm] [-n NAME] [-v VIEW] [-w [WHITELIST ...] | -b [BLACKLIST ...]] [--latest] [--list] [path]

positional arguments:
  path                  The location of the external install directory

optional arguments:
  --latest              use the latest snapshot available
  --list                print a list of the available externals to use.
                        Values in parenthesis are the view names for each external
  -b [BLACKLIST ...], --blacklist [BLACKLIST ...]
                        (not implemented) specs that should be omitted (add all others)
  -h, --help            show this help message and exit
  -m, --merge           merge existing yaml files together
  -n NAME, --name NAME  name the new include file for the externals with this name
  -v VIEW, --view VIEW  name of view to use in the environment.
                        This will default to the first view in the environment
                        i.e. the first view listed in "spack manager external --list" command
  -w [WHITELIST ...], --whitelist [WHITELIST ...]
                        (not implemeted) specs that should be added (omit all others)

Our version only works with environment that have packages partitioned into views. And it only writes externals to an externals.yaml file which is where the --merge flag comes from. I would really like to extend this feature to views and make it something that is functional in mainline Spack, but I will likely need help from the core development team to get it worked out. I have a very limited number of cycles for Spack development right now due to other project work.

Additional information

The source code for the Spack-Manager implementation is here:
https://github.com/psakievich/spack-manager/blob/main/spack-scripting/scripting/cmd/manager_cmds/external.py.

Our implementation also currently suffers from limitations in Clingo to concretize the spack spec's we write to file, and so we must use the original concretizer. See #28201

General information

• I have run spack --version and reported the version of Spack
• I have searched the issues of this repo and believe this is not a duplicate