Treat externals as concrete specs

[alalazo]

Summary

Right now, external specs are allowed to have missing pieces of information, and are "completed" during concretization with the bits that produce the best optimization values in clingo. This completion:

1. Is done regardless of what is really installed on a system, and
2. Creates a one (installed software) to many (concrete specs) mapping for externals

which is obviously wrong.

For Spack v1.1 we want to treat external specs as any other concrete spec. This means that each external spec will have a single, well-defined hash, a single set of immutable variants, etc External specs may also have dependencies, and properties along edges.

Rationale

This feature will solve issues where e.g. a missing variant is concretized differently for the same external, depending on the opportunity for optimizations. It would also solve any issue that arise because we trim the dependencies of externals.

From a maintenance point of view, it allows us to remove the code path for external specs in the solver, and consolidate it with the one for "reused" specs.

Description

Syntax to specify externals with dependencies

The syntax used to specify externals with dependencies is an extension of the current schema:

# Specification for the following DAG:
#
# o ascent@0.9.2
# |\
# | o adios2@2.7.1
# |/
# o bzip2@1.0.8
packages:
  ascent:
    externals:
    - spec: "ascent@0.9.2+adios2+shared"
      prefix: /user/path
      external_id: <ascent_id>
      dependencies: 
      - external_id: <adios2_id>
        deptypes: link
      - external_id: <bzip2_id>
        deptypes: "run"
  adios2:
    externals:
    - spec: "adios2@2.7.1+shared"
      prefix: /user/path
      external_id: <adios2_id>
      dependencies: 
      - external_id: <bzip2_id>
        deptypes: ["build", "link"]
  bzip2:
    externals:
    - spec: "bzip2@1.0.8+shared"
      prefix: /user/path
      external_id: <bzip2_id>

The extensions are the following:

• Each node being used as a dependency must have a unique external_id identifying it, and
• Each external node may specify dependencies, by their external_id, and by additional optional attributes that define the edge properties.

Complete missing information on an external node

To complete the pieces that are missing on each node (e.g. variants that are not specified), Spack will provide different strategies. Users will be able to select one strategy via the concretizer.yaml configuration. A tentative example of where to put that option might be:

concretizer:
  externals:
    completion: architecture_only|default_variants

The two strategies that will be implemented as a first cut are:

• Complete only the architecture information and don't add missing variants, or
• Complete architecture information and add default values for missing variants

Backward compatibility

External entries like mpich %gcc will be deprecated, since in the new model the mpich entry needs to point to a specific gcc external node.

To maintain backward compatibility in a case like the one above, Spack will:

1. Guess which gcc is the correct dependency for the mpich above, and
2. Issue a warning if the guess produced a valid result, informing the user of which gcc was selected, or error out if no gcc is available

Additional information

On top of this modification, we may want to extend the detection logic to automatically find external dependencies.

General information

• I have searched the issues of this repo and believe this is not a duplicate

[nolta]

I think i'm running into this issue. spack module lmod refresh failed complaining about name clashes, and it's apparently because there are multiple versions of the same external floating around. For example:

$ spack find -Ndlv --show-full-compiler openblas
-- linux-rocky9-zen5 / aocc@5.0.0 -------------------------------
jvlzw2w builtin.openblas@0.3.28%aocc@5.0.0 ~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile patches=d0b9276 symbol_suffix=none threads=openmp
kcoo3ll     builtin.glibc@2.34%gcc@14.2.0  build_system=autotools
ivwfx37     builtin.gmake@4.3%gcc@11.5.0 ~guile build_system=generic patches=599f134

wxwkmzz builtin.openblas@0.3.28%aocc@5.0.0 ~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile patches=d0b9276 symbol_suffix=none threads=openmp
icrcu7q     builtin.glibc@2.34%gcc@11.5.0  build_system=autotools
ivwfx37     builtin.gmake@4.3%gcc@11.5.0 ~guile build_system=generic patches=599f134

Note the two versions of the external glibc package, with two different hashes.

[melven]

To me, this seems to create real problems:

First build an llvm 16.0.6 and then add it as a compiler where it gets an external spec (or have an llvm 16.0.6 installed in the system).

Then I try to build julia which requires llvm with special variants. Spack then just "rebranches" the external llvm to the desired one.

Probably more an issue during the transition from "compiler-nodes" to "normal build dependencies" currently - but took me a while to track down!
(The resulting errors are missing files in the llvm directory while building julia)...

[psakievich]

@alalazo can we add a test that the externals with the the same module resolve to the same hash?

Rationale: We have a use case where administrators deploy the same compiler on multiple systems with different local paths, but with the same module name. We want to be able to get exact hash matches across the physical machines to utilize binary caches without reuse. We really want exact hash matches for all the other properties so we want reuse off for that reason, and for solver performance.

Since the compiler DAG hash drives the DAG hash of everything else, we need an effective strategy to ensure we don't need the same physical location of the compiler and can still use the binary cache.

[alalazo]

If the two specs have different prefixes, currently they don't have the same hash. I think we have two ways forward here:

1. Relax the hashing function of externals a bit more, and just use the module name (might be questionable)
2. Keep the current model, and use splicing instead

I'll check what seems the best among these two.