Software model for "compiler as dependencies"

[alalazo]

Short description

As a core developer I want to clearly define how Spack software model will change when compilers are turned into proper dependencies, so that I can assess its consistency with respect to relevant use cases and I can have a blueprint guiding the implementation.

Motivation and goal

Compilers in Spack are currently modeled as compulsory node attributes in the concretized DAG. This choice has historical reasons, since such a simplified model allowed to build a lot of useful features without having to deal first with the complexity of compiler runtimes and with other aspects that make compiler different from other kind of dependencies.

Nonetheless, treating compilers as node attributes has also shortcomings that are becoming more evident as the use cases that Spack can cover broaden in scope:

1. Some languages are more equal than others, since Spack consider only C, C++ and Fortran in the definition of a "compiler spec"
2. Enforcing a compiler on each node causes a lot of duplication when installing packages that don't need a compiler (e.g. pure Python packages, or binaries like cudatoolkit etc.). The duplication in the software model is reflected in wasted space on disk.
3. The current definition of a "compiler spec" doesn't allow a "natural" modeling of cross-compilation, since we can't easily talk about the compiler's host architecture vs. its target architecture. Currently Spack can deal with cross-compilation in a few ad-hoc ways, e.g. on Cray machines.
4. Spack cannot ensure any consistency among runtimes that are injected by compilers, since we don't model them. For instance, Spack cannot ensure that a single OpenMP runtime is selected in a complex DAG where multiple compilers are used.
5. Some compilers depend on other compilers for some functionalities, for instance Intel compilers depend on GCC for some C++ features. Currently we don't keep track of any of these dependencies, and their management is completely left to users.

To deal with the problems above we want to turn compilers from being a node attribute to be fully-fledged dependencies, and use virtual dependencies to model compiled languages. In that way:

• Packages that have no need for compilers can avoid declaring a dependency on any compiled language
• Packages that need a compiler for a language other than C, C++ or Fortran can declare a dependency on the language they need

This epic story collects all the use case requirements that are related to this change in the software model.

Requirements and use cases

• Requirements for keeping track of runtime libraries #31359
• Consistency requirements when using multiple compilers in a DAG #31361

References

• Fosdem 18 talk introducing the idea