Revisit which soft-preferences are global vs. per-package in packages.yaml

[alalazo]

Summary

Currently, we can set soft-preferences for version, providers, variants, compiler and target in packages.yaml both under the all attribute (which makes the prescription valid for any package) and under each specific package. The corresponding documentation is here. The proposal here is to change the semantic and allow:

1. version only under each package
2. target, compiler and providers only under all
3. variants valid under both (details on the semantic to be defined)

Rationale

Having all of the attributes allowed both under all and under each package name leads to semantic conflicts or to situations that make little sense. For version, it is highly unlikely that:

packages:
  all:
    version: ['1.0']

will ever make sense, since the versioning scheme is something that is tightly coupled to each package.

For compiler, providers and target there are currently questionable decisions in the solver on how to assign weights for the optimization. These decisions are forced to keep some backward compatibility with a packages.yaml structure which might contain conflicts. An example tied to target is:

spack/lib/spack/spack/solver/concretize.lp

Lines 782 to 791 in ec9803a

	% If the package does not have any specific weight for this
	% target, offset the default weights by the number of specific
	% weights and use that. We additionally offset by 30 to ensure
	% preferences are propagated even against large numbers of
	% otherwise "better" matches.
	target_weight(Target, Package, Weight)
	:- default_target_weight(Target, Package, Weight),
	node(Package),
	not derive_target_from_parent(_, Package),
	not package_target_weight(Target, Package, _).

Note that a packages.yaml with conflicting choices can lead to unexpected behavior (e.g. if two transitive dependencies specify different compilers it's not clear which one will be picked by a common dependency of both). Complicating the weight structure can also lead to unexpected, and difficult to diagnose, performance regressions - like the one in #30997. If the soft preferences for compiler, target and providers were always global, we could avoid workarounds and have a unique set of weights for each category. This will simplify the logic program and maybe (to be verified) speed-up the overall concretization process.

Allowing compiler, providers and target to be global only can be complemented by adding the ability to specify hard-requirements in packages.yaml on a package-by-package base, similarly to what drafted in #27987

Description

Description of a possible implementation is within the "Rationale" section. Still to be defined how, in case, we should deal with backward compatibility.

Additional information

For completeness, some issues with compiler and target are also stemming from the decision to minimize mismatches at a higher priority than selecting the best compiler/target. I think this was the right decision, but it's worth pointing that out in case somebody wants to explore that part of the design space.

General information

• I have run spack --version and reported the version of Spack
• I have searched the issues of this repo and believe this is not a duplicate