Allow users to create and use generalized variables in yaml files

[psakievich]

Summary

As a user I would like to see the concept of the definitions: section in the spack.yaml file extended to allow for arbitrary variables that can be used throughout all the config files. Essentially I would like to abstract out the concept of the $spack variable so that I can set similar variables myself.

A solution proposed by @becker33 in the weekly telecon would looks something like this:

spack:
  definitions:
     variables:
       - spack_manager: $spack/../ 
       - view_dir: $spack_manager/views/snapshots/20211231
     specs:
      - first: [trilinos]
      - cuda: ['+cuda cuda_arch=71', '~cuda']
      - second:
          - matrix:
              - [$first]
              - [$cuda]
  specs:
    - $second
    - cmake
  packages:
     hdf5:
        externals:
         - spec: hdf5@1.10.7
            prefix: $view_dir
         buildable: false

where the current definition of spack:definitions: now becomes spack:definitions:specs: since this section is constrained to defining specs. A second subsection spack:definitions:variables: is also added which will allow users to generate variables which can be used arbitrarily throughout the config files.

Further input from @scheibelp is that we should extend this functionality to all of the configs and allow a spack:definitions section in all of them to enable the use of user defined variables.

Tagging: @alalazo @tgamblin

Rationale

No response

Description

A preliminary attempt at this was made in #27925 was made to enable the concept of this feature through YAML anchors, but due to their more complicated syntax this is not the preferred path for a production capability. Additionally, it appears that the read/write operations with ruamel.yaml automatically overwrite the anchors and aliases. This issue may be addressed in the newer versions of ruamel and we should consider updating regardless. @alalazo mentioned he tried this recently but there is additional work that will be required with the update.

Additional information

No response

General information

• I have run spack --version and reported the version of Spack
• I have searched the issues of this repo and believe this is not a duplicate

[alalazo]

For the record the issue is mainly with:

spack/lib/spack/spack/config.py

Line 1041 in bbe69aa

def _mark_internal(data, name):

and similar functions that rely on undocumented internals of ruamel.yaml. The latest version of ruamel.yaml with Python 2.7 support is 0.16.13, see https://pypi.org/project/ruamel.yaml/0.16.13/ and the Mark class used in the function above has been removed.