[feat] Introduce new naming scheme for tests

[vkarak]

This PR introduces a new naming scheme for tests and fixture. The new naming scheme for test is enabled by setting the RFM_COMPACT_TEST_NAMES environment variable or the corresponding configuration parameter, but fixtures are always named using the new convention. There are several improvements also in the way tests are listed. Considerable parts of the code have also been adapted to follow the new conventions. More details on what this PR offers follow below.

Naming scheme

Each test receives two name attributes.

1. The unique_name: this is the unique identifier of the test within ReFrame. This cannot be set by the users. The current name attribute is an alias to this. For compatibility reasons, you can still set name (and thus the unique_name but this operation is now deprecated)
2. The display_name: this is a version of the test name that encodes useful information, such as parameterisation and scopes for fixtures. This is used when displaying tests in listings, because it is much easier to understand what the test is about.

Unique names

The unique name of the test is formed as follows:

• Normal non-parametrised tests: the name of the test is the class name, but not the qualified name of the class. The reason for not using the qualified name is that this does not offer anything at the test level and leads to names with non alphanumeric characters (qualified names include<locals> for locally defined classes) and, secondly, it facilitates the unit tests.
• Parametrised tests: Unique names of parametrised tests are formed as <test_class_name>_<variant-id>. We don't use any special character in the unique name so that it is compatible with any program and filesystem (see for example cmake builds fail when fixtures are used due to special characters in test names #2301). Compared to the current naming scheme, this one ensures unique name also in the case of a parameter that takes twice the same value as with x = parameter([1, 1, 3]. This generates three tests with unique names, whereas with the current naming scheme two of the generated tests would get the same name.
• Fixtures: Fixture names are formed as <fixture_class_name>_<short_hash>. The short-hash is the first 8 characters of a SHA256 hash that is computed from all the fixture differentiating factors, namely, the variant number of the fixture, the scope encoding (e.g., dom:gpu+gnu for an environment scope) and any variables set in the fixture.

Dependencies

The name argument passed to the depends_on() method should refer to the target test's unique name. Since there is no assumption as of how the variant numbers are assigned to the actual instances of a parameterized test, the correct way to depend on an instance of such a test is the following:

class X(RegressionTest):
    p = parameter([1, 2, 3])

class Y(RegressionTest):
    @run_after('init')
    def depend_on_x2(self):
        x_variant_num = X.get_variant_nums(p=lambda x: x == 2)[0]
        self.depends_on(X.variant_name(x_variant_num))

This way of depending on a parameterized test is also valid with the current naming scheme and should be the preferred way of writing such dependencies.

Display names

The display name of a test or a fixture encodes the parameterisation and scope information in a human-readable way and it may or may not be unique. The display name is evaluated once, the first time the display_name property is used, and it is then cached for later access. The construction of the display name requires to recursively descend in fixture chains if a test is eventually parametrised due to a fixture down in the hierarchy; that's why we cache the display name. The display name has the following syntax:

<test_class_name>(" %"<param>=<value>)+(<fixture_inst>)* where <param> can take the form <fixture_var>["."<fixture_var>]*"."<param> if the test is parametrised implicitly due to fixtures. <fixture_var> is the name of the variable fixture in the corresponding test or fixture. <fixture_inst> can take the form
<fixture_scope> takes the form of <system_name> for fixtures with session scope, <partition_name> for fixtures with partition scope, <partition_name>"+"<environment_name> for fixtures with environment scope and <test_unique_name> for fixtures with test scope. The following example demonstrates the naming scheme. Assuming the test:

import reframe as rfm


class MyFixture(rfm.RunOnlyRegressionTest):
    p = parameter([1, 2])


class X(rfm.RunOnlyRegressionTest):
    foo = variable(int, value=1)


@rfm.simple_test
class TestA(rfm.RunOnlyRegressionTest):
    f = fixture(MyFixture, scope='test', action='join')
    x = parameter([3, 4])
    t = fixture(MyFixture, scope='test')
    l = fixture(X, scope='environment', variables={'x': 10})

    valid_systems = ['*']
    valid_prog_environs = ['*']

This is how the above test and its fixtures will be displayed:

- TestA %x=4 %l.x=10 %t.p=2
    ^MyFixture %p=1 ~TestA_3
    ^MyFixture %p=2 ~TestA_3
    ^X %x=10 ~generic:default+builtin
- TestA %x=3 %l.x=10 %t.p=2
    ^MyFixture %p=1 ~TestA_2
    ^MyFixture %p=2 ~TestA_2
    ^X %x=10 ~generic:default+builtin
- TestA %x=4 %l.x=10 %t.p=1
    ^MyFixture %p=2 ~TestA_1
    ^MyFixture %p=1 ~TestA_1
    ^X %x=10 ~generic:default+builtin
- TestA %x=3 %l.x=10 %t.p=1
    ^MyFixture %p=2 ~TestA_0
    ^MyFixture %p=1 ~TestA_0
    ^X %x=10 ~generic:default+builtin

We skip the discussion about the ^ character at this moment. Notice how the fixture parameters are denoted, as well as the x variable in the fixture instantiation of X.

Formatting values

By default, parameter values are formatted by taking their string representation (i.e., calling str()). However, this may not always be convenient especially for builtin aggregate types as in the benchmark_info in the following example:

    benchmark_info = parameter([
        ('Cellulose_production_NVE', -443246.0, 5.0E-05),
        ('FactorIX_production_NVE', -234188.0, 1.0E-04),
        ('JAC_production_NVE_4fs', -44810.0, 1.0E-03),
        ('JAC_production_NVE', -58138.0, 5.0E-04)
    ])

For this reason, the parameter() built-in signature is extended to support an additional fmt argument that allows custom formatting. This argument should refer to a callable accepting a single argument (the value to be formatted) and return the formatted value. The returned value may not necessarily be a string, in which case it will be converted with str(). Passing fmt = lambda x: x[0] to the parameter definition of this example, will format the benchmark_info parameter using only the benchmark name.

Listing dependencies

Test or fixtures that are in the dependency chain of another test are not listed in separate entries, but instead they are listed indented and prefixed with the ^ character. The following shows the listing of the tests defined in deps_complex.py:

- T9
    ^T8
      ^T1
        ^T5
        ^T4
          ^T0
- T7
    ^T2
      ^T6
        ^T1
          ^T5
          ^T4
            ^T0
- T3
    ^T6
      ^T1
        ^T5
        ^T4
          ^T0

Notice that only the terminal tests are listed at first level whereas their dependencies are listed in DFS order and only once (even if a test appears in multiple dependency paths).

Command line options

This PR extends the -l and -L options, changes the behaviour of the -n option and adds a new --describe option.

Test listings

The -l and -L options accept now an optional argument that controls whether a list of tests is required or a list of concretised test cases. A concretisation of a test are the test cases that it produces, along with the test cases of its fixtures, based on the selected system partitions and programming environments. The concretisation of the same test can change in different systems as well as when the -p and/or --system options are passed. The default listing is for tests (as of now) and can be achieved with -l or -lT (and similarly for the -L option). The normal listing lists the tests by their display name only, whereas the detailed listing adds the unique test name and the file where the test was defined. Here are a couple of examples:

Normal listing

./bin/reframe -C tutorials/config/settings.py --system=generic -c tutorials/basics/hello/hello2.py -l

[List of matched checks]
- HelloMultiLangTest %lang=cpp
- HelloMultiLangTest %lang=c
Found 2 check(s)

Detailed listing

./bin/reframe -C tutorials/config/settings.py --system=generic -c tutorials/basics/hello/hello2.py -L

[List of matched checks]
- HelloMultiLangTest %lang=cpp [id: HelloMultiLangTest_1, file: '/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py']
- HelloMultiLangTest %lang=c [id: HelloMultiLangTest_0, file: '/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py']
Found 2 check(s)

Concretised listings

./bin/reframe -C tutorials/config/settings.py --system=generic -c tutorials/basics/hello/hello2.py -lC

[List of matched checks]
- HelloMultiLangTest %lang=cpp @generic:default+builtin
- HelloMultiLangTest %lang=c @generic:default+builtin
Concretized 2 test case(s)

Notice that now the concretisation suffix @<part>+<env> is added to denote where this test case will actually run. Changing the --system option in this case, although it will not change the normal listing, it will change the concretised listing since the catalina system defines two programming environments:

[List of matched checks]
- HelloMultiLangTest %lang=cpp @catalina:default+gnu
- HelloMultiLangTest %lang=cpp @catalina:default+clang
- HelloMultiLangTest %lang=c @catalina:default+gnu
- HelloMultiLangTest %lang=c @catalina:default+clang
Concretized 4 test case(s)

Selecting tests by name

The -n option matches now the display name of a test by removing first any spaces. This allows users to select tests by their parameters easily as in the following example that selects all Amber CUDA benchmarks:

./bin/reframe -c hpctestlib/sciapps/amber/nve.py -S valid_systems='*' -S valid_prog_environs='*' -n 'amber.*%variant=cuda' -l

[List of matched checks]
- amber_nve_check %benchmark_info=JAC_production_NVE %variant=cuda
- amber_nve_check %benchmark_info=JAC_production_NVE_4fs %variant=cuda
- amber_nve_check %benchmark_info=FactorIX_production_NVE %variant=cuda
- amber_nve_check %benchmark_info=Cellulose_production_NVE %variant=cuda
Found 4 check(s)

You can also perform an exact match for a test using the @<variant_num> suffix as in the following example:

./bin/reframe -c hpctestlib/sciapps/amber/nve.py -S valid_systems='*' -S valid_prog_environs='*' -n 'amber_nve_check@1' -l

[List of matched checks]
- amber_nve_check %benchmark_info=Cellulose_production_NVE %variant=cuda
Found 1 check(s)

The reason why we use @ for exact matches and the unique name is because, otherwise, we would need a separate option for unique name matches.

NOTE: Fixtures cannot be selected individually.

Describing tests

The new --describe option will return a JSON representation of a test that lists all of the tests public fields, as well as all its required variables. The --describe option acts a typical action option, so it can be combined any test selection option. Here is an example output:

./bin/reframe -c hpctestlib/sciapps/amber/nve.py -S valid_systems='*' -S valid_prog_environs='*' -n 'amber_nve_check@1' --describe

[
  {
    "@class": "amber_nve_check",
    "@file": "/Users/karakasv/Repositories/reframe/hpctestlib/sciapps/amber/nve.py",
    "@required": [
      "perf_patterns",
      "sanity_patterns",
      "num_tasks",
      "executable",
      "descr",
      "energy_tol",
      "energy_ref",
      "benchmark",
      "input_file"
    ],
    "benchmark": "Cellulose_production_NVE",
    "benchmark_info": [
      "Cellulose_production_NVE",
      -443246.0,
      5e-05
    ],
    "build_locally": true,
    "build_system": null,
    "build_time_limit": null,
    "container_platform": null,
    "descr": "Amber NVE Cellulose_production_NVE benchmark (cuda)",
    "display_name": "amber_nve_check %benchmark_info=Cellulose_production_NVE %variant=cuda",
    "energy_ref": -443246.0,
    "energy_tol": 5e-05,
    "exclusive_access": false,
    "executable": "pmemd.cuda.MPI",
    "executable_opts": [
      "-O",
      "-i",
      "mdin.GPU",
      "-o",
      "amber.out"
    ],
    "extra_resources": {},
    "input_file": "mdin.GPU",
    "keep_files": [
      "amber.out"
    ],
    "local": false,
    "maintainers": [],
    "max_pending_time": null,
    "modules": [],
    "num_cpus_per_task": null,
    "num_gpus_per_node": 0,
    "num_tasks_per_core": null,
    "num_tasks_per_node": null,
    "num_tasks_per_socket": null,
    "output_file": "amber.out",
    "perf_variables": [
      "perf"
    ],
    "pipeline_hooks": {
      "post_init": [
        "prepare_test"
      ]
    },
    "postbuild_cmds": [],
    "postrun_cmds": [],
    "prebuild_cmds": [],
    "prerun_cmds": [
      "curl -LJO https://github.com/victorusu/amber_benchmark_suite/raw/main/amber_16_benchmark_suite/PME/Cellulose_production_NVE.tar.bz2",
      "tar xf Cellulose_production_NVE.tar.bz2"
    ],
    "readonly_files": [],
    "reference": null,
    "sourcepath": "",
    "sourcesdir": null,
    "strict_check": true,
    "tags": [
      "sciapp",
      "chemistry"
    ],
    "time_limit": null,
    "unique_name": "amber_nve_check_1",
    "use_multithreading": null,
    "valid_prog_environs": [
      "*"
    ],
    "valid_systems": [
      "*"
    ],
    "variables": {},
    "variant": "cuda"
  }
]

Notice that the attributes are sorted alphabetically for easy reference. There are three special attributes starting with @.

This option, as well as the other two options returning JSON output, --show-config and --detect-host-topology override the verbosity of the output and set it to ERROR so that any framework output is suppressed except errors. The verbosity is restored to INFO right before the output of these options. This ensures that if these options are successful, their output will always be a valid JSON and it can be piped to utilities such as jq regardless of the frameworks verbosity level.

Implementation details

This PR changes primarily the way parameters are stored in the parameter space as well as how fixture names are constructed and set.

Changes in parameter implementation

Regarding parameters, there was a discrepancy in the way they were stored in the parameter space compared to variables and fixtures. The variable and fixture spaces stored TestVar and TestFixture objects, whereas the parameter space stored only the parameter name and the values. As a result, it was not straightforward to provide parameter-level metadata, such as the formatting function passed to fmt. Now the parameter space stores parameter objects, which also take care of updating their values as parameters are inherited from the parent namespaces, an operation that used to happen inside the parameter space in the past. The current implementation of updating the values of a parameter is more natural, since the parameter object stores the information as of whether to inherit parameters and to filter them. The look-and-feel of the parameter space however does not change substantially, since iterating and accessing elements over it, still behaves as before (see the unit tests).

The fact that now parameters objects are stored in their entirety in the parameter space, allows the display_name property in the RegressionTest to properly access the formatting function and format the parameters, but in the future, we could extend the metadata to include source code information (see #2308).

Changes in fixtures implementation

The formatting of the fixture's display name does not happen in the fixture namespace as it was effectively happening in the past, due to no distinction between unique and display names. Instead, only the unique name is constructed, which is the class name a unique hash that derives from the metada of the fixture stored in FixtureData. This is object is now also passed to the actual test instance, in order to be used in the actual formatting done in the display_name.

Other implementation improvements

• All references to the name attribute in the framework were replace to references either to unique_name or display_name, depending on the context.
• The syntax of all reframe tests appearing in the unit tests has been modernised. Any exceptions are noted in comments.
• A new make_test utility function is introduced (primarly due to use cases in the unit tests) that allows you to create a test programmatically as if you were declaring it with the class keyword. The following

hello_cls = rfm.make_test(
    'HelloTest', (rfm.RunOnlyRegressionTest,),
    {
        'valid_systems': ['*'],
        'valid_prog_environs': ['*'],
        'executable': 'echo',
        'sanity_patterns': sn.assert_true(1)
    }
)

is completely equivalent to

class HelloTest(rfm.RunOnlyRegressionTest):
    valid_systems = ['*']
    valid_prog_environs = ['*']
    executable = 'echo',
    sanity_patterns: sn.assert_true(1)
    
hello_cls = HelloTest

This is very useful if you want to create uniquely named tests from the same non-parametrised test.

• The run report and failure info are also adapted to show the display name of the tests.
• A mechanism is introduced to easily deprecate variables and also make variables aliases of attributes with a different name. The following demonstrates how the name attribute is deprecated and how it is made to point to _rfm_unique_name:

    name = deprecate(variable(typ.Str[r'[^\/]+'], attr_name='_rfm_unique_name'),
                     "setting the 'name' attribute is deprecated and "
                     "will be disabled in the future", DEPRECATE_WR)

The deprecate() built-in creates a variable from another target variable and backs the new variable with a DeprecatedField. Additional logic is added in the variable to check and issue deprecation warnings in case the variable is accessed or set while test hierarchies are defined. After the variable is injected into the final test object, the DeprecatedField is responsible for issuing the warnings.

Todos

• User documentation (also adding a note about how to get the variant number of a target dependency test)
• Improve the implementation of name deprecation
• Extend the get_variant_nums so that the values in the conditions argument can be simple value, in which case the equality condition should be assumed. This is simply a convenience.
• Fix the few CSCS tests that set the test's name.
• Extend the output of the --describe option with some useful properties, such as the unique_name and display_name.

Fixes #2248
Fixes #2222
Fixes #2301
Fixes #2313
Fixes #2382
Fixes #2391