Automatically configure ReFrame for the underlying system

[vkarak]

It would be nice if ReFrame could automatically detect the system it runs on (if no configuration is given) and generate a configuration file. The following things need to be detected at least:

1. System partitions
   • Modules system
   • Local host architecture (micro-arch + topology)
   • Job schedulers
   • Parallel launchers
2. Environments
   • This is a bit challenging. Spack is doing a similar thing in auto-detecting the compilers, so we could use Spack for this, but what about EasyBuild? We could potentially look for module name patterns. But what if the s/w stack is hidden behind a top-level module?
   • Any environments defined should follow a certain naming convention, so that tests can define valid_prog_environs accordingly. Similar thing should be done for valid_systems.

@ChristianKniep You can help formulate this. What information would you guys need from a system auto-detection?

[kniec]

As a MVP, I'd say focusing on single-node jobs with local execution (of containerized IMHO) while detecting the architecture and topology (core count, NUMA layout) is a great first building block.
This allows to define expected performance values for different system setups (on a 36core Skylake with hyperthreading off I expect XYZ). With that you can run this local test as a performance test on deployment.

Second step (IMHO) is to allow for multi-node jobs by adding the interconnect and add expected performance value per interconnect setup. Same goal; running a performance test after setting up the infrastructure and installing the software.
With containers you skip the module problem (even though I am biased towards containers). :)
Not sure for my use-case we do need to detect the programming environment as compilation won't be done.

[vkarak]

As a MVP, I'd say focusing on single-node jobs with local execution (of containerized IMHO) while detecting the architecture and topology (core count, NUMA layout) is a great first building block.
This allows to define expected performance values for different system setups (on a 36core Skylake with hyperthreading off I expect XYZ). With that you can run this local test as a performance test on deployment.

As a first target, that's doable. The devil might always might be in the details, as always, but I don't see a showstopper right away.

Second step (IMHO) is to allow for multi-node jobs by adding the interconnect and add expected performance value per interconnect setup. Same goal; running a performance test after setting up the infrastructure and installing the software.

In this case, the pain is to identify how you can gain access to the remote partitions, but I can see simplifications that could possibly work for an MVP. We can always assume a single "partition" or that there is a 1-1 mapping between scheduler partitions and ReFrame's partitions, which I guess is especially true when you provision a cluster on demand for running your workload. I'm not yet sure how can we query the interconnect type. If you have any pointers, that'd help.

With containers you skip the module problem (even though I am biased towards containers). :)
Not sure for my use-case we do need to detect the programming environment as compilation won't be done.

Fair point. We can have the generic builtin environment that loads nothing. And technically, you could parametrize your ReFrame test to try automatically different containers (e.g., with different compilation setups of the target application).

[vkarak]

I did a small research of different tools today to see how we can retrieve the required information for local nodes at least. archspec does a nice job in detecting the micro-architecture name based on the processor features reported in /proc/cpuinfo. For the topology, the absolute reference is hwloc. There is a Python wrapper which is not actively maintained or developed, so we cannot use it. But hwloc-ls --of xml gives the whole information in XML (why the heck don't they spit JSON!), which we can parse nicely and easily with xmltodict.

This issue needs #1709, since I see it -at least for the moment- as an initial step that would generate a configuration automatically, which would then be used to run the tests. Perhaps, we could also introduce a valid_arch field in the tests that would filter the tests based on the micro-architecture.

[vkarak]

Instead of hwloc we can take the processor topology with lscpu which spits it also in JSON, e.g., lscpu -J -e.

[kniec]

On the scope part.

1. System partitions
   a. Modules system
   b. Local host architecture (micro-arch + topology)
   b. Job schedulers
   c. Parallel launchers
2. Environments
   a. This is a bit challenging. Spack is doing a similar thing in auto-detecting the compilers, so we could use Spack for this, but what about EasyBuild? We could potentially look for module name patterns. But what if the s/w stack is hidden behind a top-level module?
   b. Any environments defined should follow a certain naming convention, so that tests can define valid_prog_environs accordingly. Similar thing should be done for valid_systems.

IMHO I would like to be as lean as possible.
1.b is the most important, maybe 1.a to define how to find the binary or which container image to use.