Expand reframe core code for VM and cloud integration

[ireed]

Reframe in its current form cannot be used for cloud clusters without additional edits to the core API files, due to fundamental differences between VM vs baremetal behavior. This causes issues with both creating and maintaining cloud-based tests using Reframe.

The issue from a SW dev perspective is VMs needing to 'ask' the cloud provider for its specs. For example, because not all cores and HW on a physical machine are allocated to a cloud node, certain commands such as lscpu or checking locality/affinity can show infothat is misleading or incorrect. Getting this info in Azure VMs require a server ping that returns the HW/VM info.

The issue from a SW management perspective is cloud users needing to maintain a special branch of Reframe that has cloud-specific logic - as @vkarak pointed out in #2444. This means not being able to access the latest Reframe code without doing regular merges, which is not good for automated or regression testing.

There are a few ways I can think of to handle both issues. None are ideal.

• A slight variation of #2440. Introduce new cloud-specific functions that are equivalent to regular reframe functions (ex: self.current_system_VM.node_data) with the same behavior as the baremetal counterparts. But, different clouds likely have different behaviors - and, clouds themselves are not necessarily static, unlike baremetal OS - making this is option a stability, validation, and compatibility nightmare.
• Add conditionals inside existing reframe functions, similar to how reframe has different logic for various schedulers. But, that may require adding support for only one VM/cloud at a time, with each provider needing to wait their turn.
• Put VM function calls in various parts of the configuration that cloud users can fill in a separate, stand-alone python file. For example, with the above referenced _detect_system() in config.py you can include a sub-function _detect_vm() that runs before any _detect_system() code and is empty by default. Cloud users fill in _detect_vm() in a separate file - cloud_config.py - with desired behavior. This would prevent the need for Reframe to account for different VM/cloud usage scenarios. And, automated cloud tests can pull the main reframe code, then pull their additional file(s) when building/initializing the environment. The downside is outsourcing reframe development to cloud devs (basically, having mini-versions of reframe for every cloud provider), causing reframe devs to [rightfully] decline customer support from any cloud user.

Thoughts?

[vkarak]

I've looked into the implementation in your fork and I think it's towards the right direction. I see two components there: one is talking to the cloud-specific API to get the instance information and the second one is how to construct the system name. Both are very specific, either to the cloud provider or the user. Even if ReFrame provided the cloud-specific part, what would be used to create the system name from all the information available about the instance?

I think a viable solution would be to allow users to specify a virtual "hostname" program through an environment variable, configuration option or command line argument. That would be an executable taking no arguments and returning a single string. ReFrame will treat the returned string as if it were the hostname and then proceed matching it to get the right system configuration. That not only integrates well with the rest of the logic, but it also allows to have configurations for families of instances by matching specific parts of this virtual hostname. What do you think?

[ireed]

The short answer to your question is a hostname program would be completely acceptable for this - either with cloud providers maintaining their own, or Reframe having a file or function for each cloud provider. But, it may be possible that we will need other info from the curl command (besides vmSize) for future work.

To decipher the code segment for the system name:

The curl lookup gives info for the expected system specs. For example, if the curl command returns vmSize = 'Standard_NV36ads_A10_v5' then we know the type and number of gpus, IB cards, and other things to expect on the node.

This info lets us map the node to a 'family' of VMs (NVadsA10 v5-series, in this example) by using other info from the curl command: sysname = offer+sku+version = nvads+a10+v5. We use this in our config file, which allows us to organize VMs into groups such as an AMD cpu partition, IB partition, etc that we use for self.valid_systems. Otherwise, we cant know which nodes can run, for example, a cuda workload on an AMD cpu without doing a long series of query commands (ex: nvidia-smi, lspci, etc) on every node - and that assumes no issues with HW, SW, or hypervisor.

Although we need both the vmSize and family, i believe the Reframe code only needs the family. We can just do a 2nd query for vmSize in our tests.

Note that we are not Reframe experts and are very much open to suggestions on better ways to do this for easier integration.

[vkarak]

That's interesting because it opens a new trajectory, that of somehow automatically generating a system config. This has come up in community discussions in the past. Have you experimented with the extended syntax for valid_systems and valid_prog_environs ? This is a relatively new feature (version 3.11 onward) and basically alleviates you from the task of having to precisely encode your system characteristics in its name in order to properly include it in a test. Your test could define valid_systems = ['%family=Standard_NV36ads_A10_v5'] and be valid for any system defining this in its extras. Similarly, it could define +nvidia-smi +lspci and be valid for any system that defines nvidia-smi and lspci in its features. That effectively decouples the tests from the system naming scheme and the only requirement is that of establishing an convention for known features and properties. Then the question of this issue becomes broader: could we have reframe generate automatically a set of system features like we do for the processor topology? It could be at this interface that the cloud integration makes more sense.

[ireed]

I have not used the extended syntax, but after looking over it, i can see that it solves a specific problem we have. To use an example of running a gpu test, we hard code every VM family that has a gpu (ex: valid_systems = [a100v3, a100v4, a10v4, etc]). But, cloud providers regularly add new VMs and deprecate older ones, so im sure you can see the problem with this approach. It sounds like i can just include feature=gpu in the config file for each gpu family, or even something like feature=openCL. This is immensely useful for cloud, if my interpretation is correct. The catch - at the moment - is needing the updated reframe code for this feature, which further supports my original post of wanting to avoid maintaining a separate fork for cloud.

Im not sure how Reframe would do automatic system features. In our case, even the info from the curl server ping doesnt tell us this info, which is why we have the large json file of all VM families. We know the system features only after matching the sysname (derived from the curl ping info) to the entry in the database.