QEMU orchestrator: implement support for remote hosts

[anirudhrb]

The current qemu orchestrator treats the local machine as the host for the VMs. Add support to allow specifying a remote
host in the runbook. The remote host should have libvirtd running and configured to allow remote connections. VMs under test are spawned on this remote host.

Example runbook for specifying a remote host

name: qemu default
...<snip>...
platform:
  - type: qemu
    admin_private_key_file: $(admin_private_key_file)
    keep_environment: $(keep_environment)
    qemu:
      hosts:
        - address: "10.77.0.5"
          username: "anirudh"
          private_key_file: $(admin_private_key_file)
    requirement:
      qemu:
        qcow2: $(qcow2)
        cloud_init:
          extra_user_data: $(extra_user_data)

In case host is not specified in the runbook, the local machine is treated as host (existing behavior).

[squirrelsc]

@cwize1 Can you have a look on this change too?

[cwize1]

This doesn't seem like it'll support testing multiple VMs on the same host.

What I recommend you do is allow the user to set the libvirt network that the VM connects to. Then the user can setup their own network bridge on the host, create a libvirt "network" that points to that bridge, and then set tell LISA to use that "network" for the VMs. Then the test runner will be able to access the VMs directly without needing to deal with port forwarding.

[squirrelsc]

We can use the consistent way for local and remote hosts. It will simplify the code logic. The only difference is the localhost doesn't need connection info.

[cwize1]

What do you mean by "the consistent way"?

[anirudhrb]

In case of port forwarding, we do have special logic for remote hosts (set up iptable rules for forwarding). So consistent way is not possible.

[squirrelsc]

Anyway, I think it can make the logic simpler, if the port forwarding or other approaches are used on local too.
If to use port forwarding, one way is to set a start port range from high end like 30000, and map the ports one by one incrementally, if a port is unused.

[anirudhrb]

I need to figure out the bridged network thing. I had tried it at first but it didn't work. I am inclined to doing it in a follow up PR.

Just to understand, in what cases do we need multiple VMs on the same host? When I do implement it how do I test it?

[cwize1]

You can create a simple test function with the following annotation:

@TestCaseMetadata(
    description="",
    requirement=node_requirement(
        node=schema.NodeSpace(
            node_count=2,
        )
    ),
)

There are a bunch of "Microsoft suite" tests that use multiple nodes, I think mainly around testing networking. I am using it to test multi-node Kubernetes clusters.

[squirrelsc]

@cwize1 The field schema.Environment.topology is used to differentiate different network topology, but it's not used so far. The value is always subnet. Please let me know, if you need to test different topology, we can discuss the requirement, and think about how to support it.

[cwize1]

@squirrelsc Unfortunately, unlike with Hyper-V which has a fairly unified networking API, libvirt/QEMU doesn't provide much assistance for networking. Unless you are doing something super simple (e.g. using the default NAT network), then a developer has to manually create the network outside of libvirt and then point the libvirt VM at that network. Typically, either Linux kernel bridges or OVS (Open vSwitch) are used but both have their challenges. OVS is a big piece of software that is complicated to use. And every Linux distro has their own networking manager API that you have to us to provision networks, such as kernel bridges. I don't think it is worth trying to pull that complexity into LISA.

[squirrelsc]

Thank you for the thoughts. IMO, it depends on test scenarios, and doesn't need to fully automate all things. The lab environments is complex, and it's acceptable for preconfigured steps. If some servers are setup with some topologies, LISA just needs to use it, and keep it unchanged. The LISA needs to be aware of the topology, and assign test cases by its requirements. If the different topology tests is needed from a test case, the platform checks the settings (from runbook) of each server, and then allocate VMs from matched servers. The ADO agents supports it by "capabilities". The capabilities just a couple of key/value pairs, it's very similar like what I want to do.

[cwize1]

.

[cwize1]

.

[squirrelsc]

BTW, please update document for the new schema.

[cwize1]

.

[squirrelsc]

1. the is_remote is incorrect here, and always be True, because it's the method itself. The is_remote() is right.
2. I'm not sure if the "remote" check is useful here. Can remote and local use the same folder structure?

[anirudhrb]

1. is_remote is correct based on usages in other files. is_remote() throws the error "bool is not callable".
2. This is an optimization. We avoid copying OS disk image for local node. That's why remote check is useful here.

[cwize1]

is_remote is marked with @property. So, it behaves like a C# style property.

[squirrelsc]

Sorry, I'm confused on the LibVirtHost, I thought its type is LibVirtHost. BTW, what's the reason it needs a LibVirtHost schema? Can it reuse the RemoteNode schema?

[anirudhrb]

I think there is some utility in having a separate schema because we could have libvirt specific fields in it. Right now we have the lisa_working_dir property that is not part of RemoteNode. In the future we could have a supported_hypervisors property for example to indicate the hypervisors (qemu, ch etc) that the host supports.

[squirrelsc]

The Qemu can inherit the RemoteNode, so it's easy to obtain the general supports of node. The Node has working_path, and it can be overwritten to read from the schema.

[cwize1]

.

[cwize1]

Is it possible to leave the default as the base VM image's directory for the local case?

[anirudhrb]

I guess it could be done if needed. Is it for backward compatibility?

[cwize1]

When running on a local dev box, it is nice to be able to easily see all the files created, particularly for cases where you keep the environment after the test completes/fails (e.g. while debugging).

[cwize1]

.