Network and migrations scheduling

Network and migrations scheduling

Associated source code

In this tutorial, we present how to create basic and custom network views and how to define VMs memory usage parameters to let BtrPlace estimate precisely the migrations duration.

On the sample associated to this tutorial, BtrPlace computes 3 times a reconfiguration plan based on the same decommissioning scenario but with different networking configurations that lead to very different actions schedules.

Create an environment (nodes + VMs)

Start to create a default model

    // New default model
    Model mo = new DefaultModel();
    Mapping ma = mo.getMapping();

Create nodes and set them online

For this example, we create 3 online nodes: 2 source and 1 destination nodes:

    // Create and boot 2 source nodes and 1 destination node
    Node srcNode1 = mo.newNode(), srcNode2 = mo.newNode(), dstNode = mo.newNode();
    ma.addOnlineNode(srcNode1);
    ma.addOnlineNode(srcNode2);
    ma.addOnlineNode(dstNode);

Create and host some VMs on nodes

We create 4 running VMs hosted 2 by 2 on both source nodes:

    // Create 4 VMs and host 2 VMs on each source node
    VM vm1 = mo.newVM(), vm2 = mo.newVM(), vm3 = mo.newVM(), vm4 = mo.newVM();
    ma.addRunningVM(vm1, srcNode1);
    ma.addRunningVM(vm2, srcNode1);
    ma.addRunningVM(vm3, srcNode2);
    ma.addRunningVM(vm4, srcNode2);

Set VMs' memory activity attributes

The VMs vm1 and vm4 are idle VM (very low memory activity) but still consume some memory, the parameter 'memUsed' must be set and represents the real amount of memory used by a VM (in MiB):

    mo.getAttributes().put(vm1, "memUsed", 2000); // 2 GiB
    mo.getAttributes().put(vm4, "memUsed", 2200); // 2.2 GiB

The VMs vm2 and vm3 execute a workload, the parameters 'hotDirtySize', 'hotDirtyDuration', and 'coldDirtyRate' (that represent the memory activity of the VMs) must be set. The following values represent a workload equivalent to stress --vm 1000 --bytes 50K (a thousand of threads that continuously allocate and release 50KiB of memory each):

    mo.getAttributes().put(vm2, "memUsed", 8000); // 8 GiB
    mo.getAttributes().put(vm2, "hotDirtySize", 56); // 56 MiB
    mo.getAttributes().put(vm2, "hotDirtyDuration", 2); // 2 sec.
    mo.getAttributes().put(vm2, "coldDirtyRate", 22.6); // 22.6 MiB/sec.

    mo.getAttributes().put(vm3, "memUsed", 7500); // 7.5 GiB
    mo.getAttributes().put(vm2, "hotDirtySize", 56); // 56 MiB
    mo.getAttributes().put(vm2, "hotDirtyDuration", 2); // 2 sec.
    mo.getAttributes().put(vm2, "coldDirtyRate", 22.6); // 22.6 MiB/sec.

These parameters correspond to:

• hotDirtySize: The total size in MiB of the VM's memory pages that are rewritten very quickly.
• hotDirtyDuration: The duration necessary to generate all the 'hot pages' represented by hotDirtySize.
• coldDirtyRate: Represent the supposed linear progression of dirty memory pages after the generation of the hot pages, in MiB/sec.

For a full understanding of these parameters, basically how they can be retrieved and how they are used to compute the migrations duration, please have a look at this nice paper.

Add placement constraints to trigger migrations

We add two Offline constraints, one for each source node, to force the VMs to migrate on the destination node:

    List<SatConstraint> cstrs = new ArrayList<>();

    // We want to shutdown the source nodes to force all hosted VMs to migrate to the destination node
    cstrs.add(new Offline(srcNode1));
    cstrs.add(new Offline(srcNode2));

Try to solve as is

    try {
        ReconfigurationPlan p = new DefaultChocoScheduler().solve(mo, cstrs); // Solve
        System.out.println(p); // Show the computed plan
        System.out.flush();
    } catch (SchedulerException e) {
        e.printStackTrace();
    }

There is list of actions scheduled:

• 0:1 {action=migrate(vm=vm#2, from=node#1, to=node#2, bw=2147483647)}
• 0:1 {action=migrate(vm=vm#1, from=node#0, to=node#2, bw=2147483647)}
• 0:1 {action=migrate(vm=vm#0, from=node#0, to=node#2, bw=2147483647)}
• 0:1 {action=migrate(vm=vm#3, from=node#1, to=node#2, bw=2147483647)}
• 1:2 {action=shutdown(node=node#0)}
• 1:2 {action=shutdown(node=node#1)}

We can see that the migrations are scheduled in parallel during the first second [0:1] and then the sources nodes are shutdown simultaneously in the next second [1:2].

This is explained by the lack of networking knowledge of BtrPlace. First, the migrations bandwidths are set to Integer.MAX_VALUE (bw=2147483647) resulting to very short (1 sec.) migrations duration. Second, the network topology is not known letting BtrPlace schedule them all in parallel.

Set a default network view

The Network class provides static methods to create 'default' network views. It basically consists to create a main non-blocking switch and to connect all the existing nodes to it. The following example connect the nodes using 1 Gbit/sec. links:

    // Connect all nodes to a non-blocking switch using 1 Gbit/s links
    Network.createDefaultNetwork(mo, 1000);

This time the resulting plan is:

• 0:85 {action=migrate(vm=vm#2, from=node#1, to=node#2, bw=1000)}
• 85:105 {action=migrate(vm=vm#3, from=node#1, to=node#2, bw=1000)}
• 105:106 {action=shutdown(node=node#1)}
• 105:123 {action=migrate(vm=vm#0, from=node#0, to=node#2, bw=1000)}
• 123:214 {action=migrate(vm=vm#1, from=node#0, to=node#2, bw=1000)}
• 214:215 {action=shutdown(node=node#0)}

As we can see, by using the default network view, BtrPlace is able to compute precise migrations duration based on a realistic allocated bandwidth of 1 Gbit/sec. (bw=1000) per migration.

With the knowledge of the network topology, BtrPlace decides to schedule the migrations sequentially to avoid concurrent migrations on the destination link. Indeed, as all links have the same bandwidth and the destination node has a single link (to receive all migrations), concurrent migrations will cause slowdowns due to VMs' memory dirty pages rate behavior.

The migrations order is however chosen randomly as it doesn't affects the total duration of the plan (that BtrPlace tries to minimize).

Also, each source node is turned off as soon as possible.

Set a custom network view

We first create a new network view and attach it to the model:

    // Create and attach a new network view
    Network net = new Network();
    mo.attach(net);

Then, we create a non-blocking switch:

    Switch swMain = net.newSwitch();

And we finally connect the nodes to the switch. In this example, the source nodes are connected using 1 Gbit/sec. links while the destination node has a 10 Gbit/sec. connection.

    // Connect all nodes to the switch using different bandwidths
    net.connect(1000, swMain, srcNode1, srcNode2); // 1 Gbit/sec.
    net.connect(10000, swMain, dstNode); // 10 Gbit/sec.

There is the computed plan:

• 0:18 {action=migrate(vm=vm#0, from=node#0, to=node#2, bw=1000)}
• 0:85 {action=migrate(vm=vm#2, from=node#1, to=node#2, bw=1000)}
• 18:109 {action=migrate(vm=vm#1, from=node#0, to=node#2, bw=1000)}
• 85:105 {action=migrate(vm=vm#3, from=node#1, to=node#2, bw=1000)}
• 105:106 {action=shutdown(node=node#1)}
• 109:110 {action=shutdown(node=node#0)}

Now the VMs are migrated 2 by 2 to make a better use of the 10 Gbit/sec. destination link.

You should have noticed that concurrent migrations always come from different sources nodes. Indeed, VMs from a source node are migrated sequentially to keep a 1 Gbit/sec. bandwidth for each migration and avoid slowdown (sequential intra-node scheduling VS parallel inter-node scheduling).

To go further with migrations scheduling, see the advanced migration scheduling howto.