Enable broker to destroy a VM and return the list of unfinished Cloudlets

[pkoperek]

Detailed information about how the feature should work

If I understand correctly, while executing the simulation, a VM can be only removed by calling DatacenterBrokerAbstract.requestIdleVmDestruction. This requires the VM to not have any Cloudlet on the exec list.

To enable more rich interaction with simulation environment, I would like to be able to destroy a VM (even if it is currently processing Cloudlets). The cloudlets could be either cancelled (and the caller would have to manually re-submit them) or could be re-scheduled on another VM internally.

Tbh I'm not sure what would be the best solution here.

An example scenario where this feature should be used

I would like to be able to deploy an autonomous agent (which is supposed to manage the resources of the cloud) - in the simplest scenario it could just adjust the number of VMs running.

A brief explanation of why you think this feature is useful

This would be very useful for implementation of a simulation environment for autonomous agents, in which they could interact with the environment directly.

Examples Available

• SynchronousSimulationDestroyVmExample1.java

[manoelcampos]

The DatacenterBrokerAbstract.requestIdleVmDestruction is a way to automatically destroy idle VMs, but you can call the host.destroyVm(vm) method to force VM destruction.
The thing is when to call that method to accomplish what you want.
For the method you mentioned, we have a defined condition to request VM destruction.
For your scenario, I can't see an explicit condition.

There may be different places where you can request the destruction of VMs.
You could extend DatacenterBrokerSimple and override the submitVmList() method to check the conditions you want and then destroy the desired VMs by calling host.destroyVm(vm).

Give me more details and I try to think about possible approaches.

[pkoperek]

Sure! The idea is to start simulation in sync mode and process it in a following loop:

while(simulation.isRunning()) {
    simulation.runFor(1.0);
    // gather metrics from the simulation, e.g. the number of VMs being run
    if(vmsCount > 100) {
       List<Vm> idleVms = getIdleVms(simulation);
       Vm toDestroy;
       if(idleVms.size() > 0) {
          toDestroy = idleVms.get(0);
       } else {
           int randomIdToDestroy = ...;
           toDestroy = getVmById(simulation, randomIdToDestroy);
       }
       toDestroy.getHost().destroyVm(toDestroy);
}

My concern is that if I call destroyVm on a Vm which is still executing cloudlets, I don't know what would happen to them (I'm guessing that they would just "hang" and never finish).

[manoelcampos]

If you destroy a VM with running Cloudlets, that is what is expected to happen.
Unfinished Cloudlets will probably remain in the cloudlet execution list and won't be moved to the finished list. But I don't think the state of such Cloudlets is changed. You need to create some simulation to check that.

Migrating Cloudlets (apps) to another VM is not a realistic task. We may migrate containers, not apps, since a container has everything inside (configs, dependencies, etc) and are managed as a black box, such as VMs.

Maybe the proper way is to resubmit the Cloudlets to another VMs.

[pkoperek]

@manoelcampos
Just pushed #211 - please have a look and let me know what you think.
Thanks!

[pkoperek]

I think there is some room for optimization: e.g. instead of cloudletsSubmittedList there probably should be a hash map (so removing/lookup operations are more efficient).

[manoelcampos]

Hello @pkoperek

Thanks for your contribution. I fixed lots of performance issues but for sure there is room for improvement. About the cloudletSubmittedList, the simulation doesn't loop up for individual objects there, so searching is not an issue. Every time a Cloudlet is submitted, it is added to this list. Therefore, add is a concern (mainly because it's used an ArrayList which has an expensive resize operation). In this case, a LinkedList would suffice and keep the interface the same.

Instead of looking up for elements into the list, in rare cases (such as when parsing traces from the Google Cluster Data) the simulation just needs to traverse this list. I think that is supposed to be faster in a LinkedList than in a HashMap.

[pkoperek]

Thanks!

My concern (and idea :)) comes from a scenario with ~60k cloudlets where they were almost constantly rescheduled (a vm having a lot of cloudlets scheduled would get killed). I noticed that one of the hotspots was caused by using a list here (e.g. this forces you to iterate over it to find out which elements should be removed).
After I fix the changes you requested I'll have a look again and try to prepare an example.