[RFC]V2V automate design

[bzwei]

Overview

This is a followup design for the V2V project described in #36. This design focuses on the automate workflow. New MiqRequest and MiqRequstTask types will be created for this purpose.

Migration Plan

We will introduce a new model ServiceTemplateTransformationPlan, initially a subclsss of ServiceTemplate. In the future we may make the Migration Plan model based on a dedicated table.
A migration plan has one infrastructure mapping (model TransformationMapping)
A migration plan has many VMs. Each VM selection is represented by model ServiceResource.
A migration plan can be ordered multiple times. Each order is called a migration request(model MigrationPlanProvisionRequest)

Sample API model representation of a migration plan can be found in ManageIQ/manageiq-api#313

The overview of migration plan gives a list of VMs and each VM’s migration status. When a VM is added to the plan its initial migration_status is planned. Then the status can move to approved, started, completed/failed.

Migration Request

A migration request (model ServiceTemplateTransformationPlanRequest, a subclass of MiqRequest) is created each time when a migration plan is ordered. The request will go through an approval process. The process has a policy to choose a subset of VMs in the plan and turn the migration_status to approved. Any completed VMs will be skipped.

For each approved VM a task (model ServiceTemplateTransformationPlanTask, a subclass of MiqRequestTask) is created. The task has status attribute and options field to capture details of the migration.

Sample API model representation of a migration request

{
   "id":"1",
   "description":"Transform VMs",
   "approval_state":"approved",
   "type":"ServiceTemplateTransformationPlanRequest",
   "created_on":"2018-02-15T16:51:29Z",
   "updated_on":"2018-02-15T16:52:19Z",
   "fulfilled_on":"2018-02-15T16:52:19Z",
   "request_state":"finished",
   "message":"Ok",
   "status":"Ok",
   "options":{  
      "delivered_on":"2017-07-15T16:51:45.323Z",
      "user_message":"Ok"
   },
   "miq_request_tasks":[  
      {  
         "description":"Transform VM abc",
         "state":"finished",
         "options":{  
            "progress":{  
               "vm_migrate":"100%",
               "disk_migrate":"100%",
               "message":"Ok"
            },
            "delivered_on":"2018-02-15T16:51:45.323Z"
         },
         "status":"Ok",
         "type":"ServiceTemplateTransformationPlanTask",
         "source_id":"10000000000003",
         "source_type":"VmOrTemplate",
      }
   ]
}

Automate Workflow

Approval

After a migration request is created, it first needs to go through the approval process. Initially we can have it auto approved. The auto approval process will do some filtering to select qualified VMs in the tasks. We can also easily set it up to require a manual final approval.
Auto approval is done by a state machine. For each VM It should take the consideration of

1. Is the VM already migrated?
2. Is the infrastructure mapping’s destination still valid?
3. Is there enough bandwidth for the VM migration?
4. Can the VM migration completed in the time window?
5. Other constraints.

Automate service model MiqAeServiceServiceTemplateTransformationPlanRequest exposes a few methods:

1. source_vms # all VMs that have not been migrated in the plan
2. validate_vm(vm) # validate all resources of the VM have valid destination
3. migration_destination(resource) # get the destination for VM’s cluster, network, or storage, useful to calculate the destination capacity
4. approve_vm(vm) # approves a VM so that it can be migrated in current request

VM migration

After the approval process, for each approved VM a service_template_transformation_plan_task is created and delivered to the v2v migration state machine. Automate service model MiqAeServiceServiceTemplateTransformationPlanTask exposes a few methods:

1. source # selected VM
2. transformation_destination(resource) # get the destination for VM’s cluster, network, or storage
3. update_transformation_progress(progress) # progress is a hash, reports the percentage, disk amount that has been completed etc.

The execution of the migration request does not need to create any service objects, therefore not shown in My Services page.

New automate classes, instances, and methods

New System/Policy instances to handle miq_events

• ServiceTemplateTransformationPlanRequest_created
• ServiceTemplateTransformationPlanRequest_approved
• ServiceTemplateTransformationPlanRequest_denied
• ServiceTemplateTransformationPlanRequest_pending
• ServiceTemplateTransformationPlanRequest_starting

New namespace Transformation and its child folders

New class Transformation/StateMachines/TransformationPlanRequestApproval

• instance Default
• method approve_request
• method pending_request
• method validate_request

New class Transformation/Email

• instance TransformationPlanRequest_Approved
• method TransformationPlanRequest_Approved
• instance TransformationPlanRequest_Denied
• method TransformationPlanRequest_Denied
• instance TransformationPlanRequest_Pending
• method TransformationPlanRequest_Pending
• instance TransformationPlan_Complete
• method TransformationPlan_Complete

[bzwei]

@gmcculloug @EvenIT @pemcg @fdupont-redhat

[pemcg]

I wonder whether we should have a single request per plan, that then translates into a task per VM if the request is approved. This RFC implies that that there's a request per VM (there may be 100 VMs in the plan)?

[ghost]

@bzwei, in my mind, the infrastructure mapping is associated to the VM, not the migration plan. For instance, a VM could match multiple infrastructure mappings, because its resources are mapped in different mappings. That's probably a corner case, but it might be difficult to change later.

At first, we will probably simplify the UX by making it look like the infrastructure mapping is associated to the migration plan. In this case, all the VMs would have the same infrastructure mapping.

I would also see an additional association for the VM object: eligible_transformation_mappings.

[ghost]

@pemcg, That's not how I read it. What I understand is that there can be multiple requests per migration plan. Each request would address a subset of VMs, using a filtering policy that sets the VMs attribute migration_status to approved. Then, the actual request runner would create a task per VM that has the migration_status set to approved. The request would be a way to group the VMs of the plan for execution.

But I might read it wrong...

[bzwei]

@pemcg We have mainly one request per plan, not one request per VM. @fdupont-redhat according to the design in Westford and the UI mockup, we allow only one mapping per plan. The mapping includes all resources used by the VMs in the plan. A plan cannot be created if no mapping satisfies the need for all VMs.

Another (or more) request can be created for the plan, it is only needed if the first one did not migrate all VMs.

[bthurber]

Since we can create multiple Infrastructure Mappings that contain VMs to migrate, and we create the Migration Plan based on the Infra Mapping, the request should be by plan.

[ghost]

@bzwei, looking at the UI mockups, it appears that the infrastructure mapping is the first thing we select when creating a migration plan. The VMs are added to the plan in a further part of the wizard. So, it means that all the VMs have to be compatible with the mapping: source cluster/networks/storages. But to me, it is a UI trick to make the UX simpler during the first iteration.

An infrastructure mapping is a collection of cluster, network and storage mappings that make sense together for the Infrastructure Admin. To me, it is similar to the placement information we have the provisioning request: a VM has eligible mappings in migration context, like it has eligible clusters/hosts/... in provisioning context. Infrastructure mapping can be filtered by RBAC, but nothing prevents a VM from having more than one eligible infrastructure mapping.

My personal feeling is that the infrastructure mapping is associated to the VM, not the migratio plan. And if we select a single infrastructure mapping when we create a migration plan, it simply means that we assign the same infrastructure mapping to all the VMs of the plan. And if later we want to provide a way to select an infrastructure mapping per VM, we will not have to do any change on the object model, while being still compatible with the current UI/UX approach.

@bthurber, the migration plan is a service templatesubclass because it is a definition of something to execute. The actual execution is done through a request. Bill's approach allows to create policies so that the request behaviour is modified when launched. And you can trigger the same template multiple times, for example to migrate the VMs that failed the first time (that's a policy). Then single VM migration is a request task, to allow better parallelization. We could even add more logic to it by creating advanced rules to orchestrate the tasks of a requests: (anti-)affinity, priority, etc...

[bzwei]

@fdupont-redhat We should not need to maintain a one-vm-to-many-mappings relationship. VM migration is a one time operation through a plan.

[fabiendupont]

@bzwei I do agree. The `transformation_mapping` association points to a single mapping. But, the `eligible_transformation_mappings` is an array of mappings that are compatible with the VM hardware (cluster/networks/storages). Le 1 févr. 2018 16:23, "Bill Wei" <notifications@github.com> a écrit : @fdupont-redhat <https://github.com/fdupont-redhat> We should not need to maintain a one-vm-to-many-mappings relationship. VM migration is a one time operation through a plan. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABk59FRLaK006_uHETAUfInHphUXwi1sks5tQdbhgaJpZM4RtmIJ> .

[bthurber]

This can be closed as we have completed dev past this point.