cosim
Closely related to the co-simulation repository, within the following, a detailed overview of the co-simulation, its functionality and application is provided.
Within the following, the theoretical background of the co-simulation and its main principles will be discussed.
Also see publications:
- F. Gaisbauer, E. Lampen, P. Agethen and E. Rukzio, "Combining heterogeneous digital human simulations: presenting a novel co-simulation approach for incorporating different character animation technologies" in The Visual Computer, Springer, 2020.
- F. Gaisbauer, J. Lehwald, P. Agethen, J. Sues and E. Rukzio, "Proposing a Co-simulation Model for Coupling Heterogeneous Character Animation Systems" , In Proc. of GRAPP 2019 (14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications), Feb. 2019.
The co-simulation is reponsible for executing and merging arbitrary MMUs without posessing semantic knowledge of the motions. Based on a set of input instructions closely related to BML, the co-simulation executes the respective MMUs and provides an amalgamated result. In particular, for each frame the result is provided as MSimulationResult which comprises a merged posture, possible scene manipulations, constraints and events.
In order to simplify the orchestration of multiple MMUs and allow the systematic execution, the MMUs must be accessed and managed in a structured way. In this context, one underlying core principle of the co-simulation is the hiearchical organization of different MMUs.
For the co-simulation the so-called generalization-specialization scheme introduced by Kallmann et al. is employed. As illustrated in the figure below, the MMUs are hiearchically stacked. Motions which affect the full Body (e.g. idle motions, Walking) have a low priority. Other motions with rather affect local Body parts (e.g. grasping, head movement) possess a higher priority.
Within the utilized hierachy, MMUs with higher priority can access the results of the previous MMUs. Moreover, the results can be manipulated or entirely overwritten. If the hierachical execution is strictly applied, the outcome of the last MMU describes the full motion. For instance, if a walking motion is executed in conjunction with a reaching motion, the walk MMU possesses a lower priority than the reach MMU. Both are executed in parallel, whereas the reach motion obtains the computed posture of the walk for each frame. The reach motion consecutively, overwrites the body parts which are actively involved in the reaching process. As outcome, the manipulated posture is provided as result.
The actual workflow is strongly based on the hiearchical execution. However, additional steps are included.
The provided implementation of the co-simulator is utilizing a hierarchical blending approach to transition between MMUs. For more information on the transition model, please consider the respective wiki page transition modeling.
The co-simulator generally operates on instructions formulated using the MInstruction format. Start and end-conditions, as well as the provided events and actions are important concepts of the co-simulation.
The simplest way of adding an instruction which should be executed by the co-simulation is to solely add a single instruction, defined using the MInstruction format. Once the instruction is specified, it can be added using the AssignInstruction method of the co-simulator. Please note, that the instruction assignment only takes place in the do-step routine. Although the AssignInstruction method of the co-simulation is executed, the respective MMU will be executed in the do-step. This is done in order to preserve the hierachy for execution and to establish a coordinated way of executing the instructions.
MSimulationState simulationState;
//Create a first instruction
MInstruction instruction1 = new MInstruction("id1","walk","Locomotion/Walk",...)
{
Properties = ...
}
//The order of instruction assignment doesn't matter
coSimulator.AssignInstruction(instruction1, simulationState);The co-simulation furthermore supports so-called composite instructions, which internally comprise multiple sub-instructions.
MSimulationState simulationState;
//Create a first instruction
MInstruction instruction1 = new MInstruction("id1","walk","Locomotion/Walk",...)
{
Properties = ...
}
//Create a second instruction
MInstruction instruction2 = new MInstruction("id2","grasp", "Pose/Grasp",...)
{
Properties = ...
}
//Create a composite instruction comprising the previously specified ones
MInstruction compositeInstruction= new MInstruction("idNested","composite","NestedType"...)
{
Instructions = new List<MInstruction>(){instruction1, instruction2}
};
//The order of instruction assignment doesn't matter
coSimulator.AssignInstruction(compositeInstruction, simulationState);The co-simulation unpacks the composite instruction and sequentially assigns the instructions in the order of the list. This process is performed recursively which means that deeply nested structures are allowed.
An important concept of the co-simulation are start and end conditions which can be formulated for each instruction. Start conditions generally specify conditions that must be fulfilled in order to start a specific instruction.
//Create a first instruction
MInstruction instruction1 = new MInstruction("id1","walk","Locomotion/Walk"...)
{
Properties = ...
}
//Create a second instruction which should be executed after the first instruction is finished
MInstruction instruction2 = new MInstruction("id2","grasp","Pose/Grasp"...)
{
StartCondition = instruction1.ID "+ ":" + mmiConstants.MSimulationEvent_End
}
//The order of instruction assignment doesn't matter
coSimulator.AssignInstruction(instruction1, ...);
coSimulator.AssignInstruction(instruction2, ...);In the following the gantt chart of the instruction execution is visualized. The co-simulation instantly starts the instruction1. For each frame, the start-condition of instruction2 is verified. The start-condition is true if the instruction1 has raised the mmiConstants.MSimulationEvent_End. Therwith, after instruction1 has raised this event, the validated expression gets true. As a consequence, the co-simulation starts instruction2.
gantt
title Instruction execution using a start-condition
dateFormat YYYY-MM-DD
axisFormat %m
section Section
Instruction1 :a1, 0, 30d
Instruction2 :a2, after a1, 30d
Analogously, end-conditions can be formulated. In this case, the corresponding MMU will be terminated, once the condition is true.
//Create a first instruction
MInstruction instruction1 = new MInstruction("id1","walk","Locomotion/Walk"...)
{
Properties = ...
}
//Create a second instruction which should be executed after the first instruction is finished
MInstruction instruction2 = new MInstruction("id2","grasp", "Pose/Grasp"...)
{
EndCondition= instruction1.ID "+ ":" + mmiConstants.MSimulationEvent_End
}
//The order of instruction assignment doesn't matter
coSimulator.AssignInstruction(instruction1, ...);
coSimulator.AssignInstruction(instruction2, ...);In the illustrated scenario, instruction1 and instruction2 will be started simultanously. Once instruction1 is finished and raises the mmiConstants.MSimulationEvent_End, the co-simulation automatically finishes the second instruction as well.
gantt
title Instruction execution using an end-condition
dateFormat YYYY-MM-DD
axisFormat %m
section Section
Instruction1 :a1, 0, 30d
Instruction2 :a2, 0, 30d
The start and end-conditions are generally represented as strings similar to BML. A condition is represented by a reference to an id (MSimulationEvent ID), as well as an event_type. The event_type can be either one of the defined evnts in mmiConstants, or a custom event.
The individual conditions can be furthermore extended to logical expressions as illustrated below.
//A condition consists of a reference represented as an ID as well as an event_type, both are separated by a :
condition = id:event_type
//A condition can consist of multiple sub-conditions connected by logical operators and brackets &&,||,(,),
condition = [condition (||) | OR | AND | (&&) condition]?
//Example for a specific condition which consists of two sub-conditions
condition = instruction1.ID:mmiConstants.MSimulationEvent_End && instruction2.ID:mmiConstants.MSimulationEvent_StartIn certain cases it is not known in before how the specific sequence of instructions should look like. Therwith, certain instructions must be terminated during runtime, whereas the knowledge is only available at the time when a new instruction is added. For this purpose the so-called actions are utilized. The actions describe a defined set of operations the co-simulation can perform to other instructions if the events are provided.
//Create a first instruction
MInstruction instruction1 = new MInstruction("id1","walk","Locomotion/Walk"...)
{
Properties = ...
}
//Create a second instruction which should be executed after the first instruction is finished
MInstruction instruction2 = new MInstruction("id2","carry","Object/Carry"...)
{
EndCondition= "" //unkown -> we don't know yet, when the carry should be finished
}
//The order of instruction assignment doesn't matter
coSimulator.AssignInstruction(instruction1, ...);
coSimulator.AssignInstruction(instruction2, ...);Somewhen later, a new instruction might be added for releasing the object. At this point in time, the carry should be terminated.
//Create a release instruction
MInstruction instruction3 = new MInstruction("id3","release","Object/Release"...)
{
//Presently the action must be specified using the properties
Properties = new Dictionary<string,string>()
{
//Create an entry in the dictionary with key = CoSimTopic.OnStart and the respective value
{CoSimTopic.OnStart, "instruction2.ID + ":" + CoSimAction.EndInstruction}
}
//Identically, in future the action field can be used for this purpose
//OnStart->8315395235:StartInstruction,OnEnd->3523534:StartInstruction
Action = CoSimTopic.OnStart +"->" + instruction2.ID + ":" + CoSimAction.EndInstruction
}
//Once the instruction is assigned -> The new action is considered and instruction2 will be terminated at startup of instruction3
coSimulator.AssignInstruction(instruction3, ...);A further important aspect of the co-simulation is the specification of the utilized priorities. If using the explicit interface of the MMICoSimulator, the class provides a dedicated function for this purpose.
public void SetPriority(Dictionary<string, float> priorities)Using this function, for each MMU to be considered a key-value pair consisting of the motiontype and the priority must be specified. The higher the priority value, the higher the priority, the later the MMU is executed for each frame.
If the co-simulation is accessed remotely, the interface is identical to a MMU. Therewith, the priorites must be specified using the initialize function.
It is important to note, that the co-simulation only executes given instructions if the CheckPrerequisites() method of the MMU returns true. The co-simulation validates for each frame, whether the function returns true (the validation is only done for those instructions which can be started according to the defined start condition).
Subsequent MMUs are evaluated. Constraints are provided to the previous MMU.
Co-simulation solves the constraints if not satisfied. Arbitrary additional solvers can be defined. The solvers are executed in the merging stage. For each solver it is first checked if the solving is required. If this check is true, the respective solver is executed. The co-simulator allows to add arbitrary solvers using the defined ICoSimulationSolver interface.
/// <summary>
/// Basic interface for a solver which can be utilized for processing the results of MMUs
/// </summary>
public interface ICoSimulationSolver
{
/// <summary>
/// Indicates whether a solving is required.
/// E.g. posture is already solved and satisfies the constraints
/// </summary>
/// <param name="result"></param>
/// <returns></returns>
bool RequiresSolving(MSimulationResult result);
/// <summary>
/// Solve method is responsible for processing the results obtainted by MMUs
/// </summary>
/// <param name="currentResult">The current (merged) result</param>
/// <param name="mmuResults">The raw result data of the mmus</param>
/// <returns>A simulation result which is merged based on the given input </returns>
MSimulationResult Solve(MSimulationResult currentResult, List<MSimulationResult> mmuResults);
}MMICoSimulation is the main project which contains all classes and functions related to the co-simulation. The project is entirely realized in C#. In the following, the most important components are discussed in detail.
The MMICoSimulator is the core class comprising the actual co-simulation. The class inhertis from the so-called IMotionModelUnitAccess. Therewith, the MMICoSimulator provides all methods of a MMU and shares the same interface.
classDiagram
IMotionModelUnitIface<|-- IMotionModelUnitAccess
IMotionModelUnitAccess<|-- MMICoSimulator
The MMICoSimulationMMU is a project, in which, the MMICoSimulator is encapsulated within a MMU. This project can be used for creating nested MMUs, or hosting a standalone co-simulation. As illustrated by the class diagram, the CoSimulationMMU inherits from MMUBase, which is a base class for developing MMUs in C#. The class comprises all functions of the MMU interface.
classDiagram
IMotionModelUnitIface<|-- IMotionModelUnitDev
IMotionModelUnitDev<|-- MMUBase
MMUBase<|-- MMICoSimulationMMU
In particular, the CoSimulationMMU internally hosts an instance of the MMICoSimulator. The CoSimulationMMU is rather short in terms of lines of code, since most functionality is encapsulated.
In the Initialize method, the MMU sets up a connection to the MMI framework and launcher. In this context, the MMus that should be loaded can be specified (see code example).
//Instance of the MMU
MotionModelUnit.Iface coSimulationMMU;
....
//Dictionary containing the properties
Dictionary<string,string> properties = new Dictionary<string,string>();
//Add the motion types that should be loaded (if nothing is defined, all will be loaded) and the corresponding priorities
properties.Add("Pose/Idle",1.ToString());
properties.Add("Locomotion/Walk",1.ToString());
//Initialize the coSimulation with the avatar description and properties
coSimulationMMU.Initialize(avatarDescription, properties);Consecutively, the CoSimulationMMU can be used like a default MMU. Using the AssignInstruction the defined MInstructions are forwared to the respective MMU using the internally incorporated CoSimulation. All events, scene manipulations and posture changes are forwared and provided as result.
//Instance of the MMU
MotionModelUnit.Iface coSimulationMMU;
//The present simulation state of the avatar
MSimulationState simulationState;
//A priori defined idle instruction with motion type Pose/Idle
MInstruction idleInstruction;
//A priori defined walk instruction with motion type Locomotion/Walk
MInstruction walkInstruction;
//A priori defined reachinstruction with motion type Pose/Reach
MInstruction reachInstruction;
//Assigning the instructions is straight forward (identically to a MMU)
//The CoSimulationMMU internally forwards the instruction assignments to the respective MMUs
coSimulationMMU.AssignInstruction(idleInstruction,simulationState);
coSimulationMMU.AssignInstruction(walkInstruction,simulationState);
coSimulationMMU.AssignInstruction(reachInstruction,simulationState);
The MMICoSimulationStandalone project provides a standalone co-simulation which can be used from different programming languages. The project hosts a virtual Adapter which accesses the MMICoSimulationMMU.
In general multiple options. Hosting the co-simulation as standalone application. Debugging of code.
Debugging of results: For debugging the co-simulation, the so-called CoSimulationDebugger is provided for the Unity engine. Allows to playback the motion and analyze each frame in detail including all steps in the hiearchical execution.