Integration in UE4

Integration in UE4

The MOSIM Plugin for Unreal establishes a connection between the MOSIM framework and the Unreal Engine. It allows thus the visualization of MOSIM generated movements and the execution of behaviors defined inside MOSIM with Characters in the Unreal Engine.

Installation Procedure

• The plugin was developed for Unreal Engine version 4.24.3, other versions have not been tested so far
• The Plugin is located in the directory “MOSIMPlugin” inside the repository
• To integrate the Plugin in your project, you have two choices
  • Integrate as Level Plugin: Copy the whole Plugin-Folder into <Your_Project>/Plugins
  • Integrate as Engine Plugin: Copy the whole Plugin-Folder into <Engine_Path>/Engine/Plugins Either way, you need to recompile your Engine or Project afterwards. Detailed descriptions with screenshots are also available in the README.md within the repository.

Utilizing the Plugin

The deploy.bat script does not include all relevant components needed for the Unreal Engine setup. In addition, please do the following:

• compile MOSIM\Core\CoSimulation\MMICoSimulation.sln
• replace MOSIM\Core\CoSimulation\CoSimulationStandalone\bin\Release\description.json with MOSIM\Core\CoSimulation\CoSimulationStandalone\description.json
• copy the contents ofMOSIM\Core\CoSimulation\CoSimulationStandalone\bin\Release into MOSIM\build\Environment\Adapters\CoSimulationStandalone
• The file structure of the Environment should contain at least the following:

• Adapters
  • CSharpAdapter
  • CoSimulationStandalone
  • UnityAdapter
• Launcher
• MMUs
• Services
  • RetargetingService
  • StandaloneSkeletonAccess

• When the Unreal Engine is started, enable the „Developers Content“ in the Content Browser to use the additional MOSIM components for Unreal
• The MOSIM Plugin includes at the moment 7 different components

MMIAvatar

• This class has to be applied for the MOSIM Avatar
• Inherits from the Unreal class “APawn”
• Is able to execute MOSIM generated MInstructions and can thus visualize the motions controlled by the MMUs
• The following options are available in the details panel:
  • Mesh -> Skeletal Mesh: Select the 3D model for your avatar
  • MOSIM Properties -> Reference Posture File: Select *.mos file neccessary for the retargeting service. The plugin automatically searches through the project content and assigns the first file it finds. Please check.

MMISceneObject

• Class that allows the user to mark objects (AActors) in the scene as MMISceneObjects and therefore as MSceneObjects
• Are thus made known to the MOSIM framework
• Inherits from USceneComponent
• Different Options are available for selecting the object type, the ones currently important are:
  • Walk Target
    • Apply this option for objects which are applied as walk target
    • No colliders are transferred to MOSIM
    • Otherwise the path planning service will find no path
  • Tool
    • Apply this option for objects which are carried by the Avatar
    • No colliders are transferred to MOSIM and gravity is enabled for the object
    • Otherwise the Avatar is not able to grasp the object and the object will start to oscillate as soon as it is carried

Simulation Controller

• Central instance that orchestrates the simulation
• Is realized as bodiless AActor, as the simulation controller needs access to the scene
• Spawn the simulation controller somewhere in the scene

AjanAgent

• UActorComponent that allows the user to connect an Actor (e.g. an MMIAvatar) with Ajan (has to be added as component to the MMIAvatar)
• The Ajan Agent component connects itself via Http Post requests to Ajan when “begin play” is executed
• The following options are available in the details panel:
  • Ajan Agent Template Type
    • Dropdown list for selecting the type of the Ajan Agent template, displays the agent templates that are available in Ajan
    • Will be populated after the simulation was started
  • AJANExecute
    • Defines the endpoint of the execute command in Ajan
  • Template Repo URI
    • URI for the Ajan Template Repo
  • Knowledge Repo URI
    • URI for the Ajan Knowledge Repo
  • Delete Ajan Agent Instance at the Simulation Stop
    • When checked, the Agent instance is automatically deleted in Ajan when the Unreal simulation terminates
  • Ajan Actions
    • Functions controlling the interaction of Ajan with the Unreal Scene
    • Except “Synchronize MScene Objects” all functions require a working thrift connection between Unreal and Ajan, thus the simulation has to be started first.
  • Create Agent
    • Creates the Ajan Agent for the current MMIAvatar. Select the correct Ajan Agent Template (in Ajan Agent Template Type) first.
  • Delete Agent
    • Delete the instance of the Ajan Agent
    • is executed automatically if “Delete Ajan Agent Instance at the Simulation Stop” is activated
  • Execute Agent
    • Executes the Ajan Agent and thus the corresponding Behavior Tree for the selected MMIAvatar (Create Agent has to be executed beforehand)
  • Synchronize MScene Objects
    • Load all MSceneObjects in the scene into the Knowledge Repository defined with the Knowledge Repo URI

Avatar Behavior

• Helper class, that allows the executed instructions for MMIAvatars in the scene without a connection to Ajan
• Realized as bodiless AActor, needs access to the scene, has to be spawned in the scene
• Will be unnecessary, as soon as several MMIAvatars can be executed by Ajan
• Usage (at the moment it is not guaranteed, that all functionality works correctly):
  • When the simulation starts and MMIAvatars are in the scene that do not have an AjanAgent component, the plugin will stop and assign the actors one after another to the AvatarBehavior class and will ask for instructions. The user has to assign then one of the available instructions to the MMIAvatar.
• The details panel of the Avatar Behavior has the following options:
  • Walk Target Name
    • Target, the MMIAvatar will walk to in case an instruction containing a walk motion is assigned
  • Reach Target Name
    • Object, that the MMIAvatar reaches for when a reach or carry motion is contained in the selected instruction
  • Carry to Target Name
    • Target, to which the “Reach Target” is carried to in case a carry instruction is contained in the assigned instruction
  • Walk Target Carry to Target
    • Target, to which the MMIAvatar walks, when a carry motion is contained in the assigned instruction
  • Avatar Name
    • Name of the MMIAvatar, that is assigned to the AvatarBehavior class
  • Assign Avatar
    • Assigns Avatar with the “Avatar Name”
  • The following series of instructions (behaviors) can be assigned
    • Assign Carry Instruction
    • Assign Carry To Target Instruction
    • Assign Idle Instruction
    • Assign Reach Instruction
    • Assign Walk and Idle instruction
    • Assign Walk Instruction
  • Stop Instruction
    • Will stop the current instruction

MMIArea

• Class derived form UActorComponent
• Detects overlapping colliders from objects marked with the component and informs Ajan
• Unrequired and outdated, has not to be utilized any more

The Unreal Plugin for MOSIM Architecture

SimulationController

The SimulationController orchestrates the complete simulation. In the following, the most important functionality of the SimulationController is explained in a chronological order.

SimulationController::SimulationController() In the constructor, the SimulationController instantiates the UnrealSceneAccess. The Unreal Scene Access is the central instance, where all MMISceneObjects and MMIAvatars are registered and information about the scene is stored, that is transferred to the MOSIM framework.

SimulationController::BeginPlay()/SimulationController::Setup() When the simulation is executed in the Engine, “SimulationController::Setup()” is called. This method registers all MMIAvatars and MMISceneObjects in the scene at the UnrealSceneAccess via “SimulationController::RegisterAllAvatarsAndObjects()”. Pointers to the respective objects are stored in a map in the UnrealSceneAccess using each Objects UUID as key. Furthermore, the “RemoteSceneAccessServer” is started, a Thrift server, that allows Ajan to access the scene during the simulation. The final initialization of the MMISceneObjects is triggered in the “SimulationController::Setup()” method. Thus is done here, as not all settings of the MMISceneObjects are defined when they are instantiated, e.g. if the collider is transferred or not or if gravity is enabled. In addition, the “AMMIAvatar::Setup()” method is executed. There, the MMIAvatars connect to the MOSIM framework, load and initialize the “CoSimulationMMU”, connect to the “Retargeting” and the “SkeletonAccess” services and set up the retargeting including the initial posture. If the MMIAvatars contain an AjanAgent component, “SimulationController::Setup()” ends, otherwise it looks for an “AvatarBehavior” instance in the scene and will ask the user to assign instructions to the current MMIAvatar.

SimulationController::Tick()/SimulationController::DoStep() For each tick of the engine, “SimulationController::DoStep()” is executed. In this method, the information about the scene (stored in the UnrealSceneAccess as MSceneUpdate) is pushed into the Adapters. The current posture of the MMIAvatar is read via “AMMIAvatar::ReadCurrentPosture()” and passed to the CoSimulationMMU via a DoStep() querry (“MotionModelUnitAccess::DoStep()”) and passed to the RemoteSkeletonAccess via “SkeletonAccess::SetChannelData()”. The returning MSimulationResult of the DoSetp() querry to the CoSimulationMMU is passed to the different Avatars and the respective “MPostureValues” are assigned via “AMMIAvtar::ApplyPostureValues()”. Furthermore, the manipulations of the MMISceneObjects in the scene are stored in the “UnrealSceneAccess” with “UnrealSceneAccess::ApplyManipulations()”. Unlike for the AMMIAvatar, where all updates are already assigned in the SimulationController during the Tick, the MMISceneObjects retrieve the MOSIM based updates from the “UnrealSceneAccess”.

UnrealSceneAccess

This class stores and manages the Unreal Scene as it is transferred to the MOSIM framework. Furthermore, it provides the necessary functionality for the “RemoteSceneAccessServer”, applied by Ajan to access the scene. Pointers to all MMISceneObjects and MMIAvatars in the scene are stored in two maps (MMISceneObjectsByID and MMIAvatarByID) with their UUIDs as respective keys. Setting access to these maps is given by “UnrealSceneAccess::AddMMIAvatar()” and “UnrealSceneAccess::AddMMISceneObject()” and the respective read access is given by “UnrealSceneAccess::GetMMIAvatarsVector()” and “UnrealSceneAccess::GetMMISceneObjectsVector()”. Generally, the maps are filled in “SimulationController::RegisterAllAvatarsAndObjects()”, which is called during “SimulationController::Setup()” at the simulation start. Each MMISceneObject and MMIAvatar removes its entry with “UnrealSceneAccess::RemoveMMISceneObject()” and “UnrealSceneAccess::RemoveMMIAvatar()”, usually at the respective destructor. Thus, the classes “SimulationController”, “UMMISceneObject” and “AMMIAvatar” contain pointers to the “UnrealSceneAccess”. However, “MMISceneObjectsByID” and “MMIAvatarsByID” are applied by the Plugin itself. The information about the scene, which I transferred to the MOSIM framework and thus to the adapters is stored in an MSceneUpdate instance (UnrealSceneAccess::SceneUpdate). The MSceneUpdate is maintained by a series of methods in the “UnrealSceneAccess”, all with the ending (…_SceneUpdate()), called when updates happen in the respective class. For instance, “UnrealSceneAccess::AddAvatar_SceneUpdate()” is called during “AMMIAvatar::Setup()” and “UnrealSceneAccess::AddSceneObject_SceneUpdate()” during “MMISceneObject::Setup()”. The “UnrealSceneAccess::SceneUpdate” is transferred with “UnrealSceneAccess::PushScene()” via a thrift client to the Adapters in the MOSIM framework. Most of the functionality in the “UnrealSceneAccess” provides the functionality for the “RemoteSceneAccessServer”. However, as thrift servers require a shared_ptr to the class containing the methods answering the queries and shared_ptrs are conflicting with Unreals own garbage collector, a wrapper class called “UnrealSceneAccessServer” was developed, that transfers the thrift queries to the ”UnrealSceneAccess”. As the “UnrealSceneAccessServer” is not directly assigned to an Unreal class, shared_ptrs of this class are possible. UnrealSceneAccess::ApplyManipulations() is called by “SimulationController::ApplySceneUpdate()” in “SimulationController::Tick()”. This method updates the transform and the physics of the MMISceneObjects, according to the SceneManipulations sourced in the last CoSimulator step. The updates are first written into the respective fields of the MSceneObject instance inside the MMISceneObjects. If any changes took place, ”UMMISceneObject::UpdateRootComponent()” or “UMMISceneObject::UpdatePhysics()” are called. These methods update the physics properties or the transform of the parent actor according to the changes that were written into the fields of the MSceneObject.

AMMIAvatar

This class is the representation of the MOSIM Avatar in the Unreal Engine. It is derivated from APawn. APawn was favored compared to ACharacter, as APawn allows a simpler C++ based access to the skeleton.

AMMIAvatar::AMMIAvatar() Initializes the MAvatar type and calls “AMMIAvatar::UpdateBaseName()” which removes additions made by Unreal to the name of the MMIAvatar as it is displayed in the engine.

AMMIAvatar::Setup() Called by “SimulationController::Setup()”. This method connects the MMIAvatar with the MOSIM framework, loads the MMUs, connects to the required services and initializes the posture. The port for the RemoteCoSimulationAccess service, started by the CoSimulationMMU is defined. For each MMIAvatar, this port is incremented by one, starting at 9011. The port is passed to the MMU later via the properties during the initialization of the MMU. Following that, the posture of the AMMIAvatar including the retargeting is initialized. A JSON file containing the global reference posture is loaded via the method “AMMIAvatar::LoadAvatarPosture()”. TODO: enable the user to define the file. The connection to the Retargeting and the SkeletonAccess Services is established. The retargeting is started by passing the global reference posture to the retargeting service with the call “RetargetingAccess::SetupRetargeting()”, which returns the MAvatarDescription. The MAvatarDescription is in turn then applied to initialize the anthropometry with “SkeletonAccess::InitializeAnthropometry()”. The setup of the posture is finished by passing the starting posture to MAvatar.PostureValues with “AMMIAvatar::ReadCurrentPosture()”, what includes the coordinate transformation of the location and rotation of the joints from Unreal coordinates to the MOSIM coordinates and the retargeting from the applied skeleton to the intermediate skeleton with “RetargetingAccess::RetargetToIntermediate()”. With this step, the setup of the MAvatar instance is finished, therefore “UnrealSceneAccess:AddAvatar_SceneUpdate()” is called. The MAvatar is added to the MSceneUpdate instance in the “UnrealSceneAccess”, which is transferred to the Adapters in the MOSIM framework. The connection to the MOSIM framework is established with the help of the MMUAccess class. The method “MMUAccess::Connect()” starts a thrift client that connects to the register and asks for the registered adapters. For each of the adapters, a “RemoteAdapterAccess” class is instantiated that is applied for handling the connection to the adapters and with an additional thrift client for the MMUs. A new session is created for each AMMIAvatar. The “CoSimulationMMU” is loaded and initialized. During the initialization, the access port for the “RemoteCoSimulationAccess” started by the CoSimualtionMMU is passed to the MMU with the properties map.

AMMIAvatar::Tick() The current posture is passed to the MAvatar instance via “AMMIAvatar::ReadCurrentPosture()”. Changes in the posture caused by Unreal, not by MOSIM are detected with this call. The updated posture is stored in the MSceneUpdate instance maintained in the “UnrealSceneAccess” by calling “UnrealSceneAccess::PostureValuesChanged_SceneUpdate()”. As MOSIM can be operated at a different time step than Unreal, this update is required. In contrast, the updates on the posture calculated by the Co-Simulator are passed in the “SimulationController::DoStep()” method.

UMMISceneObject

This class allows the user to mark AActors in the Unreal scene as MSceneObjects and makes them thus known to the MOSIM framework. It is derived from “USceneComponent”. This class brings the settings (transform, physics, colliders, etc.) of the parent AActor in a form that is applicable in the MOSIM framework and can therefore be set in the MSceneObject instance. Thus, the class contains a series of static methods e.g. for performing coordinate transformations between Unreal and MOSIM.

UMMISceneObject::UMMISceneObject() In the constructor, connections to the required components of the parent AActor are established. In detail, the parent AActors root component is accessed, it is tested whether the parent AActor has a static or a skeletal mesh component, the primitive component of the actor is stored and the respective body instance. The body instance is required for manipulating the parent AActors physics and the root component for manipulating the transform. Finally, “UMMISceneObject::SetupBase()” is called. The Setup functionality of the UMMISceneObject was divided into a basic setup and a full setup. Here the basic setup method is called, as in the constructor, it is not fully clear, which information of the parent AActor are required to be transferred to the MOSIM framework. For example, if the object is defined as a walk target, the colliders are not transferred, or as a tool, what results in disabling the gravity for the object. In detail in “UMMISceneObject::SetupBase()”, the transform of the MSceneObject is updated (including a coordinate transformation), the names (with -> unique name, without automatic additions made by Unreal -> not unique name, required for the Task List Editor), the physics of the MSceneObject and the Root- and PhysicsHistory is initialized. The update of the transform includes also the search for a parent MSceneObject in the scene hierarchy above the current MSceneObject (“UMMISceneObject::SetParent()”). A history of the last transform state and the physics of the parent AActor is included, as only updates exceeding a certain threshold value (defined by “UMMISceneObject::rootThreshold” and “UMMISceneObject::physicsThreshold”) are realized. This ensures that e.g. the location of the object does not start to oscillate due to number truncation.

UMMISceneObject::Setup() Called by: “SimulationController::Setup()” Extended version of the UMMISceneObject setup. Includes the update of the colliders. In the final step, the MSceneObject instance is passed to the UnrealSceneAccess MSceneUpdate instance.

UMMISceneObject::TickComponent() In this method, the transform and the physics of the MSceneObject is updated, if there were changes in the UMMISceneObjects parent AActor that exceed the aforementioned thresholds. For both the transform and the physics for detecting future changes and the MSceneUpdate instance in the “UnrealSceneAccess” are updated. In summary, updates to the UMMISceneObject are possible from two directions: First, the MOSIM Co-Simulator can compute updates, which are obtained in “SimulationController::DoStep()”. The actual update is then performed in “UnrealSceneAccess::ApplyManipulations()”. There, the updates are written into the respective fields in the MSceneObject instance and the transform as well as the physics of the parent AActor of the UMMISceneObject are updated. These changes are detected by “UMMISceneObject::TickComponent()”, and the history as well as the UnrealSceneAccess is updated accordingly. Second, the Unreal Engine performs updates on the parent AActor of the UMMISceneObject. These changes are detected as well during the TickComponent call by “UMMISceneObject::HasRootChanged()” and “UMMISceneObject::HasPhysicsChanged()” and the MSceneObjects fields as well as the history and the UnrealSceneUpdate are updated.

UMMISceneObject::setMSceneObjectName() Method for finding a unique name for the MSceneObject, not used in the scene so far. Required by the Task List Editor.

AjanAgent

Class derived from UActorComponent that enables the user to create and execute Ajan Agents. Has to be added as component to the respective MMIAvatar. The class requires a running Ajan instance, as it connects itself during “begin play” via a Http Post request with Ajan. By then, also the Ajan template list is updated. UAjanAgent::SynchronizeMSceneObjects() loads all MSceneObjects in the scene into the Knowledge Repository defined with the Knowledge Repo URI and uses for that also a Http post request. The function can thus be called also in the edit mode of the engine. UAjanAgent::EnableTakListEditorConnection() looks for a High Level Task List Editor AActor in the scene and stores a pointer to that class, if an instance was found. This method is required for the future Ajan-TaskListEditor connection. All other methods (UAjanAgent::CreateAgent(), UAjanAgent::ExecuteAgent() and UAjanAgent::DeleteAgent()) establish thrift connections to Ajan. They have to be executed in the order listed above. UAjanAgent::CreateAgent() requires information gathered by the Http request executed at “begin play”, therefore the functions are only executable in the play mode of the engine. As Ajan can only handle one open thrift connection at once, the thrift clients are created locally in each functions body and are deleted (what includes closing the thrift connection), when the scope of the function is left. For selecting the Ajan template in the details panel of the Ajan Agent component, a dynamic dropdown menu is required. Therefore, a details panel specializer class is necessary, which is called AjanAgentDetailsPanel. The class is connected in UAjanAgent::UAjanAgent() with UAjanAgent::InitializeCustomizedLayout() and de-registered in the destructor.