Components of the Framework
The proposed framework comprises several components interconnected by a platform independent middleware. The following figure visualizes the overall framework and the involved components.
The target engine (e.g. a Unity Project) is executing the simulation and visualizes the result. It connects to the MOSIM Framework over the thrift (TCP/IP) framework to the central registry in the MOSIM Launcher, which is running in a separate program. The services usually are individual executable software components, which connect and register at the central registry and can be called by either the target engine, other services or MMUs. Adapters are individual software components, connecting to the central registry. They load MMUs from the hard-drive and provide a communication layer to the rest of the framework. Hence, MMUs itself do not have to deal with the network connection and scene buffering. These features are provided by the adapters to all loaded MMUs. The CoSimulator can be either directly integrated into the target engine (by default) or run as a standalone software (e.g. in case of Unreal Engine). It organizes, schedules and merges the execution of different MMUs.
We have three phases of the MOSIM Framework execution.
- Connection
- Initialization
- Execution
During Connection, all components start and connect to the central registry. This step should be performed before starting the target engine. Once the target engine starts, it connects to the framework, loads all components and initializes the adapters and CoSimulator based on the specific avatar-description of the avatar to be simulated and the scene environment. After all components have been initialized, the target engine can start the simulation. It can execution motion behaviors that are either manually defined or derived by the high-level task list and behavior execution engine. In order to execute MMUs, instructions are assigned to the co-simulator, which in turn evaluates the prerequisites of MMUs, schedules the MMUs and assigns the instructions to the MMUs. At each frame, the target engine calls the DoStep method of the CoSimulator to query a new frame for a given elapsed time. The CoSimulator in turn evaluates all MMUs, which will simulate partial motion data based on their instruction. The returned motion is then returned to and displayed in the target engine.
The term Target Engine is used to describe the central clock and entity containing the scene-graph. It is the end-point using the MOSIM Framework to simulate a specific digital human model (target avatar) and is responsible for visualizing the scene and human model. It provides functionality in a communication layer to access the scene-graph and processes changes to the scene-graph triggered by the MOSIM Framework.
More Information
The target engine contains the specific target avatar that is to be simulated. It initializes a new session in the MOSIM Framework with the anthropomorphic description of the target avatar. It provides a scene access service and a scene manipulation service to enable engine agnostic interaction with the scene graph. During the simulation, it provides the central clock to and queries frames from the co-simulation with the DoStep method of the co-simulator (see below).
Relevant interface functions:
- MSceneAccess
- MSceneManipulationService
- MAvatarDescription
- MCoSimulator
Motion Model Units (MMUs) simulate motions for specific tasks. MMUs are not stateless and have to implement the MotionModelUnit interface. Sessions have to be initialized with an anthropometric description of the specific avatar that has to be simulated. Instructions have to be assigned to the MMU through the co-simulator to provide context to the MMU and a goal (e.g. where to reach to, where to walk to, etc.). MMUs can utilize services to solve additional tasks, e.g. for path-planning, inverse-kinematics, and posture blending.
More Information
A MMU has to implement the MotionModelUnit interface containing the following functions:
- Initialize
- AssignInstruction
- DoStep
- ...
Relevant Interface Functions:
- MotionModelUnit
- MSimulationState
- MSimulationResult
A co-simulation is utilized to orchestrate the specific scheduling and execution of MMUs. It implements the same functional interface as a MMU. It utilizes boundary constraints of MMUs to schedule the execution and combines the result of different MMUs to produce a single output pose per frame to be visualized in the target engine. A single instance of a co-simulator always simulates a single humanoid target avatar. The co-simulator does not simulate any other entities in the scene (e.g. robots, production line, etc.).
More Information
The Co-Simulation should provide an additional co-simulator access service to enable communication with the behavior modeling. Although different implementations of a co-simulation are possible, a reference implementation was provided. This reference implementation utilizes a hierarchical execution scheme.Relevant Interface Functions:
- MotionModelUnit
- MSimulationState
- MSimulationResult
- ...
More Information
More Information
Services are used to encapsulate functionality that is required by different MMUs. For example, inverse kinematics (IK) solvers, path planning, motion blending, and retargeting are steps that many MMUs require. In order to re-use this functionality, several service interfaces are defined for these common tasks. In addition, a generic service interface was proposed to enable further implementation of services.
More Information
There are different service interfaces, providing very specific functionality. In addition, there is a generic interface, to enable the further extension of new services without the requirement to change the core framework.Relevant Interface Functions:
- ...
If the MMUs are realized as separate standalone applications, each MMU must implement the communication by itself, which induces potential error sources and implementation overhead. As a result, Adapters are proposed to encapsulate this overhead. Adapters are a proxy for the communication with MMUs, registering them at the central registry (see below), managing the scene synchronization including (optionally) buffering the scene, and managing connections to services and other entities in the MOSIM Framework. Adapters are language-specific (e.g. C# adapter, Unity adapter, python adapter) and thus allow for direct inter-process communication with MMUs. It is possible to write custom adapters for custom frameworks (e.g. a RAMSIS adapter).
More Information
A reference implementation of several adapters is provided. Most of them expect a MMU directory, containing resources (e.g. dll-files) and MMU descriptions in a textual json format. The python adapter is an exception, as python MMUs require highly specified python environments and, in case of python 3, import rules are very strict with respect to the file structure.Relevant Interface Functions:
- ...
The launcher serves two purposes. First of all, as the MOSIM Framework consists of several, distinct components that have to be started as individual programs, the launcher provides functionality to automatically start all adapters, services, and MMUs. The second purpose is to be a central registry. All components communicate over the thrift-software and hence over a TCP/IP connection. The registry gathers all components including their IP-address and port to access the component.
More Information
...Relevant Interface functions:
- ...
Unlike traditional simulation techniques (e.g. for cyber-physical systems using FMI), motion modeling and simulation techniques are not all implemented in C and cannot be wrapped easily in binary *.dll files with C-headers. In order to be able to combine a heterogeneous set of methods and frameworks, the method of remote procedure calls using the Apache thrift software was chosen. Several data types and functional interfaces are defined in the thrift meta-language and allow the communication with remote function calls over a TCP/IP connection.
More Information
...There is no consensus on how a digital human model is represented. However, many solutions utilize kinematic models using a hierarchy of different joints. In order to be able to transfer human motion, there has to be at least a consensus between the exchanging partners, on how the motion is represented. For this purpose, we proposed a new description of skeletal hierarchies and defined a specific hierarchy (intermediate skeleton) which is utilized to allow efficient and fast exchange of motion information. In order to assist MMU developers, a retargeting service and configuration tool was developed.
More Information
Relevant Interface Functions:
- MSkeletonAccess
- MRetargetingService
- MAvatarDescription
- MAvatarPosture, MAvatarPostureValues
- MJoint, MJointType, MChannels
There has to be a consensus of the coordinate system and the semantics of global axes. For the MOSIM framework, a left-handed coordinate system using the X-Axis as left, Y-Axis as up and Z-Axis as forward was chosen.