MMUDevelopment

MMU Development

Modular Motion Units are central components of the MOSIM framework, generating motion for specific tasks (e.g. for the motion type categories below). They are partial motion models, that do not have to simulate everything in a monolithic software, but can focus on specific components of the motion. For example, an MMU implementing the type "Object/Grasp" can focus purely on the motion of the hand and the fingers, while not dealing with the problem of how to walk near the target object or how to reach it with the arm.

Motion Types

graph TD;
  Default-->Locomotion;
  Default-->Pose;
  Default-->Object;
  Default-->UseTool;
  Locomotion-->Walk;
  Walk-->Jog;
  Walk-->Run;
  Walk-->Crouch;
  Locomotion-->Turn;
  Locomotion-->ClimbLadder;
  Locomotion-->ClimbStairs;
  Pose-->Idle;
  Pose-->Reach;
  Pose-->WaiveHand;
  Pose-->StandingUp;
  Pose-->SittingDown;
  StandingUp-->_Chair;
  StandingUp-->_Car;
  SittingDown-->Floor;
  SittingDown-->Chair;
  SittingDown-->Car;
  Object-->Release;
  Object-->Move;
  Object-->Carry;
  Object-->Grasp;
  Object-->Close;
  Object-->Open;

Loading

MMU compatibility with behavior and other MMUs

New MMUs can be created to have same base functionality to already existing ones. This is the case where behavior designed for already existing MMUs can be reused. There are two important conditions to fulfill to be able to reuse existing behavior with new MMU:

1. New MMU have to use same motion type, and

2. runtime parameters must be exactly same.

If runtime parameters can accept some additional values not specified in the existing behavior, then new MMU would run with the old behavior but at limited functionality. In such a case creating new behavior would be beneficial, keeping it backward compatible with older MMUs. MMU must implements CheckPrerequisites method, which is the only tool that is used to verify if the MMU can run with the provided runtime parameters, scene state, and scene parameters, if such are required by the MMU. When high-level task is broken down to a low level tasks executed by MMUs, behavior checks data contained in the description.json files attached to the MMUs that represent motion type required for specific action. The first MMU that fulfills task requirements is instantiated and CheckPrerequisites function is run to verify that the MMU can do the job. If CheckPrerequisites returns false, then another MMU is tested.

MMU uses CheckPrerequisites to check availibility of supported scene parameters and returned values could be grouped into two main categories:

1. Fail1: Scene parameters are not available,
2. Fail2: Object is too far from the avatar

The error information is returend by LogData list of the response of the CheckPrerequisites function. This information is divided into two parts computer code that should be standardized across MMUs (Fail1, Fail2) and after the semicolon, the message to the end user which should help identifying the source of error.

To make sure new MMU is compatible with existing MMUs from the same motion type, aligning scene parameters can be beneficial. This way scene objects containing constraints or geometrical structure required by MMU can be reused with multiple MMUs. For example MMU realizing sitting on a chair might require from the chair object to describe by constraints where is the seat, from which sides it is accessible (consider armchair and a stool), is there a backrest and if there is where it is located. In this context it is important to check scene parameters required by existing MMUs, and their interpretation. For example, grasping points should follow general specification to make sure every object can be grasped by various grasping MMUs, without need for definition of additional scene parameters.

This is not yet part of any standard, but scene objects containing meta information/constraints that are intended for the use by MMUs could be to some extent standardized so that scene objects can support multiple MMUs, and not be MMU specific. MMU should never require scene parameters set to be fixed to a required set. The list of scene parameters can be arbitrarily long and use of MMU should not limit adding extra parameters to the object, which if not needed by the MMU should be simply neglected. For example grasping MMU might require grasping points definition on a chair, at the same time it does not care about seat location or armrest locations, which might be used by sitting down MMU or some other MMU.

Nativ MMU Development

In order to integrate your motion synthesis technique to the MOSIM framework, it is sufficient to implement the MMU Interface and use either one of the provided Adapters (Unity, Python or C#) or implement your own Adapter.

MMU Development using the Adapters (C#, Unity)

In order to use the C# or Unity Adapter, it is sufficient to implement the MMU Interface and compile the MMU according to the adapter specification (including the required *.dlls and description.json file). Examples for this can be found in the Basic MMUs inside the MOSIM_Core repository.

How to develop a C# MMU

• Creating a new library project (.NetFramework 4.6 or older)
• Integration of MMIStandard and MMICSharp into project
• Creating a new class for implementing the MMU functionality which inherits from MMUBase
• (Optionally) annotation of the provided events and MMUDescription using the MAttributes
• Building of the MMU and its artifacts and optionally automatic generation of the description file

Automatic Description Generation

The C# repository contains a program for automatically generating description files based on annotated meta-data in the code. The application must be executed in the postbuild of the MMU.

At the top of the class, the MMUDescription must be annotated. This MMUDescriptionAttribute contains all mandatory properties required to automatically generate an MMUDescription.

   [MMUDescriptionAttribute("Felix Gaisbauer", "1.0", "CarryMMUNested", "carryNested", "","A carry MMU", "A carry MMU realized as nested co-simulation")]
   public class CarryMMUNestedImpl:MMUBase
   {
     ... class code in here
   }

In addition to the MMUDescription, the parameters/constraints required by the MMU must be further specified using the MParameterAttribute. These attributes describe the name of the parameter, the type as well as if the parameter is required or mandatory. Moreover a textual description is provided.

   [MParameterAttribute("TargetID", "ID", "The id of the object which should be carried", true)]
   [MParameterAttribute("Hand", "{Left,Right}", "The hand of the carry motion", true)]
   public override MBoolResponse AssignInstruction(MInstruction instruction, MSimulationState simulationState)
   {
     ... program logic in here
   }

Analogously, custom events which are provided by the MMU can be annotated above the DoStep function.

   [MSimulationEventAttribute("PositioningFinished", "PositioningFinished")]
   public override MSimulationResult DoStep(double time, MSimulationState simulationState)
   {
     ... program logic in here
   }

After having defined all meta information, next, the description file can be automatically generated. Please insert in the postbuild events of Visual studio as outlined in the following:

"../../../MMUDescriptionAutoGenerator\bin\Debug\MMUDescriptionAutoGenerator.exe" "$(TargetDir)$(ProjectName).dll"

This will execute the generated executable of the MMIDescriptionAutoGenerator. As result a MMUDescription file will be stored in the project output directory.

Below, the outcome of the auto-generated description file is illustrated.

{
  "Name": "CarryMMUNested",
  "ID": "20ab6123-22b1-4226-ae0b-41edeef91844",
  "AssemblyName": "CarryMMUNested.dll",
  "MotionType": "carryNested",
  "MotionSubType": "",
  "Language": "C#",
  "Author": "Felix Gaisbauer",
  "Version": "1.0",
  "Prerequisites": null,
  "SupportedProportions": null,
  "Properties": null,
  "Dependencies": null,
  "Events": [ "PositioningFinished" ],
  "LongDescription": "A carry MMU realized as nested co-simulation",
  "ShortDescription": "A carry MMU",
  "Parameters": [
    {
      "Name": "TargetID",
      "Type": "ID",
      "Description": "The id of the object which should be carried",
      "Required": true
    },
    {
      "Name": "Hand",
      "Type": "{Left,Right}",
      "Description": "The hand of the carry motion",
      "Required": true
    }
  ],
  "SceneParameters": null,
}

Debugging a MMU

The repository contains an application named DebugAdapter which can be used for debugging developed MMUs. The core idea of the DebugAdapter is to host a virtual adapter in debug mode which incorporates the desired MMU as direct project reference. The Adapter just acts as a proxy forwarding the commands. The adapter actively registers at the central MMIRegister and handles incoming requests, which will be forwarded to the MMU. Ultimately, this allows to debug the created MMUs. For Unity MMU a similar mechanism is integrated in the MMUGenerator.

Below the DebugAdapter example class is illustrated. Please note that the code could be integrated in any project for testing purposes. A good practise is to create a class within the developed MMU project. By switching the application type from library to console application, the virtual adapter could be instantly started without requiring external software.

    /// <summary>
    /// Class used for debugging the MMU
    /// </summary>
    class Debug
    {
        static void Main(string[] args)
        {
            using (var debugAdapter = new DebugAdapter.DebugAdapter(typeof(CarryMMUImpl)))
            {
                Console.ReadLine();
            }
        }
    }

Unity specifics

Since Unity MMUs heavily rely on the Unity engine and its functionality, the MMUs consists of two parts: First, the source code in C#. Second, AssetBundle files which contain resources of the UnityEngine (e.g. the digital human model, GameObjects). The development of UnityMMUs should be done using the MMUGenerator. The MMUGenerator has a separate wiki entry.

MMU Development using the Adapters (Python)

As python software usually requires optimized python environments, we provided the MMIStandard and MMIPython implementation as pip-packages. After installing these packages into your python environment (e.g. by using the deploy script), you can easily start a python adapter using a similar script as below

import MMIPython.PythonAdapter as PythonAdapter
from MyMMUPackage import MyMMU

description = ""

with open("./description.json", "r") as f:
	description = " ".join(f.readlines())
if __name__ == "__main__":
	PythonAdapter.start_adapter([(description, MyMMU)])

MMU Development using the MMUGenerator

We provide an open-source Generator for MMUs integrated in Unity. This generator can assist in packaging and distributing animation-based motion models from Unity. For more information, please consider the documentation of the MMU Generator

Motion Transfer

Motion is transferred using an intermediate skeleton structure and MAvatarPostureValues. You can either generate this data on your own, following the explanation in the documentation for the intermediate skeleton or by using the retargeting service. For more information, please consider the respective documentation.