Skip to content

Maya-ACE: A Reference Client Implementation for NVIDIA ACE Audio2Face Service

License

Notifications You must be signed in to change notification settings

NVIDIA/Maya-ACE

Repository files navigation

Maya ACE

A Reference Client Implementation for NVIDIA ACE Audio2Face Service

Maya-ACE is a reference implementation designed as a client for the ACE Audio2Face service, which leverages NVIDIA's cutting-edge Digital Human Technology to generate high-quality, audio-driven facial animation. With Maya-ACE, users can effortlessly access and utilize the Audio2Face service through a simple, streamlined interface, or dive into the source code to develop their own custom clients.

This repository includes a Maya plugin, gRPC client libraries, test assets, and a sample scene—everything you need to explore, learn, and innovate with the ACE Audio2Face service. Whether you're looking to integrate this technology into your own projects or simply experiment with its capabilities, Maya-ACE provides a robust foundation.

The project is released under the MIT license, granting you freedom to use and modify the code, with the responsibility to ensure its appropriate use. Please note that NVIDIA assumes no liability for any issues arising from the use of this software.

The ACE Audio2Face service is accessible through NVIDIA NIM , and all the information for the service can be found from the Document Portal.

preview

Contents

What can be done

overview

Send Audio, Receive Animation

Maya-ACE allows users to send audio inputs and receive corresponding facial animations. These animations can be directly connected to a blendshape node, enabling you to animate any character in Maya seamlessly.

Learning with User-Friendly Interface

Maya-ACE provides all the necessary functionalities through a straightforward UI. It serves as an excellent tool for learning and experiencing the ACE Audio2Face service, helping users gain a deeper understanding of how it works.

Seamless Transition to Unreal Engine and Tokkio

For those looking to expand their workflow, users can elevate their projects by transitioning to the Kairos Unreal Engine integration or Tokkio: an interactive avatar virtual customer service assistant product SDK. This allows for the continued use of Audio2Face within other platforms, sharing the same parameters from Maya-ACE for a consistent experience.

Integration with Standard Maya Nodes

Maya-ACE is designed to work seamlessly with standard Maya nodes, including the blendshape node, making it adaptable to drive any character in your scene.

Customizable and Extendable

The source code and scripts provided with Maya-ACE can be modified to create a custom pipeline within Maya or to develop a client for other platforms, giving users the flexibility to tailor the tool to their specific needs.

Requirements

Other Maya versions and platforms can be supported in the future.

Getting Started

Quickstart

  1. Set an environment variable , NVCF_API_KEY, with a valid API key
    • To get a key, click the green Get API Key text on the right of the page and proceed with instructions.
  2. Download the mace package, unzip and copy contents to a Maya module path
  3. Download the sample maya scene and sample audio files: English(male) or Chinese(female)
  4. Launch Maya, load maya_aceclient plugin, and open the sample scene.
  5. Adjust Time Slider preferences : framerate=30, playback speed=30fps, looping=once
  6. Disable Cached Playback on the Time Slider
  7. Import an audio into the Maya scene and set it to the Time Slider sound
  8. Adjust Time Slider range to fit to the audio length
  9. Select AceAnimationPlayer1 node and open Attribute Editor
    • You may turn off DAG Objects only from the Display menu to see AceAnimationPlayer1 on the outliner window.

    • To open attribute editor, click menu -> Windows -> General Editors -> Attribute Editor
  10. Click the option menu next to Audiofile attribute, and select the audio imported before.
  11. Click Request New Animation button. Wait for the Received ### frames message.
  12. Click Maya's play button. Watch how face is moving

Set up with sample assets

  1. Get an API key and setup Maya-ACE for Maya. Please follow Quickstart 1. and 2.
  2. Download sample fbx files: Mark, Claire
  3. Launch maya with a new scene. Load maya plugins; maya_aceclient and fbxmaya
  4. Create references of sample fbx files
  5. Adjust Time Slider preferences : framerate=30, playback speed=30fps, looping=once
  6. Disable Cached Playback on the Time Slider
  7. Import a sample audio: English or Chinese
  8. Adjust timeslider range to fit to the audio
  9. Select c_headWatertight_mid of Mark, and click menu->ACE->Attach a new Animation Player
    • (Optional) Tips to optimize the viewport for faces
      • Hide unused groups; such as c_mouth_grp_mid, r_eye_grp_mid, l_eye_grp_mid, root
      • Setup a viewport camera with focal length between 75mm and 150mm looking at the face(s)
      • Change viewport camera's Near Clip Plane to 10
      • Assign standardSurface1 material to face(s)
  10. Open Attribute Editor, Select AceAnimationPlayer1 node
  11. Enter valid Network Address, Api Key, and Function ID on the Attribute Editor.
  12. To change audio, Select an option from the AudioFile, and also update Time Slider to use the same audio.

  13. Click Request New Animation button, and wait for the Received ### frames message.
  14. Click Maya's play button. Check animation on Mark's face.
  15. Select both AceAnimationPlayer1 and c_headWatertight_mid of Claire in order, and click menu->ACE->Connect an existing Animation Player
    • You may turn off DAG Objects only from the Display menu to see AceAnimationPlayer1 on the outliner window.
  16. Click Maya's play button again. Check animation on both faces.

User Interface

Attribute Editor - AceAnimationPlayer

Network and Audio

Information to connect ACE Audio2Face service, and audio file information to request animation.

  • Network Address: A full url with protocol and port number. example: https://grpc.nvcf.nvidia.com:443
  • Api Key: A valid api key acquired from the web page
  • Function Id: A valid function id that is specific for the service and an AI model. Find a proper Function ID from the web page
    • Examples (as of August 21, 2024)
      • Mark model: 945ed566-a023-4677-9a49-61ede107fd5a
      • Claire model: 462f7853-60e8-474a-9728-7b598e58472c
  • Audiofile: A path to an audio file to request animation from the service. It can be also selected from imported audios through the drop down option.

Emotion Parameters

Parameters to control generated emotion and preferred(manual) emotion. Audio2Face generate animation with the emotion input which includes generated emotion and preferred emotion. Please watch this video to understand how it works.

  • Emotion Strength: the strength of the overall emotion; the total of auto emotion and preferred(manual) emotion.
    • emotion = emotion strength * (preferred weight * preferred emotion + (1.0 - preferred weight) * generated emotion)
  • Preferred Emotion: Enable/disable and the ratio of the user driven emotion in the overall emotion (1.0 = 100% preferred emotion, 0.0 = 100% generated emotion).
  • Auto Emotion: Parameters to control generated emotion.

Face Parameters

Parameters to control overal face animation. Please check Audio2Face Microservice documents for the updated information.

Blendshape Multipliers

Override specific expressions by multiplying the Audio2Face result.

Blendshape Offsets

Override specific expressions by adding constant values to the Audio2Face result;

  • each output = (raw result * multipler) + offset

Main Menu

Attach a new Animation Player

Create a new AceAnimationPlayer and connect to the selected blendshape node

Connect an existing Animation Player

Connect the select AceAnimationPlayer to the secondly selected blendshape node

Export A2F Parameters

Export a json file with parameters from a AceAnimationPlayer

  • DISCLAIMER: This export may not fully reflect recent updates from the server or ACE services. The following limitations may result in potential discrepancies:
    • Parameters not set and controlled by the Maya plugin will be exported with default values.
    • Default values in the exported configuration files may differ if the A2F server is deployed with custom settings.
    • To ensure consistency between the client and server, carefully compare the server's configuration with the exported parameters.
    • For the most accurate and up-to-date information, please refer to the official website or the server configuration.

Best Practices

Proposed Workflow

Generally, it requires multiple tries to find an optimal setting. Users are expected to work with an iterative process with using Maya-ACE in their projects. All animations should be downloaded to see the result through any changes; such as paramters. The typical workflow is,

  1. Adjust Parameters
  2. Update animation by clicking Request New Animation button
  3. Play the scene and check the animation with audio
  4. Repeat from 1 to 3 until satisfied with the result
  5. Bake animation of the blendshape weights
  6. Save the scene

Parameter Tuning Guide

Please check ACE Parameter Tuning Guide to understand individual parameters.

Save Animation through Bake Animation

Maya provides Bake Simulation to freeze and save animation that is downloaded from the service.

  1. Select blendshape node and select all blendshape weights from the channel box
  2. Click Bake Simulation through menu; (Animation Layout) menu -> Keys -> Bake Simulation

Connecting Custom Blendshapes

Maya-ACE assumes to use ARKit compatible blendshape set , but it is still usable with a partial set of blendshapes through this process.

Once AceAnimationPlayer receives animation, it will update output weights with blendshape names that is received from the service, and then Connect existing Animation Player menu will use the names to find the best possible blendshape targets based on the names.

Rules to match blendshape names

  • The blendshape name must match with one of the arkit face location
  • Case insensitive

Steps to use the name-based connection

  1. Finish receiving animation and update Output Weights of AceAnimationPlayer node with blendshape names
  2. Select the AceAnimationPlayer node and the blendshape node to connect
  3. Click Connect existing Animation Player from the menu -> ACE

Troubleshooting Tips

Communication errors

These are common errors when requesting new animation through Maya-ACE.

Error Messages Possible Reasons (not all)
Unauthenticated wrong api key, using http for an https endpoint
no authorization was passed missing api key
SSL_ERROR_SSL mismatched SSL version
Deadline Exceeded bad network connection, slow machine or service, long audio
Cloud credits expired out of credits
DNS resolution failed network error, wrong address
Connection refused service is unavailable

Cannot attach or connect AceAnimationPlayer

To attach AceAnimationPlayer, the selection should be a blendshape node or a mesh node with one blendshape connected.

To connect an existing AceAnimationPlayer to a new blendshape node, the selection should be one AceAnimationPlayer node and a blendshape node or a mesh with a blendshape node connected. There could be errors when,

  • Selection is not in the correct order
  • Selected multiple AceAnimationPlayer node or multiple blendshape node
  • Selected a mesh connected with multiple blendshape nodes

Please make sure the selection is correct for each operation.

Build and Test

Maya Ace Client

Setting up Development Environment

Step 1: Build Tools

Please install the following build tools are accessible in your PATH environment variable.

  1. MSBuild (Visual Studio 2019+)
  2. python3
Step 2: Prepare Maya devkit

To build Maya plugins, Maya devkit is required and must be located at the correct path.

Download Maya devkit from Autodesk website and locate the devkitBase directory under deps directory of the local repo.

./deps
└───devkitBase
    ├───cmake
    ├───devkit
    ├───include
    ├───lib
    └───mkspecs

To run maya tests and use plugins, Maya application is required, and MAYA_LOCATION environment variable should be set to the Maya's install directory.

By default, Maya 2024 is expected, and the build script will copy plugin(s) to _build/windows-x86_64/release/plguins/mace/plug-ins/2024.

Step 3: Download Dependencies

The script will download required dependencies and place in _build\target-deps

.\fetch_deps.bat

For convenience, Premake5 will be downloaded to .\deps\premake. Please use this version for subsequent builds or add it to your PATH variable. If you already have Premake5 installed, it should also be compatible.

Local Build

For a quick release build with Visual Studio 2022

build.bat

To specify visual Studio version other than 2022, please change vs2022 to your installed Visual Studio version

  • To run MSBuild.exe, it needs to use Developer Powershell or to set up the system path.
# build maya_acecleint plugin
premake5.exe vs2022 --solution-name=maya-ace --aceclient_log_level=1
MSBuild.exe /NoWarn:MSB8003,MSB8005 .\_compiler\maya-ace.sln /p:Configuration=release /p:Platform=x64 /verbosity:minimal /m /t:mace

# for debug log level
premake5.exe vs2022 --solution-name=maya-ace --aceclient_log_level=3

Running Unit Tests

Note: Please build first before running the test. The build process will also pull pip packages needed for unit tests.

# run unit test with mayapy
.\test_maya.bat

#  run tests which match the given substring
.\test_maya.bat -k test_plugin
Example: Overriding Maya Version for Tests (Powershell)
$ENV:MAYA_LOCATION="C:\Program Files\Autodesk\Maya2025"
.\test_maya.bat
Example: Overriding Maya Version for Tests (Windows Command Prompt)
SET "MAYA_LOCATION=C:\Program Files\Autodesk\Maya2025"
.\test_maya.bat

Launching Maya with test environment

.\run_maya.bat

Starting a local mock ace server

.\venv\Scripts\activate
.\run_mock_server.bat

ACE Client Library

A Static library that handles communication with ACE; sends and receives data.

Local Build

For a quick release build with Visual Studio 2022

build.bat

To specify visual Studio version other than 2022, please change vs2022 to your installed Visual Studio version

  • To run MSBuild.exe, it needs to use Developer Powershell or to set up the system path.
# build aceclient test
premake5.exe vs2022 --solution-name=tests-aceclient --aceclient_log_level=1
MSBuild.exe /NoWarn:MSB8003,MSB8005 .\_compiler\tests-aceclient.sln /p:Configuration=release /p:Platform=x64 /verbosity:minimal /m /t:tests-aceclient

Running Unit Tests

# run tests for aceclient
.\test_aceclient.bat

ACE gRPC C++ Library

Please read Generating the ACE gRPC module to know how to update the grpc generated files.

Additional Knowledge

Audio File Requirements

Maya-ACE uses AudioFile to read audio files, and supported formats may vary from Maya's supported audio file formats'.

We recommend using the formats below for the best compatiblity and efficiency.

  • file format: wav
  • data format: 16bit PCM or 32bit IEEE Float
  • samplerate: 16kHz, 32kHz, or 48kHz
  • channels: 1 channel (mono)

About ACE

NVIDIA ACE is a suite of real-time AI solutions for end-to-end development of interactive avatars and digital human applications at-scale.

Additional information about ACE is available from NVIDIA Documentation Hub.

For ACE customers to get support, please contact through NVIDIA Enterprise Support

About

Maya-ACE: A Reference Client Implementation for NVIDIA ACE Audio2Face Service

Resources

License

Stars

Watchers

Forks

Packages

No packages published