An implementation of the Salient Poses algorithm for Maya
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
SalientPosesPerformance @ da323a6
cgcmake @ 439cfe4
docs
eigen-git-mirror @ fbca041 CMake - include Eigen library directly Oct 19, 2018
results
scripts Fix - check for `None` when querying connections and keyframes Oct 24, 2018
src
.gitignore Ignored Python's compiled files Apr 10, 2018
.gitmodules Use http instead of ssh for SPP submodule Dec 19, 2018
CMakeLists.txt
LICENSE Switch from GPL-3.0 to MIT Oct 21, 2018
README.md
_config.yml Set theme jekyll-theme-minimal Sep 21, 2018
module.txt
setup.bat QOL - add executables to setup IDE Oct 21, 2018
setup.command
setup.sh

README.md

Salient Poses (Maya implementation)

Salient Poses is an algorithm that uses a technique called "keyframe reduction" to convert hard-to-edit motion capture into easy-to-edit keyframe animation. This project contains a Maya plug-in that enables artists to run Salient Poses directly inside of Maya.

Want to jump right in?

Grab the latest release, add it to Maya, and then follow along with the video tutorial!

Video Tutorial

Table Of Contents

This document starts off with an general introduction to the algorithm, outlines exactly what you get, provides the typical getting started guide, and then gives some detail on the code:

  1. Easy-to-Edit Motion Capture!
  2. What's in this Project?
  3. Getting Started
  4. Digging Into the Code

Please see Salient Poses website for more detailed information selected results, links to academic work, and more.

Finally, know that I'm always happy to help! Feel free to post bugs and feature requests to the issues board or otherwise click here to join the Slack channel and chat with me.

Easy-to-Edit Motion Capture!

Motion capture has become a core component of the animation pipeline in the visual effects industry. Whether it's a live-action blockbuster film or an indie game being developed in a back-alley office, motion capture is likely to be involved. While this technology is awesome - it allows actors to truly embody a fantasy character - it does have its problems.

The Problem

Say we start of with a mocap animation (here's one that I grabbed from Adobe's Mixamo):

Salient Example (Original)

While the animation looks nice, it actually has lots of keyframes. Let's take a look at the keyframes, visualized here as blue outlines. There are so many keyrames; one for every frame. In this case there are 60 per second!

Salient Example (Mocap Keyframes)

While having all of these keyframes involved is necessary during recording - we want to capture the actor's performance precisely - they involve a large memory footprint (problematic for video games) and make the motion hard to change (problematic for motion editors). Here's a picture of just some of the data for the animation above. Can you imagine loading all the animations for a protagonist character in a video game (there are 1000s of unique clips for a main character in recent triple-A games)? How about trying to adjust the motion using this data?

Salient Example (Mocap Data)

The Solution

To address this problem, I've developed a new algorithm for compressing and editing motion capture. Titled "Salient Poses", this algorithm uses an optimal keyframe reduction technique to simplify motion-capture animation. Conceptually, Salient Poses converts motion capture into hand-crafted keyframe animation.

More precisely, the algorithm works by finding potential set of important - that is, "salient" - poses. In each set of poses, the choice of poses has been carefully determined so that it most accurately reconstructs the original motion. Once found, we can create a new animation using just these poses. Here's an illustration of one possible optimal set of poses selected for the animation above:

Salient Example (Selected Keyframes)

Comparing these poses to the original motion-capture, we can already see the benefit: the motion can be expressed with fewer poses. Having fewer poses in the animation means a smaller memory footprint and also that editors invest less time for editing (fewer changes are required). And, furthermore, the data is now much sparser than before:

Salient Example (Data Before And After)

With all that done, the last step is to create a new motion using on the selected poses. To do this the algorithm performs inbetweening, which is the process of deciding how to transition between the poses to best recreate the original animation. It's hard to describe exactly how the in-betweening works, but you might imagine it as recreating the curve traced by each of the character's joints.

Salient Example (Data Before And After)

Before and After

To sum the whole process up, here's a look at the original animation (right side) and the same animation after compression with Salient Poses (left side). In this particular case the compressed animation contains only 7 keyframes (those illustrated above), paired with the reconstructed curves. Compared to the original animation, with 112 keyframes, that's around 94% compression!

Salient Example (Anim Before And After)

What's in this project?

This repository presents an implementation of the Salient Poses algorithm as a command-based Maya plug-in that I've designed for interactive use in Maya by motion-editors, animators, technical artists, and hobbyists. The tool is built with C++ and uses OpenCL to redistribute some of the heavier number crunching.

Since this implementation is designed for interactive use, it comes with its own interface. Using the interface you can specify parameters around the selection and then apply that selection to perform the keyframe-reduction. Once applied, the resulting animation consists of only the keyframes paired with curves that provide the inbetweens - much like hand-crafted animation.

Note: If you are looking for something that is super fast, runs offline, and can be automated, please check out the Salient Poses - Performance implementation. I'm also planning to work on a Blender implementation in the future). And, if that's not enough, let me know you're favorite animation tool, as I'd be happy to program up a custom implementation for you!

Getting Started

Getting the plug-in installed is simple. The steps are:

  1. ensure you have an OpenCL device (most Intel, AMD, and NVIDIA chips are fine),
  2. download the latest version matching your machine and Maya version from the releases page, and finally
  3. add the plug-in and its supporting Python scripts to Maya.

Each release is a compressed file that contains a compiled plug-in file and a scripts directory. The easiest way to add the plug-in and scripts to Maya is to load the compiled plug-in file using Maya's Plug-in Manager. On Windows you are looking to select SalientPosesMaya.mll, on OSX its SalientPosesMaya.bundle, and Linux its SalientPosesMaya.so.

Loading the compile plug-in file

From there, copy and paste the scripts directory into your Maya project's scripts (see this page if you're not familiar with projects).

Alternatively, if you want the plug-in to be available across all of your projects, you can configure Maya's MAYA_PLUG_IN_PATH to include the directory containing the compiled plug-in file and its PYTHONPATH to include the scripts folder. See this page if you're not familiar with configuring Maya's environment variables.

Setting Maya's environment variables

Usage

Note: don't forget to check out the video tutorial as well.

The interface guides you through the two phases: selection and reduction. Selection is process of choosing a range of potential keyframes for a given animation. Reduction is the process of first removing keyframes and then tweaking the resulting inbetweens to recreate the original motion. The interface will help you to manage these two steps.

Objects for selection versus objects for reduction

It's important to note that the objects you use for the selection step do not need to be the same as those for the reduction. For example, you might want to choose keyframes from the movement of the hand, but apply the reduction to the entire body. As an another example, you might choose keyframes from right-leg (perhaps so that the keyframes capture footsteps), but then apply the reduction to the hips. If you feel particularly crazy, you might even choose keyframes from a baked physics simulation and apply the reduction to something else entirely! Of-course, nothing is stopping you from using the same objects for both selection and reduction. Understanding the difference between these two steps is important because it gives you the power to control how the keyframes are distributed through the simplified version - which is essential for getting the result best-suited to your task.

Opening the tool

First, open the interface by running this Python code from Maya's Script Editor. Note: you can alternatively load the shelf included in the release, which provides a button that launches this interface.

import salient_menu; salient_menu.show()

Here's what you should see:

A look at the interface

Configure OpenCL device

First off, check the device listed next to the OpenCL Device label. You can click on the drop-down to show all OpenCL-compatible devices; choose whichever you prefer. For most cases, you won't notice a difference, but this is helpful if you want to ensure that Maya won't be holding up a particular device.

Using the tool

Now that we are setup, let's use the tool. Follow these five steps:

  1. Start / End Frame. First check that the start and end frame are set appropriately. These are set automatically, based on Maya's timeline configuration, when you open the interface. You can change these variables to isolate just a part of the animation for selection and reduction if you wish.

  2. Fixed Keyframes. Next, enter any particular keyframes that you would like to ensure are kept during reduction. Enter any keyframes you want kept as as comma-separated whole numbers; for example if you want to keep frames 30, 100, and 150 as keyframes then you'd specify: 30,100,150. Setting fixed keyframes can be important for some editing tasks; for example, you might want to keep the frame at the very top of a jumping motion as a keyframe.

  3. Evaluate. Now, select a set of animated objects in Maya; the outliner is useful for doing this. The choice of what to include is up to you - explore different choices to get a feeling for how the tool works. If you're just interested in compression, selecting the joints along a character's spine and limbs is generally the best option. Otherwise, if you've got a specific change to make, I find that choosing objects around the leading part is most useful (the leading part is an animation term that I first read about in John Lasseter's famous paper). Remember, the objects you select don't have to be those used later in the reduction step. Once you've got the objects selected, press the evaluate button and the tool will starting computing a range of optimal selections.

Note: The word optimal means that each particular set of keyframes is as-good-as-possible in terms of how well the simplified animation will recreate the original motion.

Choosing a selection

  1. Reduction. At this point, you need only choose a particular set of keyframes and then apply the reduction. You do this by adjusting the slider in the interface. You'll see the keyframes illustrated interactively as blue silhouettes. Once you've found a set of keyframes you would like to try, select the objects for reduction (this will usually be all controls for the character and any other props) and press the Reduce button. This will take a few seconds.

Before and After

After you've applied the reduction, play back the animation and also examine the graph editor to see if the result is satisfactory. There's no right answer; but generally you're looking for smooth curves that look similar to the original.

  1. Explore. Perhaps the result you get the first time wasn't quite right. Or maybe it was and that's great! In either case, the tool is designed to help you explore and compare multiple solutions quickly. You can explore by:
  • using the slider to increase and decrease the number of keyframes,
  • changing the start and end frame, and also
  • setting fixed keyframes.

You can even apply a reduction, do a playblast, and then undo the reduction. Note: the undo button works like a stack, you can do multiple reductions and then undo each one. As you explore, its important to think carefully about the trade-off between compression and error. If you are using the tool to compress assets for a video-game, you want to ensure that you retain a high level of detail while using as few keyframes as possible. This will help you to save on the memory footprint for your game. If you are using the tool for editing, try to examine how well the keyframes are distributed: is there enough keyframes for you to make an adjustment without distorting the motion? The graph within the interface helps to communicate this tradeoff (the red line indicates detail is lost for the current reduction).

That's it. Please remember that I'm happy to help you along the way if you ever get stuck. Feel free to jump onto the Slack Channel and chat with me, or otherwise check the Salient Poses website for more information!

Digging Into the Code

This algorithm works in a few steps, which I like to think of as:

  1. analysis, building a table that expresses the importance of all potential keyframes,
  2. selection, optimally choosing a set of poses for each level of compression, and
  3. reconstruction, where we create the new animation from a given set of poses.

If you'd like to peek into the code, it's best to start with the SalientPoses - Performance project (a high-performance command-line tool for running the algorithm offline).

If you're just interested in changing the interface, start by looking at this Python file and then some of the other files in the scripts directory. The implementation of the menu uses PySide2 and should be familiar enough to anyone with a little experience using Python.

Otherwise, start by looking at the Error Table class (this performs the analysis step), then the Selector class (this chooses optimal set of poses), and also the Interpolator class (this performs reconstruction using a basic curve-fitting technique).

Once you get that far, check out the SelectCommand and the ReduceCommand files. These two classes use the Error Table, Selector, and Interpolator to realize the algorithm in Maya.

If you've got this far and wait more information, jump on the Slack Channel with me or otherwise check the Salient Poses website for more information!