This repository is part of the code release for the Siggraph Asia 2023 paper "From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans".
[paper] [project page]
Given a SMPL body motion sequence as .npz or .pkl we enable getting a scaled OpenSim skeleton model (.osim) and the corresponding skeleton motion (.mot).
input_output.mp4
Left: Input SMPL sequence Right: Output OpenSim sequence
We provide here code for fitting an OpenSim skeleton model from SMPL sequences using AddBiomechanics. AddBiomechanics is a free online tool designed to align an OpenSim skeleton model to motion capture data.
If you are loooking for the SKEL model code, please check the SKEL repository.
This repo was tested with python3.8
First, install the dependencies and the package with the following line:
pip install -e .
Download the file: SMPL_python_v.1.1.0.zip from the SMPL download page. And run:
cd ../SMPL2AddBiomechanics
python scripts/setup_smpl.py /path/to/SMPL_python_v.1.1.0.zip
Download the file: skel_models_v1.x.zip from the SKEL download page. And run:
cd ../SMPL2AddBiomechanics
python scripts/setup_bsm.py /path/to/skel_models_v1.1.zip
To generate a personalized custom marker set on the BSM model, you will need OSSO. Please check the OSSO installation instructions to set it up.
Below is an example that takes the SMPL motion models/bsm/sample_motion/01_01_poses.npz and generates measurements and synthetic motion capture data from the sequence.
python smpl2ab/smpl2addbio.py -i models/bsm/sample_motion/01 This generates an output folder containing the following files:
output
└── 01
├── _subject.json
└── trials
└── 01_01_poses.trc
The _subject.json file contains the estimated subject measurements :
{"sex": "male", "massKg": "68.94", "heightM": "1.80"}
, and the trc file contains the synthetic motion capture data for the input sequences.
This information can be used as input for AddBiomechanics to align the OpenSim model to the motion capture data. Note that you will need to use the developer version of https://dev-addbiomechanics.org/ that support the BSM model.
In the 'Upload Custom OpenSim Model' section, upload the BSM model models/bsm/bsm.osim.
In the "Motion Capture Files" section, upload the trc file from output/01/trials/01_01_poses.trc.
Below is shown the form filled with the data from the example above:
Then click on "Process" and wait for the optimization to finish. You will then be able to download the optimized model.
Once the process done, you can download the results (in this case 01.zip), extract the zip file and run the following command to visualize the results:
python smpl2ab/show_ab_results.py --osim_path=addbio_res_folder/01/Models/match_markers_but_ignore_physics.osim --mot_path=addbio_res_folder/01/IK/01_01_poses_segment_0_ik.mot --smpl_motion_path=models/bsm/sample_motion/01/01_01_poses.npzThis script shows the superimposition of the input SMPL sequences, the markers that were created in green, and the OpenSim skeleton aligned by AddBiomechanics.
superimp_res.mp4
The marker locations wrt the bones depend on the body shape. While AddBiomechanics does optimize the location of the marker, we propose a method that leverages OSSO to generate personalized custom markers set on the OpenSim model.
To use this option, add the '--osso' flag to the command line. This will generate a bsm.osim file in the output folder that contains a version of the BSM model with personalized markers.
python smpl2ab/smpl2addbio.py -i models/bsm/sample_motion/01 --osso --displayThen proceed as above to upload the data to AddBiomechanics. But this time, upload the bsm.osim file from output/01/bsm.osim in the "Upload Custom OpenSim Model" section.
Note that this option only works with the BSM model. If you want to use your own OpenSim model, you will need to manually add the markers on the model.
If you want to run AddBiomechanics locally, you can download and install the source code from the AddBiomechanics GitHub page.
The script AddBiomechanics/blob/main/server/engine/src/engine.py runs the optimization given a folder of data.
You will need
- A SMPL motion sequence
- A biomechanical skeleton model .osim with markers defined on it (BSM will be used by default)
- A dictionary giving, for each marker of the .osim, the corresponding vertex index on SMPL (The BSM correspondence will be used by default)
Below, we detail those items in more depth.
You can download SMPL sequences from the AMASS download page, by clicking the SMPL+H button.
Here we show an example for the subject 10 of CMU. The AMASS data consists in a folder with this structure:
CMU
└── 10
├── 10_01_poses.npz
├── 10_02_poses.npz
├── 10_03_poses.npz
├── 10_04_poses.npz
└── 10_05_poses.npz
Each .npz is a sequence.
You should pass the path to the subject folder (in this example /path/to/CMU/10) to the smpl2addbio.py script.
This is an OpenSim model with markers defined on it, in .osim format. By default, the downloaded BSM model will be used. You can also use your OpenSim model:
python smpl2ab/smpl2addbio.py -i models/bsm/sample_motion/01 --osim /path/to/your_model.osim --marker_dict /path/to/smpl_markers_dict.yamlNote that in this case, you will need to provide a dictionary giving, for each marker of the .osim model, the corresponding vertex index on SMPL, see the next section for more detail.
To fit the OpenSim model to the SMPL sequence, we need to have the same markers defined on both models. So for each marker of the .osim model, you need to provide the corresponding vertex index on SMPL.
By default, we use the BSM marker set. You can find it as a dictionary in SMPL2AddBiomechanics/smpl2ab/data/bsm_markers.yaml.
If your model has a different marker set, you can create your own dictionary similar SMPL2AddBiomechanics/smpl2ab/data/bsm_markers.yaml.
This dictionary must contain the same markers as your .osim model. To help you to create this dictionnary, you can run the following script:
python smpl2ab/utils/kin_helpers.py -o /path/to/your/model.osim -m -mrYou can check visually that the SMPL markers defined in the dictionary match the OpenSim model markers by running:
python smpl2ab/show_markers.py --osim_path /path/to/your/model.osim --smpl_markers_path /path/to/smpl_markers_dict.yaml
If you use this software, please cite the following work and software:
@inproceedings{keller2023skel,
title = {From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans},
author = {Keller, Marilyn and Werling, Keenon and Shin, Soyong and Delp, Scott and
Pujades, Sergi and Liu, C. Karen and Black, Michael J.},
booktitle = {ACM ToG, Proc.~SIGGRAPH Asia},
volume = {42},
number = {6},
month = dec,
year = {2023},
}
This code and model are available for non-commercial scientific research purposes as defined in the LICENSE.txt file.
For any questions about BSM and SKEL, please contact skel@tuebingen.mpg.de.
For commercial licensing, please contact ps-licensing@tue.mpg.de