Skip to content

Releases

ExMR v0.1.5: portrait phone videos, foot contacts, export frame rate

Choose a tag to compare

@twehrbein twehrbein released this 29 Oct 07:24

No new dependencies or model files: existing installations update with git pull. Fresh installs pin ffmpeg 9.0.2 (was 7.1.1), see docs/INSTALL.md; existing installations can keep their ffmpeg.

Fixes

  • Portrait phone videos: in videos with rotation metadata (most phone videos filmed upright), the mesh ended up beside the person. ffmpeg applied the rotation when extracting the frames, but the person tracker and the camera motion estimation (SimpleVO, DPVO) decoded the video themselves without it. Every stage now reads the extracted frames, so all of them see the same images and the same frame count, whatever the video's metadata (rotation, mirroring, variable frame rate).
  • ffmpeg crashing at the end of a run: ffmpeg could segfault after all videos were written, a known NVIDIA driver bug when several NVENC sessions close in one process. Each output video now has its own ffmpeg process, and when ffmpeg fails, its exit code and messages are reported.
  • Caches across versions: a run into an output directory of an earlier version reused that version's boxes, keypoints, features and camera rotations. The cache keys now include the ExMR version, so rerunning a video processed with v0.1.4 (e.g. a rotated one) recomputes them.

New

  • Foot contacts: results.pt and results.npz contain foot_contact (L, 2), whether the [left, right] foot touches the ground: its ankle or toe is predicted static and it is the lowest part of the body (within 5 cm), e.g. to lock the feet when retargeting the motion to another character. See docs/OUTPUTS.md.
  • --export_fps: the AMASS and BVH files at another frame rate than the video's, e.g. 24 fps for film, 60 for games, or 30 for the SMPL-X Blender add-on and GMR with 25 fps videos. Rotations are interpolated on the sphere (slerp), the root position linearly; the other outputs keep the video's frames.
  • Shadow in the world view: the world view (<name>_global.mp4, <name>_incam_global.mp4) draws the mesh's shadow straight down on the ground; a gap between a foot and its shadow means the foot is above the ground.
  • A hint when the feet float: without --snap_to_ground, the demo suggests it when the feet are more than 5 cm above the ground where they touch it, usually vertical drift.

Changes

  • World frame facing +z: the world-frame motion starts at x=z=0 facing +z, as BVH, Blender and Unity expect a character to start. World-frame results (and the AMASS and BVH files) are therefore rotated about the vertical axis compared with v0.1.4.
  • World-frame ankles closer to the network's: the foot IK's toe targets stay within 3 cm of the network's toes. Before, the IK could pull a toe predicted static back by several centimeters by turning the ankle by up to tens of degrees, against the network's ankle.
  • Camera-frame results change slightly for all videos, as the tracker and the camera motion now read the extracted JPEG frames instead of decoding the video.
  • The README example uses --snap_to_ground (its ground is flat), as the online demo's example does.
  • Library use: Tracker.get_main_track, SimpleVO and DPVOOdometry take the directory of the extracted frames (exmr.preprocessing.video.extract_frames) instead of the video path, and exmr.preprocessing.video.encode_frames is removed. exmr.pipeline.render(..., videos=...) renders only the requested views. New: Options.export_fps, exmr.export.resample_motion and exmr.model.global_traj.foot_contacts.

The online demo runs v0.1.5: its results file contains the foot contacts, it shows the floating-feet hint, and its example uses snap to ground.

Clone with --recursive: the source archives below do not include the git submodules.

ExMR v0.1.4: joints, BVH, robots, faster demo

Choose a tag to compare

@twehrbein twehrbein released this 26 Oct 10:01

No new model files: existing installations update with git pull. The dependency pins (requirements.txt, constraints.txt) and the prebuilt nvdiffrast wheel are for fresh installs, see docs/INSTALL.md.

New

  • Joints in the results: results.pt and results.npz contain the 3D joints in the camera and world frame (joints_incam, joints_global), their projection into the image (joints_2d) and joint_names: the 55 SMPL-X joints and 21 surface points (nose, eyes, ears, toes, heels, fingertips). exmr.pipeline.compute_joints computes them for results of earlier versions.
  • BVH export: every run also writes <video name>.bvh, the world-frame motion as a skeleton animation of the 55 SMPL-X joints in centimeters, for Blender, Maya, MotionBuilder and similar.
  • Unity, Unreal and humanoid robots: the AMASS file now stores 16 betas, as loaders such as GMR expect; docs/OUTPUTS.md describes the FBX route through the SMPL-X Blender add-on, retargeting to humanoid robots with GMR, and which output to use for what.
  • --start, --end: process only a part of the video, in seconds or as [hh:]mm:ss, cut frame-accurately and losslessly.
  • --f_mm: the camera's focal length (35 mm equivalent), if known, for the model, SAM 3D Body, the camera motion estimation and rendering.
  • Faster demo on the example video (RTX 4090, -s): fast 1 min 1 s, balanced 1 min 17 s, accurate 2 min 59 s (v0.1.3: 1:09, 1:23, 3:22). On GeForce GPUs the FP16 stages accumulate in FP16 (accuracy unchanged in our evaluation), RTMW's session warms up while the tracker runs, DPVO's input is capped at 1080p size, and the SMPL-X model is loaded once per run.
  • Install without compiling: nvdiffrast comes as a prebuilt wheel for the pinned torch; all dependencies are pinned. HaMeR's files are downloaded from the HaMeR authors' Hugging Face Space (2.7 GB instead of the 6 GB archive). Windows: see the WSL2 note in the install docs.
  • Library use: exmr.pipeline exposes the pieces run_video is built from (hand_boxes, model_inputs, compute_joints, save_results, render), e.g. for pipelines with preloaded models.

Changes

  • compute_joints, exmr.export.save_amass and save_bvh take a SmplxModel instead of the path of SMPLX_NEUTRAL.npz (see the examples in docs/OUTPUTS.md).
  • On GeForce GPUs, results differ from v0.1.3 at the level of run-to-run noise (FP16 accumulation); other GPUs are unchanged.
  • run_video checks for nvdiffrast and for the SAM 3D Body submodule before any work.

Clone with --recursive: the source archives below do not include the git submodules.

ExMR v0.1.3: Blender export and world-frame post-processing

Choose a tag to compare

@twehrbein twehrbein released this 22 Oct 09:18

Changes since v0.1.0. No new dependencies or model files: update with git pull.

New

  • Blender export: every run also writes <video name>_amass.npz, the world-frame motion as an AMASS SMPL-X file that the SMPL-X Blender add-on imports. See Blender and AMASS; exmr.export.save_amass also converts results of earlier versions.
  • --smoothing {global,both,none}: temporal smoothing of the output motion (Gaussian, ~33 ms). The default global smooths only the world-frame motion; both also smooths the camera-frame motion.
  • --snap_to_ground: removes vertical drift (the person seems to fly or sink) in long videos of a single flat ground, by letting the body touch the ground at least every 0.5 s.

Changed

  • The world-frame motion is placed on the ground with the full SMPL-X mesh (lowest vertex at y=0) instead of the lowest joint.
  • Static camera (-s): the world trajectory uses the moving-camera corrections and then removes slow drift away from the camera.
  • World-frame results therefore differ from v0.1.0. Camera-frame results are unchanged with the default settings.

Installation: docs/INSTALL.md (clone with --recursive; the source archives below lack the git submodules).