Skip to content

ExMR v0.1.5: portrait phone videos, foot contacts, export frame rate

Latest

Choose a tag to compare

@twehrbein twehrbein released this 29 Oct 07:24
· 1 commit to main since this release

No new dependencies or model files: existing installations update with git pull. Fresh installs pin ffmpeg 9.0.2 (was 7.1.1), see docs/INSTALL.md; existing installations can keep their ffmpeg.

Fixes

  • Portrait phone videos: in videos with rotation metadata (most phone videos filmed upright), the mesh ended up beside the person. ffmpeg applied the rotation when extracting the frames, but the person tracker and the camera motion estimation (SimpleVO, DPVO) decoded the video themselves without it. Every stage now reads the extracted frames, so all of them see the same images and the same frame count, whatever the video's metadata (rotation, mirroring, variable frame rate).
  • ffmpeg crashing at the end of a run: ffmpeg could segfault after all videos were written, a known NVIDIA driver bug when several NVENC sessions close in one process. Each output video now has its own ffmpeg process, and when ffmpeg fails, its exit code and messages are reported.
  • Caches across versions: a run into an output directory of an earlier version reused that version's boxes, keypoints, features and camera rotations. The cache keys now include the ExMR version, so rerunning a video processed with v0.1.4 (e.g. a rotated one) recomputes them.

New

  • Foot contacts: results.pt and results.npz contain foot_contact (L, 2), whether the [left, right] foot touches the ground: its ankle or toe is predicted static and it is the lowest part of the body (within 5 cm), e.g. to lock the feet when retargeting the motion to another character. See docs/OUTPUTS.md.
  • --export_fps: the AMASS and BVH files at another frame rate than the video's, e.g. 24 fps for film, 60 for games, or 30 for the SMPL-X Blender add-on and GMR with 25 fps videos. Rotations are interpolated on the sphere (slerp), the root position linearly; the other outputs keep the video's frames.
  • Shadow in the world view: the world view (<name>_global.mp4, <name>_incam_global.mp4) draws the mesh's shadow straight down on the ground; a gap between a foot and its shadow means the foot is above the ground.
  • A hint when the feet float: without --snap_to_ground, the demo suggests it when the feet are more than 5 cm above the ground where they touch it, usually vertical drift.

Changes

  • World frame facing +z: the world-frame motion starts at x=z=0 facing +z, as BVH, Blender and Unity expect a character to start. World-frame results (and the AMASS and BVH files) are therefore rotated about the vertical axis compared with v0.1.4.
  • World-frame ankles closer to the network's: the foot IK's toe targets stay within 3 cm of the network's toes. Before, the IK could pull a toe predicted static back by several centimeters by turning the ankle by up to tens of degrees, against the network's ankle.
  • Camera-frame results change slightly for all videos, as the tracker and the camera motion now read the extracted JPEG frames instead of decoding the video.
  • The README example uses --snap_to_ground (its ground is flat), as the online demo's example does.
  • Library use: Tracker.get_main_track, SimpleVO and DPVOOdometry take the directory of the extracted frames (exmr.preprocessing.video.extract_frames) instead of the video path, and exmr.preprocessing.video.encode_frames is removed. exmr.pipeline.render(..., videos=...) renders only the requested views. New: Options.export_fps, exmr.export.resample_motion and exmr.model.global_traj.foot_contacts.

The online demo runs v0.1.5: its results file contains the foot contacts, it shows the floating-feet hint, and its example uses snap to ground.

Clone with --recursive: the source archives below do not include the git submodules.