Repository navigation
Anatomical SOMA GPC prior, G1 wrist frame and Dex3 robot, native teleop, backend fixes
This release ships the generative prior for the anatomical SOMA humanoid with
the tracker it is trained on, moves the G1 hand frame to the wrist and adds the
G1 with Dex3 hands as a robot, extends the native Metatron deployment with hand
teleoperation, motion tracking and a live metrics dashboard, and corrects a set
of simulator backend behaviours found by review.
Pretrained models
- data/pretrained_models/gpc_prior/anatomic_soma_bones: a new Generative
Pre-trained Controller (GPC) prior for the anatomical SOMA, an autoregressive
transformer (6 layers, 4 heads, width 1024) over the anatomical tracker's FSQ
codes (40 scalars with 9 levels, 8 tokens of 5), conditioned on
max_coords_obs, with the tracker's decoder embedded. NVIDIA Open Model
License, with model card, saved configuration and provenance. Its recipe is
examples/experiments/gpc/prior_anatomic_soma.py (the GPC prior of
gpc/prior.py rolled out in the anatomical tracker's own world). - data/pretrained_models/motion_tracker/anatomic_soma_bones_fsq: the weights
change. The anatomical tracker was retrained with the head tracked explicitly
(head position among the tracked bodies and a head orientation term), after
the released checkpoint was reported to hold the head about 16 degrees
nose-down in every simulator. These are the exact weights the new prior is
frozen against, so the two bundles share one motion vocabulary. The saved
configuration carries the head terms and the body angular-speed cap below;
the provenance checksum changes. On the shipped ten-clip pack in MuJoCo the
head pitch error drops from 18.7 to 2.2 degrees with tracking unchanged. - data/pretrained_models/motion_tracker/g1-bones-deploy: the weights are
unchanged; the saved configuration is regenerated for the G1 wrist hand frame
(hand groups and tracked bodies name the wrist links; the two rubber-hand
markers join the scored self-contact bodies, which carry no collision
geometry, so the cost is unchanged).
Robots
- G1 hand frame at the wrist. common_naming_to_robot_body_names and
trackable_bodies_subset for g1 name left/right_wrist_yaw_link instead of the
massless rubber-hand markers, a link every G1 hand variant has. The marker
bodies stay in the asset. A recipe that resolves the hand groups fresh now
tracks a point about 12 cm closer to the elbow; policies trained on the
marker frame keep their saved configurations and resume unchanged, but under
a freshly resolved g1 config the same goal names a different body. The
packaged Metatron G1 release names its tracked bodies explicitly and is
unaffected. - body_subtree_names(robot_config, names): a kinematic-subtree helper in
protomotions/robot_configs/base.py that returns the named bodies and every
body below them. The BeyondMimic recipe (mlp_bm_l2c2.py) uses it to exclude
the hands and feet with their markers or fingers from the self-contact cost
instead of naming G1 links by hand; the Metatron teacher puts its payload
randomization on the leaf bodies of each hand chain. - g1_dex3: the Unitree G1 with Dex3-1 hands as a registered robot (47 bodies,
43 DOF: the 29 body joints plus 7 finger joints per hand), asset
protomotions/data/assets/mjcf/g1_dex3.xml, config
protomotions/robot_configs/g1_dex3.py, guide docs/source/robots/g1_dex3.rst.
The body gains, body map and tracked bodies are shared with g1 through
module-level helpers in robot_configs/g1.py; finger gains follow the GEAR
SONIC humanoid-object-interaction experts. - Anatomical SOMA: the body angular-speed cap (max_angular_velocity) moves
from 100 to 1000, matching the other humanoids. Isaac Lab reads this field
in degrees per second and Isaac Gym in radians per second (now documented on
the field); MuJoCo and Newton do not read it. This changes the anatomical
robot's physics on those two backends, including inference with the shipped
tracker.
Simulators
- MuJoCo: joint effort readback is the robot-joint slice of the actuator and
applied forces after the final physics substep, instead of the last explicit
control command, so effort-based rewards and metrics on MuJoCo use the forces
the engine applied (implicit PD and force limits included). Training and
evaluation rewards that include effort change on MuJoCo; integration, states
and observations do not. A configured fixed root is welded at its authored
pose; armature overrides refresh the derived model constants (physics
changes when the configured armature differs from the compiled asset); the
explicit PD law is limited to the robot joints when projectiles are present;
friction and payload variations reach the physics. - Newton: joint gains are assigned by complete joint name through a shared
mapping (protomotions/simulator/newton/joint_mapping.py) that refuses missing
or ambiguous names, instead of a lookup that could assign gains to the wrong
joints when names overlapped or orders differed. The state buffers are
preserved after an odd number of substeps so readback sees the completed
step. Root velocities are reported at the link origin (Newton reports the
free root's linear velocity at its centre of mass); resets and pushes apply
the inverse conversion. - Isaac Gym: joint properties (stiffness, damping, armature, friction, limits)
are written in the loaded asset's joint order. Previously a differing common
order assigned them to the wrong joints. - Isaac Lab: angular pushes include the centre-of-mass correction that keeps
the requested change in link-origin velocity.
Training and evaluation
- Motion-free training saves its environment state as env.ckpt instead of
env_null.ckpt. To resume an earlier motion-free run with its environment
state, rename env_null.ckpt to env.ckpt. Runs with a motion file keep their
checkpoint names. - Configuration overrides of bare and fixed-length sequences apply correctly;
the agent output directory is created before the first save; evaluation
without a reweighting interval skips motion reweighting; gradient norms
report zero when no gradient exists; PoseLib names the missing bodies in its
errors; terrain segmentation is initialized before it is filled; terrain
plotting works with Matplotlib versions that removed plt.cm.get_cmap. - The process-owned Weights & Biases run is finished before the simulator shuts
down, so a run's final state is recorded on normal exit, on error and on
interrupt. - MdpComponent.get_params() returns a copy of the static parameters, so a
caller that folds constants into it cannot alter the component. - Tutorials and executable examples report a failure before simulator cleanup
can replace its exit status with zero.
Inference and deployment
- Native Metatron deployment (deployment/native): a live metrics stream (one
JSON frame per 50 Hz control cycle over UDP, optionally to a file) with a
read-only dashboard (tools/live_metrics_viewer.py: overview, task, joints,
thermal, timing and settings tabs) and a replay renderer; PICO headset hand
teleoperation with floor levelling, hand goals anchored at the hand-over,
rotation goals, torso height from the headset and an arm-IK residual on the
seven joints of each active arm; a motion-tracking mode fed by the web demo's
motions, re-anchored to the robot each cycle; operator settings declared by
the task (default, qualified value and hard bound) through an atomically
written file, replacing the UDP command port; input presses latched so a tap
shorter than one control cycle counts; simulation disturbances and a grid
runner for qualification. TensorRT engines always run as captured CUDA
graphs (the flag is gone). Xbox motion tracking and a kick-only TTS cue are
documented in the deployment README. Nothing in this stack was run on the
physical robot in this release. - Deployment input classification reads the shared sensor namespace set, so a
newly registered sensor namespace receives the correct provider.
Documentation
- The pretrained-model catalog, quickstart and GPC guide list the anatomical
GPC prior; the anatomical tracker card describes the head tracking and the
prior relationship; the Robots section gains the g1_dex3 page and the G1
page notes the wrist frame. - Installation: the Isaac Lab path uses uv venv and uv pip install instead of
uv sync, whose cross-platform lock fails at the pinned Isaac Lab commit. - The pose library page points at the checks that exist.
Compatibility summary
- Weights changed: the anatomical SOMA tracker. New: the anatomical SOMA GPC
prior. Unchanged: every other checkpoint. - Saved configurations changed: the anatomical tracker (head terms, angular
cap) and the G1 deployment tracker (wrist hand frame). Existing runs resume
with the configuration they were trained on; new configurations resolve the
wrist frame, the anatomical angular cap and the env.ckpt name. - Numerical changes on existing setups: MuJoCo effort readback and armature
overrides; Newton gain assignment, odd-substep readback and root velocities;
Isaac Gym joint-property order; Isaac Lab angular pushes; the anatomical
SOMA angular cap on Isaac Lab and Isaac Gym.
Validation
- The anatomical GPC prior's embedded decoder and encoder are identical,
tensor for tensor, to the shipped anatomical tracker; the prior strict-loads
and steps in MuJoCo. No rollout-quality evaluation of the prior was run. - The anatomical tracker checkpoint tracks the shipped ten-clip pack in MuJoCo
(10 of 10, mean body position error 0.045 m) and 256 BONES-SEED clips (256 of
256, 0.049 m); head pitch error 2.2 degrees against 18.7 before. - Every shipped bundle strict-loads into the model its saved configuration
builds, commands joint targets inside the robot's joint range and steps in
MuJoCo; every bundle regenerates byte for byte from its config.yaml. - Backend fixes were verified against the real engines where available:
MuJoCo effort and fixed-root probes, Newton 1.0.0 on CPU, Isaac Gym Preview
4 on an RTX 4090 for the joint-property order; Isaac Lab pushes by unit test
only. - Native deployment: 24 native CTests and 60 dashboard tests pass; every
teleop and tracking scenario ran in MuJoCo, and the PICO headset path was
exercised live against the PC service. - Continuous integration on the candidate passed (lint, licenses, secret scan,
CPU tests, release projection); the projected tree's shipped-Python header
sweep and the model-licence check pass locally.
Known limitations
- The anatomical GPC prior has no rollout-quality evaluation; its card states
training simulator only. - The native teleop, tracking and dashboard were not run on the physical G1.
- Isaac Lab push and Newton root-velocity corrections rest on unit tests and
pinned-engine checks, not on a training run.
Merge: one commit, merged as-is with "Rebase and merge" on the public
repository.