Skip to content

v3.3

Latest

Choose a tag to compare

@tesslerc tesslerc released this 10 Nov 07:09

Anatomical SOMA GPC prior, G1 wrist frame and Dex3 robot, native teleop, backend fixes

This release ships the generative prior for the anatomical SOMA humanoid with
the tracker it is trained on, moves the G1 hand frame to the wrist and adds the
G1 with Dex3 hands as a robot, extends the native Metatron deployment with hand
teleoperation, motion tracking and a live metrics dashboard, and corrects a set
of simulator backend behaviours found by review.

Pretrained models

  • data/pretrained_models/gpc_prior/anatomic_soma_bones: a new Generative
    Pre-trained Controller (GPC) prior for the anatomical SOMA, an autoregressive
    transformer (6 layers, 4 heads, width 1024) over the anatomical tracker's FSQ
    codes (40 scalars with 9 levels, 8 tokens of 5), conditioned on
    max_coords_obs, with the tracker's decoder embedded. NVIDIA Open Model
    License, with model card, saved configuration and provenance. Its recipe is
    examples/experiments/gpc/prior_anatomic_soma.py (the GPC prior of
    gpc/prior.py rolled out in the anatomical tracker's own world).
  • data/pretrained_models/motion_tracker/anatomic_soma_bones_fsq: the weights
    change. The anatomical tracker was retrained with the head tracked explicitly
    (head position among the tracked bodies and a head orientation term), after
    the released checkpoint was reported to hold the head about 16 degrees
    nose-down in every simulator. These are the exact weights the new prior is
    frozen against, so the two bundles share one motion vocabulary. The saved
    configuration carries the head terms and the body angular-speed cap below;
    the provenance checksum changes. On the shipped ten-clip pack in MuJoCo the
    head pitch error drops from 18.7 to 2.2 degrees with tracking unchanged.
  • data/pretrained_models/motion_tracker/g1-bones-deploy: the weights are
    unchanged; the saved configuration is regenerated for the G1 wrist hand frame
    (hand groups and tracked bodies name the wrist links; the two rubber-hand
    markers join the scored self-contact bodies, which carry no collision
    geometry, so the cost is unchanged).

Robots

  • G1 hand frame at the wrist. common_naming_to_robot_body_names and
    trackable_bodies_subset for g1 name left/right_wrist_yaw_link instead of the
    massless rubber-hand markers, a link every G1 hand variant has. The marker
    bodies stay in the asset. A recipe that resolves the hand groups fresh now
    tracks a point about 12 cm closer to the elbow; policies trained on the
    marker frame keep their saved configurations and resume unchanged, but under
    a freshly resolved g1 config the same goal names a different body. The
    packaged Metatron G1 release names its tracked bodies explicitly and is
    unaffected.
  • body_subtree_names(robot_config, names): a kinematic-subtree helper in
    protomotions/robot_configs/base.py that returns the named bodies and every
    body below them. The BeyondMimic recipe (mlp_bm_l2c2.py) uses it to exclude
    the hands and feet with their markers or fingers from the self-contact cost
    instead of naming G1 links by hand; the Metatron teacher puts its payload
    randomization on the leaf bodies of each hand chain.
  • g1_dex3: the Unitree G1 with Dex3-1 hands as a registered robot (47 bodies,
    43 DOF: the 29 body joints plus 7 finger joints per hand), asset
    protomotions/data/assets/mjcf/g1_dex3.xml, config
    protomotions/robot_configs/g1_dex3.py, guide docs/source/robots/g1_dex3.rst.
    The body gains, body map and tracked bodies are shared with g1 through
    module-level helpers in robot_configs/g1.py; finger gains follow the GEAR
    SONIC humanoid-object-interaction experts.
  • Anatomical SOMA: the body angular-speed cap (max_angular_velocity) moves
    from 100 to 1000, matching the other humanoids. Isaac Lab reads this field
    in degrees per second and Isaac Gym in radians per second (now documented on
    the field); MuJoCo and Newton do not read it. This changes the anatomical
    robot's physics on those two backends, including inference with the shipped
    tracker.

Simulators

  • MuJoCo: joint effort readback is the robot-joint slice of the actuator and
    applied forces after the final physics substep, instead of the last explicit
    control command, so effort-based rewards and metrics on MuJoCo use the forces
    the engine applied (implicit PD and force limits included). Training and
    evaluation rewards that include effort change on MuJoCo; integration, states
    and observations do not. A configured fixed root is welded at its authored
    pose; armature overrides refresh the derived model constants (physics
    changes when the configured armature differs from the compiled asset); the
    explicit PD law is limited to the robot joints when projectiles are present;
    friction and payload variations reach the physics.
  • Newton: joint gains are assigned by complete joint name through a shared
    mapping (protomotions/simulator/newton/joint_mapping.py) that refuses missing
    or ambiguous names, instead of a lookup that could assign gains to the wrong
    joints when names overlapped or orders differed. The state buffers are
    preserved after an odd number of substeps so readback sees the completed
    step. Root velocities are reported at the link origin (Newton reports the
    free root's linear velocity at its centre of mass); resets and pushes apply
    the inverse conversion.
  • Isaac Gym: joint properties (stiffness, damping, armature, friction, limits)
    are written in the loaded asset's joint order. Previously a differing common
    order assigned them to the wrong joints.
  • Isaac Lab: angular pushes include the centre-of-mass correction that keeps
    the requested change in link-origin velocity.

Training and evaluation

  • Motion-free training saves its environment state as env.ckpt instead of
    env_null.ckpt. To resume an earlier motion-free run with its environment
    state, rename env_null.ckpt to env.ckpt. Runs with a motion file keep their
    checkpoint names.
  • Configuration overrides of bare and fixed-length sequences apply correctly;
    the agent output directory is created before the first save; evaluation
    without a reweighting interval skips motion reweighting; gradient norms
    report zero when no gradient exists; PoseLib names the missing bodies in its
    errors; terrain segmentation is initialized before it is filled; terrain
    plotting works with Matplotlib versions that removed plt.cm.get_cmap.
  • The process-owned Weights & Biases run is finished before the simulator shuts
    down, so a run's final state is recorded on normal exit, on error and on
    interrupt.
  • MdpComponent.get_params() returns a copy of the static parameters, so a
    caller that folds constants into it cannot alter the component.
  • Tutorials and executable examples report a failure before simulator cleanup
    can replace its exit status with zero.

Inference and deployment

  • Native Metatron deployment (deployment/native): a live metrics stream (one
    JSON frame per 50 Hz control cycle over UDP, optionally to a file) with a
    read-only dashboard (tools/live_metrics_viewer.py: overview, task, joints,
    thermal, timing and settings tabs) and a replay renderer; PICO headset hand
    teleoperation with floor levelling, hand goals anchored at the hand-over,
    rotation goals, torso height from the headset and an arm-IK residual on the
    seven joints of each active arm; a motion-tracking mode fed by the web demo's
    motions, re-anchored to the robot each cycle; operator settings declared by
    the task (default, qualified value and hard bound) through an atomically
    written file, replacing the UDP command port; input presses latched so a tap
    shorter than one control cycle counts; simulation disturbances and a grid
    runner for qualification. TensorRT engines always run as captured CUDA
    graphs (the flag is gone). Xbox motion tracking and a kick-only TTS cue are
    documented in the deployment README. Nothing in this stack was run on the
    physical robot in this release.
  • Deployment input classification reads the shared sensor namespace set, so a
    newly registered sensor namespace receives the correct provider.

Documentation

  • The pretrained-model catalog, quickstart and GPC guide list the anatomical
    GPC prior; the anatomical tracker card describes the head tracking and the
    prior relationship; the Robots section gains the g1_dex3 page and the G1
    page notes the wrist frame.
  • Installation: the Isaac Lab path uses uv venv and uv pip install instead of
    uv sync, whose cross-platform lock fails at the pinned Isaac Lab commit.
  • The pose library page points at the checks that exist.

Compatibility summary

  • Weights changed: the anatomical SOMA tracker. New: the anatomical SOMA GPC
    prior. Unchanged: every other checkpoint.
  • Saved configurations changed: the anatomical tracker (head terms, angular
    cap) and the G1 deployment tracker (wrist hand frame). Existing runs resume
    with the configuration they were trained on; new configurations resolve the
    wrist frame, the anatomical angular cap and the env.ckpt name.
  • Numerical changes on existing setups: MuJoCo effort readback and armature
    overrides; Newton gain assignment, odd-substep readback and root velocities;
    Isaac Gym joint-property order; Isaac Lab angular pushes; the anatomical
    SOMA angular cap on Isaac Lab and Isaac Gym.

Validation

  • The anatomical GPC prior's embedded decoder and encoder are identical,
    tensor for tensor, to the shipped anatomical tracker; the prior strict-loads
    and steps in MuJoCo. No rollout-quality evaluation of the prior was run.
  • The anatomical tracker checkpoint tracks the shipped ten-clip pack in MuJoCo
    (10 of 10, mean body position error 0.045 m) and 256 BONES-SEED clips (256 of
    256, 0.049 m); head pitch error 2.2 degrees against 18.7 before.
  • Every shipped bundle strict-loads into the model its saved configuration
    builds, commands joint targets inside the robot's joint range and steps in
    MuJoCo; every bundle regenerates byte for byte from its config.yaml.
  • Backend fixes were verified against the real engines where available:
    MuJoCo effort and fixed-root probes, Newton 1.0.0 on CPU, Isaac Gym Preview
    4 on an RTX 4090 for the joint-property order; Isaac Lab pushes by unit test
    only.
  • Native deployment: 24 native CTests and 60 dashboard tests pass; every
    teleop and tracking scenario ran in MuJoCo, and the PICO headset path was
    exercised live against the PC service.
  • Continuous integration on the candidate passed (lint, licenses, secret scan,
    CPU tests, release projection); the projected tree's shipped-Python header
    sweep and the model-licence check pass locally.

Known limitations

  • The anatomical GPC prior has no rollout-quality evaluation; its card states
    training simulator only.
  • The native teleop, tracking and dashboard were not run on the physical G1.
  • Isaac Lab push and Newton root-velocity corrections rest on unit tests and
    pinned-engine checks, not on a training run.

Merge: one commit, merged as-is with "Rebase and merge" on the public
repository.