Release v0.3.0

Highlights

V2D v0.3.0 adds commercial CARI4D, multi-view human-object interaction reconstruction, and an end-to-end reference workflow from egocentric video to a fine-tuned GR00T policy.

Commercial CARI4D (NEW!)

Metric 4D human-object reconstruction from monocular third-person RGB video.

Summary

• Human and object reconstruction: converts RGB video, human/object masks, and a textured object mesh into camera-space human and object motion.
• Initialization: combines batched MoGe 2 depth estimation, SAM 3D Body initialized from SAM2 bounding boxes and masks, and FoundationPose registration and tracking.
• Contact refinement: combines CoCoNet prediction with full-clip contact-guided refinement to improve contact and temporal consistency.
• Containerized execution: provides a Docker build and a single inference wrapper, with downloadable model weights, resumable stages, and final video and report outputs.

Docs: reconstruction/modules/v2d_cari4d/README.md

Multi-View HOI Reconstruction (NEW!)

A pipeline for camera calibration and 4D human-object interaction reconstruction from multiple stereo views.

Summary

• Camera calibration: estimates camera extrinsics from chessboard recordings, with reported reprojection RMSE below 0.3 pixels.
• Human and object reconstruction: outputs 4D human poses in MHR/SOMA and 6-DoF object trajectories. The demo reports approximately 1.5 cm CD for both human and object reconstruction.
• Auxiliary outputs: produces anonymized RGB streams, estimated depth, human/object masks, a ground-plane estimate, and per-frame and aggregate accuracy metrics.
• Visualization: generates a multi-view overlay video for reviewing reconstructed human and object trajectories.
• Agentic operation: provides a local quickstart, module-level CLI documentation, and a skill for prerequisite checks and pipeline execution.

Known limitations

• The published MV-HOI calibration capture does not include the metadata.yaml that --rosbag_path expects when given a directory. Until the captures are republished, pass the sequence's .mcap file to --rosbag_path instead of the directory.

Docs: reconstruction/docs/mv_hoi_local_pipeline.md

Egocentric Hand-Object Reconstruction Pipeline

Summary

• Added vertex-based temporal smoothing for more stable reconstructed trajectories.
• Added GeoCalib-based gravity alignment for the reconstruction world frame.
• Added post-reconstruction scaling to a measured baseline for metric alignment.

Known limitations

• Ego reconstruction (DynHaMeR, HaMeR, and HaMeR_mesh) may exhibit inconsistent object scale, orientation flips, and hand–object misalignment, particularly for nearly symmetric objects or automatically generated meshes. Results may vary between runs due to non-deterministic camera trajectory estimation and mesh generation.

Docs: reconstruction/docs/ego_e2e_setup.md

Robotic Grounding

Summary

• Retargeting: adds object projection to reduce penetration and unifies retargeting and playback across floating-hand and arm-hand embodiments.
• Cross-embodiment training: reuses one CHORD-based training recipe, with motion speed, control, and reward tuning for stable expert training.
• Training performance: reports approximately 50% acceleration in Isaac Lab training compared with v0.2.
• Expert evaluation: achieved 126/128 successes (98.4%) for floating Sharpa and 117/128 successes (91.4%) for Vega Sharpa on the demonstrated task.

Docs: robotic_grounding/README.md

End-to-End Reference Workflow (NEW!)

A unified workflow from egocentric human video through reconstruction, retargeting, simulation scene preparation, RL expert training, data generation, GR00T N1.7 fine-tuning, and closed-loop evaluation in simulation and the real world.

Summary

• Video-to-policy integration: connects monocular egocentric reconstruction directly to retargeting and extends the workflow beyond RL expert policies to GR00T dataset generation and fine-tuning.
• Data collection: collects complete-cycle successes with synchronized cameras and semantic audits. The demo collected 100 successful expert episodes with trajectory variation and randomized lighting and robot, object, and background appearance.
• GR00T evaluation: achieved 87/100 complete task cycles in simulation, including lifting, sustained holding, controlled return, and stable placement.
• Resumable execution: adds run_e2e.sh with shared configuration, stage-specific inputs, manifests, unified logs, status reporting, dry runs, and resume support.
• Agentic operation: includes skills for the workflow, including $groot-finetune for episode selection, dataset preparation, fine-tuning, and evaluation.
• Validation: retains environment diagnostics, scene inspection outputs, simulation QA videos, dataset audits, action metrics, and closed-loop results and videos.