Release v0.2.0

Video Ingestion Agent

What's new

• Cosmos3-Nano backend — config-only model switch; scripts/serve.py auto-detects the model family and applies the required vLLM flags (Reasoner tower only, ~8B-tier footprint)
• Unified locked environment — one uv.lock across Docker and host: vLLM 0.21 (up from the 0.12.x pin), transformers 5.x, torch 2.11 + CUDA 13
• Local backend restored — in-process HuggingFace inference works again (torchcodec video decoding); offline single-GPU workflows no longer require a vLLM server
• Configurable API endpoint — models.api_url overrides the api backend gateway; 401/403 now fail immediately with an actionable message instead of retrying silently
• Database integrity — re-ingesting a video replaces its previous rows in place (no more orphaned segments/entities); SQLite foreign-key enforcement with ON DELETE CASCADE; databases from older versions migrate automatically on first ingest
• Loud failures — a run that produces zero clips exits 1 and writes nothing (no placeholder segments, no resume marker), so scripted callers can detect it

Known limitations

1. Partial model-call failure is not reported — If the model backend becomes unavailable partway through an ingestion run, the clips processed after that point are skipped and the run still reports Pipeline completed successfully! with exit code 0. On a re-ingestion of a video already in the database, the previous run's rows are purged first, so the incomplete results replace a previously complete graph. There is currently no signal a caller can assert on to distinguish a complete run from a partial one. A total backend failure at startup does exit 1; only the mid-run partial case is silent.
2. Retrieval returns no entity-graph results for actions canonicalised with underscores — The task decomposer emits some actions in an underscore-joined form (pick_up, move_and_position). Stored action_type values are gerund phrases, and the relaxed search clauses compare substrings, so an underscore form cannot match at any relaxation level (0–3) and the query falls through to the visual fallback. Single-word actions are unaffected — stir matches stirring, place matches placing down. Rephrasing the query so the action resolves to a single verb is an effective workaround. Deferred to a future release.

Egocentric Hand-Object Reconstruction Pipeline

We have added the following updates to the hand-object reconstruction pipeline.

Summary

• Added support for alternative hand-tracking pipeline based on WiLoR + SAM2 + HaMeR.
• Added support for gaussian splat based refinement of reconstruction results, reducing temporal jitter.
• Added support for running egocentric reconstruction with pre-made object mesh
• Added support for setting up and execution reconstruction with Agentic skills (Claude / Codex).
• Added support for gravity alignment of reconstruction results

Known Limitations

• 6565996 Object tracking results may be non-deterministic. This can result in different trajectories, and potentially object flipping across a run.
• 6208016 Hand tracking results may be jittery or over-smoothed in some instances, lagging the video.

Docs: reconstruction/docs/ego_e2e_setup.md

Robotic Grounding (NEW!)

Robotic Grounding is new in v0.2.0. It converts human-object motion into simulation scenes and robotic policies and data.

Summary

• End-to-end grounding pipeline — a host-side orchestrator runs dataset loading, post-processing, object-asset generation, IK, retargeting, support-surface reconstruction, visualization, Isaac smoke tests, and quality assessment across the required containers
• Robot and dataset coverage — hand motion can be retargeted to Sharpa Wave; v0.2 also supports G1 policy training from the included pre-processed partitions
• Portable motion and scenes — the schema carries robot trajectories, object/contact labels, coordinate-frame metadata, and IK diagnostics; scene assembly supports rigid and articulated objects plus reconstructed support surfaces
• Isaac Lab workflows — RL entry points cover zero-action scene smoke tests, PPO training, checkpoint evaluation, and policy export, with kinematic replay and static scene inspection for debugging
• Quality and visualization — geometric assessment, fail-fast validation for empty or stale output, interactive Viser/HTML playback, MP4 recording, and a browser gallery make processed sequences easier to review before training
• Agentic operation — Skills guide onboarding, command generation, diagnostics, and whole-body training workflows, while local and OSMO launch paths support single-machine and scaled runs

Known Assumptions

• An NVIDIA GPU and NVIDIA Container Toolkit are required; NVIDIA driver 580.126.09 and CUDA 13.0 are recommended for the robotic-grounding container
• Users must obtain the source datasets and license-gated MANO models separately and fetch Git LFS assets after cloning; ARCTIC additionally requires the articulated object assets distributed by ArtiGrasp

Known Limitations

• Whole-body retargeting (SOMA to G1) is deferred to 0.3. Raw SOMA input sequences are not included in this release and no staging procedure is documented for them. The example HOI sequences remain under legal review, so the retargeting path cannot be exercised end to end; only training on the pre-processed G1 partitions is in scope for 0.2.
• Kinematic replay is a validation and visualization path, not a physics-valid rollout: it teleports robot and object states rather than simulating their dynamics
• Auto-generated rigid-object URDFs are bootstrap assets with placeholder mass and inertia; calibrate them before drawing conclusions from contact dynamics

Docs: robotic_grounding/README.md