Release v0.2.0

Video Ingestion Agent

What's new

• Cosmos3-Nano backend — config-only model switch; scripts/serve.py auto-detects the model family and applies the required vLLM flags (Reasoner tower only, ~8B-tier footprint)
• Unified locked environment — one uv.lock across Docker and host: vLLM 0.21 (up from the 0.12.x pin), transformers 5.x, torch 2.11 + CUDA 13
• Local backend restored — in-process HuggingFace inference works again (torchcodec video decoding); offline single-GPU workflows no longer require a vLLM server
• Configurable API endpoint — models.api_url overrides the api backend gateway; 401/403 now fail immediately with an actionable message instead of retrying silently
• Database integrity — re-ingesting a video replaces its previous rows in place (no more orphaned segments/entities); SQLite foreign-key enforcement with ON DELETE CASCADE; databases from older versions migrate automatically on first ingest
• Loud failures — a run that produces zero clips exits 1 and writes nothing (no placeholder segments, no resume marker), so scripted callers can detect it

Known limitations

1. Partial model-call failure is not reported — If the model backend becomes unavailable partway through an ingestion run, the clips processed after that point are skipped and the run still reports Pipeline completed successfully! with exit code 0. On a re-ingestion of a video already in the database, the previous run's rows are purged first, so the incomplete results replace a previously complete graph. There is currently no signal a caller can assert on to distinguish a complete run from a partial one. A total backend failure at startup does exit 1; only the mid-run partial case is silent.
2. Retrieval returns no entity-graph results for actions canonicalised with underscores — The task decomposer emits some actions in an underscore-joined form (pick_up, move_and_position). Stored action_type values are gerund phrases, and the relaxed search clauses compare substrings, so an underscore form cannot match at any relaxation level (0–3) and the query falls through to the visual fallback. Single-word actions are unaffected — stir matches stirring, place matches placing down. Rephrasing the query so the action resolves to a single verb is an effective workaround. Deferred to a future release.

Egocentric Hand-Object Reconstruction Pipeline

We have added the following updates to the hand-object reconstruction pipeline.

Summary

• Added support for alternative hand-tracking pipeline based on WiLoR + SAM2 + HaMeR.
• Added support for gaussian splat based refinement of reconstruction results, reducing temporal jitter.
• Added support for running egocentric reconstruction with pre-made object mesh
• Added support for setting up and execution reconstruction with Agentic skills (Claude / Codex).
• Added support for gravity alignment of reconstruction results

Known Limitations

• 6565996 Object tracking results may be non-deterministic. This can result in different trajectories, and potentially object flipping across a run.
• 6208016 Hand tracking results may be jittery or over-smoothed in some instances, lagging the video.

Docs: reconstruction/docs/ego_e2e_setup.md

Robotic Grounding (NEW!)

Robotic Grounding is new in v0.2.0. It converts human-object motion into simulation scenes and robotic policies and data.

Summary

• End-to-end grounding pipeline — a host-side orchestrator runs dataset loading, post-processing, object-asset generation, IK, retargeting, support-surface reconstruction, visualization, Isaac smoke tests, and quality assessment across the required containers
• Robot and dataset coverage — hand motion can be retargeted to Sharpa Wave; v0.2 also supports G1 policy training from the included pre-processed partitions
• Portable motion and scenes — the schema carries robot trajectories, object/contact labels, coordinate-frame metadata, and IK diagnostics; scene assembly supports rigid and articulated objects plus reconstructed support surfaces
• Isaac Lab workflows — RL entry points cover zero-action scene smoke tests, PPO training, checkpoint evaluation, and policy export, with kinematic replay and static scene inspection for debugging
• Quality and visualization — geometric assessment, fail-fast validation for empty or stale output, interactive Viser/HTML playback, MP4 recording, and a browser gallery make processed sequences easier to review before training
• Agentic operation — Skills guide onboarding, command generation, diagnostics, and whole-body training workflows, while local and OSMO launch paths support single-machine and scaled runs

Known Assumptions

• An NVIDIA GPU and NVIDIA Container Toolkit are required; NVIDIA driver 580.126.09 and CUDA 13.0 are recommended for the robotic-grounding container
• Users must obtain the source datasets and license-gated MANO models separately and fetch Git LFS assets after cloning; ARCTIC additionally requires the articulated object assets distributed by ArtiGrasp

Known Limitations

• Whole-body retargeting (SOMA to G1) is deferred to 0.3. Raw SOMA input sequences are not included in this release and no staging procedure is documented for them. The example HOI sequences remain under legal review, so the retargeting path cannot be exercised end to end; only training on the pre-processed G1 partitions is in scope for 0.2.
• Kinematic replay is a validation and visualization path, not a physics-valid rollout: it teleports robot and object states rather than simulating their dynamics
• Auto-generated rigid-object URDFs are bootstrap assets with placeholder mass and inertia; calibrate them before drawing conclusions from contact dynamics

Docs: robotic_grounding/README.md

Release v0.1.0

Video To Data v0.1.

Highlights

Initial release of Video to Data, including Video Ingestion Agent and Egocentric Hand-Object reconstruction pipeline.

Video Ingestion Agent (New!)

An agentic workflow that turns robot demonstration videos into a queryable action database (graph.db + vector.db), with natural-language retrieval and an optional Gradio web UI.

Summary

• Ingestion pipeline — LangGraph workflow: segment → verify/refine → entity graph → SigLIP-2 embeddings → SQLite → HTML report
• Retrieval agent — EGAgent-style search with task decomposition and progressive query relaxation
• Gradio webapp — Ingest, Query, Database, Settings, and optional Reconstruct tab (ego hand + object reconstruction)
• Batch ingestion — Multi-GPU sharded processing into shared WAL databases
• Model backends — vLLM (default, pinned to 0.12.x), local HuggingFace, or remote API (NIM / OpenAI-compatible)

Known Limitations

Fixes planned for a future release.

• Total API auth failure still exits 0 — If all VLM/LLM calls fail (e.g. bad NIM_API_KEY), the pipeline may still exit 0 and write placeholder segments/embeddings. Validate credentials first; inspect graph.db after each run.
• Mid-run vLLM loss is silent — Server disconnects are swallowed; the pipeline continues with partial data and exits 0. Monitor vLLM health and verify entity counts after long runs.
• Retrieval action mismatch — Decomposer emits snake_case actions (pick_up); ingestion stores gerund phrases (picking up and placing down). Graph search returns nothing; results rely on weak visual fallback. save_traces: true does not write trace files.
• --share often renders blank — Upstream Gradio CORS issue on *.gradio.live URLs (~17% success in EVT-01). Use SSH port forwarding (ssh -L 7860:127.0.0.1:7860 ) instead.

Docs: video_ingestion_agent/README.md

Egocentric Hand-Object Reconstruction Pipeline (New!)

End-to-end egocentric hand + object reconstruction from a single monocular RGB video and a text prompt naming the manipulated object.

Summary

• Inputs / outputs: takes a monocular egocentric RGB video and object prompt;
  outputs per-frame MANO hand pose, a textured object mesh, object 6-DoF poses,
  and a shared roughly metric hand-object trajectory.
• Calibration: optionally runs AnyCalib undistortion, then computes MoGe depth
  and stabilized intrinsics as the default metric anchor.
• Object reconstruction / tracking: detects the prompted object with Grounding
  DINO, tracks masks with SAM2, reconstructs a textured mesh with SAM3D, and
  tracks/smooths object 6-DoF poses with FoundationPose and EKF filtering.
• Hand reconstruction: recovers per-frame MANO hand pose with ViPE + Dyn-HaMR
  and converts Dyn-HaMR depth/intrinsics into the pipeline format.
• Hand object alignment: aligns hand tracks and object poses to the selected
  depth source for a shared roughly metric hand-object trajectory.

Known Assumptions

• GPU requirement: one NVIDIA GPU with at least 24 GB VRAM and NVIDIA Container
  Toolkit installed.
• The object must be clearly visible in the reference frame, and the prompt must
  match that object. Grounding DINO and SAM3D both depend on this single frame;
  occlusion, truncation, blur, or an incorrect prompt can cause no detection or
  a degraded mesh.
• SAM3D reconstructs from one masked view, so unseen or occluded geometry may be
  hallucinated. The visible silhouette is usually the most reliable part.

Known Limitations

• Object scale may be off: reconstructed objects can render smaller or larger
  than their real-world size. (6244025)
• Hands in video are assumed upright: hands entering from the side or with
  unusual orientation may not reconstruct correctly. (6243962)
• Object may lose tracking during fast movement: rapid motion can introduce
  jitter, lag, or loss of synchronization between the object and hand. (6208016)

Docs: reconstruction/docs/ego_e2e_setup.md