Release v0.1.0
Video To Data v0.1.
Highlights
Initial release of Video to Data, including Video Ingestion Agent and Egocentric Hand-Object reconstruction pipeline.
Video Ingestion Agent (New!)
An agentic workflow that turns robot demonstration videos into a queryable action database (graph.db + vector.db), with natural-language retrieval and an optional Gradio web UI.
Summary
- Ingestion pipeline — LangGraph workflow: segment → verify/refine → entity graph → SigLIP-2 embeddings → SQLite → HTML report
- Retrieval agent — EGAgent-style search with task decomposition and progressive query relaxation
- Gradio webapp — Ingest, Query, Database, Settings, and optional Reconstruct tab (ego hand + object reconstruction)
- Batch ingestion — Multi-GPU sharded processing into shared WAL databases
- Model backends — vLLM (default, pinned to 0.12.x), local HuggingFace, or remote API (NIM / OpenAI-compatible)
Known Limitations
Fixes planned for a future release.
- Total API auth failure still exits 0 — If all VLM/LLM calls fail (e.g. bad NIM_API_KEY), the pipeline may still exit 0 and write placeholder segments/embeddings. Validate credentials first; inspect graph.db after each run.
- Mid-run vLLM loss is silent — Server disconnects are swallowed; the pipeline continues with partial data and exits 0. Monitor vLLM health and verify entity counts after long runs.
- Retrieval action mismatch — Decomposer emits snake_case actions (pick_up); ingestion stores gerund phrases (picking up and placing down). Graph search returns nothing; results rely on weak visual fallback. save_traces: true does not write trace files.
- --share often renders blank — Upstream Gradio CORS issue on *.gradio.live URLs (~17% success in EVT-01). Use SSH port forwarding (ssh -L 7860:127.0.0.1:7860 ) instead.
Docs: video_ingestion_agent/README.md
Egocentric Hand-Object Reconstruction Pipeline (New!)
End-to-end egocentric hand + object reconstruction from a single monocular RGB video and a text prompt naming the manipulated object.
Summary
- Inputs / outputs: takes a monocular egocentric RGB video and object prompt;
outputs per-frame MANO hand pose, a textured object mesh, object 6-DoF poses,
and a shared roughly metric hand-object trajectory. - Calibration: optionally runs AnyCalib undistortion, then computes MoGe depth
and stabilized intrinsics as the default metric anchor. - Object reconstruction / tracking: detects the prompted object with Grounding
DINO, tracks masks with SAM2, reconstructs a textured mesh with SAM3D, and
tracks/smooths object 6-DoF poses with FoundationPose and EKF filtering. - Hand reconstruction: recovers per-frame MANO hand pose with ViPE + Dyn-HaMR
and converts Dyn-HaMR depth/intrinsics into the pipeline format. - Hand object alignment: aligns hand tracks and object poses to the selected
depth source for a shared roughly metric hand-object trajectory.
Known Assumptions
- GPU requirement: one NVIDIA GPU with at least 24 GB VRAM and NVIDIA Container
Toolkit installed. - The object must be clearly visible in the reference frame, and the prompt must
match that object. Grounding DINO and SAM3D both depend on this single frame;
occlusion, truncation, blur, or an incorrect prompt can cause no detection or
a degraded mesh. - SAM3D reconstructs from one masked view, so unseen or occluded geometry may be
hallucinated. The visible silhouette is usually the most reliable part.
Known Limitations
- Object scale may be off: reconstructed objects can render smaller or larger
than their real-world size. (6244025) - Hands in video are assumed upright: hands entering from the side or with
unusual orientation may not reconstruct correctly. (6243962) - Object may lose tracking during fast movement: rapid motion can introduce
jitter, lag, or loss of synchronization between the object and hand. (6208016)