🌐 Towards Spatio-Temporal World Scene Graph Generation
from Monocular Videos

Rohith Peddi, Saurabh, Shravan Shanmugam, Likhitha Pallapothula, Yu Xiang, Parag Singla, Vibhav Gogate

   

📧 For access to the ActionGenome4D dataset, please email rohith.peddi@utdallas.edu

_{🔄 This page is under continuous update}

---

📢 News & Updates

• [Jun 2026] — Final ActionGenome4D annotations and trained model checkpoints will be released in the coming weeks alongside an updated paper.
• [Jun 2026] — Released the 3D BBox Annotation Tool and the MLLM Pipeline used for constructing ActionGenome4D annotations.
• [May 2026] — ActionGenome4D annotations, trained checkpoints, and VLM evaluation code available upon request by email to rohith.peddi@utdallas.edu
• [Mar 2026] — Released the arXiv paper describing the WorldSGG framework.

---

📋 TODO — Upcoming Releases

• ActionGenome4D dataset annotations
• Trained model checkpoints (PWG, MWAE, 4DST)
• VLM / MLLM evaluation code
• 4D scene reconstruction pipeline code

---

🔍 Overview

1. World Scene Graph Generation (WSGG) Task

Comparison of the conventional frame-based Video Scene Graph Generation (left) which produces per-frame 2D scene graphs with the proposed World Scene Graph Generation task (right) which jointly reasons about 3D oriented bounding boxes and spatio-temporal relationships in a persistent world frame.

---

2. ActionGenome4D Dataset

_{Animated walkthrough of the WSGG task formulation}

The ActionGenome4D dataset construction pipeline, showing the geometric annotation stages (i) scene construction via π³ with bundle adjustment, (ii) floor determination via PromptHMR, (iii) 3D OBB construction via multi-scale erosion alongside the semantic annotation stages (iv) MLLM inference for relationship prediction and (v) custom relationship correction tool with a custom 4D annotation correction tool (vi) for manual quality assurance.

Human Mesh Determination

_{Human mesh estimation and floor alignment for scene 0DJ6R}

4D Object Reconstruction

_{4D static scene reconstruction with refined object masks for scene 0DJ6R}

---

3. 4D Scene Reconstruction Pipeline

The 4D scene reconstruction pipeline processes raw Action Genome videos through four stages: (i) adaptive frame sampling via SIFT + RANSAC homography, (ii) feed-forward 3D inference using π³ for both static and dynamic point clouds, (iii) static-dynamic scene decomposition, and (iv) per-frame geometric alignment via Trimmed ICP with Weighted Kabsch fitting — producing a unified 4D scene representation with refined camera poses and mask-aware merging.

---

4. Manual Relationship Correction

The manual relationship correction interface allows for human-in-the-loop review and fine-grained modification of generated relationships, ensuring high-quality ground-truth annotations.

Scene Graph Corrector — Part 1

_{World frame relationship annotation workflow}

Scene Graph Corrector — Part 2

_{Continued world frame relationship annotation}

---

5. Manual 3D Annotation Correction

The manual 3D annotation correction tool provides a 3D annotation interface for aligning reconstructed point clouds with the ground plane. Through a multi-step process of rotation and translation adjustments, annotators correct the floor alignment to ensure accurate world-frame coordinate systems for all objects in the scene.

Monocular 3D Annotations Corrections

_{Correcting monocular 3D bounding box annotations}

World Annotations Corrections

_{Correcting 3D oriented bounding box annotations in the world frame}

---

6. Annotation Tools

The following open-source tools were developed for constructing the ActionGenome4D annotations:

Tool	Description
3D BBox Annotation Tool	Interactive 3D bounding-box annotation and correction interface for point-cloud scenes
MLLM Pipeline	Multi-modal LLM pipeline for automated relationship annotation and human-in-the-loop correction

3D BBox Pipeline — Demo Videos

FrameBBox Annotation	WorldBBox Annotation	End-to-End Pipeline
ReleaseV1.mp4	ReleaseV1_3.mp4	ReleaseV1_2.mp4

---

7. WorldWise: WSGG Model Architecture

The WorldWise architecture operates in two stages: Stage 1 performs monocular 3D detection using DINOv3 features with a factorized 3D head to produce 2D bounding boxes, 3D OBB parameters, and class logits. Stage 2 generates the world scene graph through four specialized encoders — object spatial encoder, object motion encoder, global structural encoder, and camera temporal encoder — followed by a masked autoencoder for unobserved object representation, and spatio-temporal decoders for relationship classification.

---

8. WorldRAG: MLLM Evaluation Pipeline

The WorldRAG pipeline leverages Vision Language Models for unlocalized world scene graph generation. It consists of three modules: (a) a Coarse Event Graph Construction module that segments video into key frame segments and builds an event graph with entity, action, and scene nodes, (b) an Object Discovery module that identifies objects in the world using VLM-based embedding similarity matching, and (c) a Graph RAG module that retrieves and re-ranks relevant event graph nodes for relationship prediction via a Large Language Model.

---

🙏 Acknowledgements

This code builds upon the following excellent repositories. We thank all the authors for releasing their code.

Repository	Description
Pi3	3D object detection
PromptHMR	Human mesh recovery
Cut3R	3D scene reconstruction
RAFT	Optical flow estimation
DepthAnything	Monocular depth estimation
UniDepth	Universal depth estimation

---

📄 Citation

If you find this work useful in your research, please consider citing:

@misc{peddi2026spatiotemporalworldscenegraph,
      title={Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos}, 
      author={Rohith Peddi and Saurabh and Shravan Shanmugam and Likhitha Pallapothula and Yu Xiang and Parag Singla and Vibhav Gogate},
      year={2026},
      eprint={2603.13185},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.13185}, 
}