EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation

This repository contains the anonymized code for the EgoInteract simulator used in the paper submission.

The pipeline is built on Unity3D and leverages the Unity Perception package to generate large-scale, photorealistic synthetic datasets for Interaction Understanding and Anticipation. It integrates assets from HM3D (environments) and Objaverse-XL (objects) to create diverse interaction scenarios.

Dataset

The dataset associated with this simulator is hosted on Hugging Face.

Installation

Prerequisites

• Unity Hub and Unity Editor (Recommended version: 6000.0.66f2).
• FinalIK

Steps

1. Clone the repository:

2. Open the Project:

   • Add the project to Unity Hub.
   • Open it using the Unity Editor. Wait for the packages (Perception, etc.) to install automatically.

3. Data Setup (Important):

   • Objaverse-XL: Download the object assets and place them in Assets/Models/Objaverse-XL (or follow the specific path in your project).
   • HM3D Environments: Due to licensing and size, HM3D assets must be downloaded separately. Import the GLB/FBX files into Assets/Prefabs/Environments.

Quick Start

We provide a sample environment and a set of test objects directly included in the repository. This allows you to run and verify the pipeline immediately without downloading the full external datasets.

1. Open the main simulation scene: Assets/Scenes/TAS.unity.
2. Press the Play button in the Unity Editor.
   • The simulation will start loading environments, placing characters, and capturing frames.
   • Data (images, JSON annotations, semantic masks) will be generated in the Perception/Output/ directory by default.

Note: For full-scale generation, you will need to download and import the complete HM3D and Objaverse-XL datasets as described in the Data Setup section.

License

This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).