Important

EGO-EXO4D DATASET ANNOUNCEMENT: Ego-Exo4D V2 is now available to the public. V2 contains 1286.30 video hours (221.26 ego-hours) across 5035 takes with more annotations Please refer to the changelog for details on what has changed.

EGO4D UPDATE: V2.1 has been released due to the addition of the Goal-Step annotations and accompanying "grouped videos". Please refer to the documentation for more information.

Ego4D & Ego-Exo4D

Ego-Exo4D is a large-scale multi-modal multi-view video dataset (including 3D) and benchmark challenge. The dataset consists of time-synchronized videos of participants recorded with at least one first-person (egocentric Aria glasses) and third-person (exocentric GoPro cameras) perspective cameras.

• Please refer to the website, documentation, paper, blog post and video introduction.

Ego4D is the world's largest egocentric (first person) video ML dataset and benchmark suite, including over 3700 hours of annotated first-person video data.

• Please refer to the website, documentation or paper for more information.

Getting Started

• To access the data, please refer to the Documentation:
  • For Ego-Exo4D: refer to the Getting Started page.
  • For Ego4D: refer to the Start Here page.
• To download the data, refer to:
  • For Ego-Exo4D: Ego-Exo4D's Downloader CLI README
  • For Ego4D: Ego4D's CLI README
• Explore Ego4D or Ego-Exo4D here (you'll need a license): Ego4D Visualizer
• Read the Summary below for details about this repository.

Summary

The Ego4d repository (ego4d python module) includes:

• Ego-Exo4D Downloader CLI for the Ego-Ego4D dataset (available as the command egoexo)
• Ego4D Downloader CLI for the Ego4D dataset (available as the command ego4d)
• A simple API abstracting common video reading libraries (TorchAudio, PyAV),
• An API for feature extraction, which includes lightweight wrappers for common models, such as: Omnivore and SlowFast
• Notebooks (for Ego4D and Ego-Exo4D) serving as examples/tutorials to analyze & use the dataset
  • Colab notebooks for Ego4D serve as additional examples for the benchmarks (VQ, NLQ and STA) can be found on: https://ego4d-data.org/docs/challenge/
• Research code to train models on the dataset, e.g. clep
  • NOTE: baseline code for Ego-Exo4D is coming soon!
  • NOTE: baseline code for the Ego4D benchmarks exists on separate GitHub repositories, see the EGO4D organization and docs

Please see Structure of the Repository below for details.

Setup

Please follow the below instructions to setup the downloader CLI for Ego4d and to install the ego4d python module.

Option 1: From the PyPi package

Tip

Please ensure you have a conda or pyenv environment created & activated. If you're unsure on how to do so, you can follow Option 2: Step 1.

pip install ego4d --upgrade

NOTE: Please ensure you are on at least Python 3.10

Option 2: Clone/Download the Code

Ensure you have cloned or downloaded the code to your local disk. All instructions assume you are the root of the repository.

Step 1: Create or Use an Environment

Create a conda environment to enable pip installation:

conda create -n ego4d python=3.11 -y
conda activate ego4d

If you are using an existing conda (or pyenv) environment: please ensure you have installed at least Python 3.10.

Step 2:

pip install .  # run from the root of Ego4d

Now you should be able to import ego4d:

python3 -c 'import ego4d; print(ego4d)'

You can check that the ego4d module links to the correct file on your file system from the output of the above command.

Structure of the Repository

The repository contains multiple directories covering a specific theme. Each theme contains an associated README.md file, please refer to them.

All python code is located in the ego4d and associated subdirectories. The goal for each subdirectory is to cover one specific theme.

• ego4d: the ego4d python module exists
  • cli: The Ego4D CLI for downloading the dataset
  • features: Feature extraction across the dataset
  • research: Everything related to research and usage of the dataset (dataloaders, etc).
    • research/clep: Contrastive Language Ego-centric video Pre-training
• viz: visualization engine

Visualization and Demo

• For a demo notebook: Annotation Notebook
• For the visualization engine: Viz README

License

Ego4D is released under the MIT License.