Habitat Challenge is an autonomous navigation challenge that aims to benchmark and accelerate progress in embodied AI. In its first iteration, Habitat Challenge 2019 is based on the PointGoal task as defined by Anderson et al. .
To participate in the challenge follow the instructions at link. The challenge is built on top of the habitat stack. The winners of challenge will be announced at CVPR-2019 at the Habitat: Embodied Agents Challenge and Workshop. Details on the prizes coming soon. If you are interested in participating please fill out your name and email address in this form and we will get back to you when the submission phase goes live.
One fundamental difficulty in creating a challenge around embodied AI tasks is the transition from static predictions (as in supervised learning) to sequential decision making (as in reinforcement learning). In classical 'internet AI' challenges (e.g. ImageNet, COCO, VQA), it is possible to release a static testing dataset and ask participants to simply upload their predictions on this set. Embodied AI tasks (navigation, instruction following, question answering) typically involve sequential decision making and agent-driven control, making it infeasible to pre-package a testing dataset. Essentially, embodied AI challenges require participants to upload code not predictions. The uploaded agents may then be evaluated in novel (unseen) test environments.
We leverage the frontend and challenge submission process of the EvalAI platform, and build backend infrastructure ourselves. Participants in Habitat Challenge are asked to upload (on EvalAI) Docker containers with their agents that are then evaluated on a live AWS GPU-enabled instance.
Participation and Starter Code
The challenge is hosted on EvalAI, to participate please refer to the instructions. To get started:
- Install the Habitat-Sim and Habitat-API packages.
- Download the Gibson dataset following the instructions here. After downloading extract the dataset to folder
habitat-api/data/scene_datasets/gibson/folder (this folder should contain the
.glbfiles from gibson). Note that the
habitat-apifolder is the habitat-api repository folder.
- Download the dataset for Gibson pointnav from link and place it in the folder
habitat-api/data/datasets/pointnav/gibson. If placed correctly, you should have the train and val splits at
- An example PPO baseline for the Pointnav task is present at
habitat-api/baselines. To start training on the Gibson dataset using this implementation follow instructions in the README. Set
tasks/pointnav_gibson.yamlto train on Gibson data. This is a good starting point for participants to start building their own models. The PPO implementation contains initialization and interaction with the environment as well as tracking of training statistics. Participants can borrow the basic blocks from this implementation to start building their own models.
- To evaluate a trained PPO model on Gibson val split run
evaluate_ppo.pyusing instructions in the README at
tasks/pointnav_gibson.yaml. The evaluation script will report SPL metric.
- You can also use the general benchmarking script present at
tasks/pointnav_gibson.yamland inside the file
- Instructions for submission will be made available soon when our challenge submission infrastructure goes live.
The objective of the agent is to navigate successfully to a target location specified by agent-relative Euclidean coordinates (e.g. "Go 5m north, 3m west relative to current location"). Importantly, updated goal-specification (relative coordinates) is provided at all times (as the episode progresses) and not just at the outset of an episode. The action space for the agent consists of turn-left, turn-right, move forward and STOP actions. The turn actions produce a turn of 10 degrees and move forward action produces a linear displacement of 0.25m. The STOP action is used by the agent to indicate completion of an episode. We use an idealized embodied agent with a cylindrical body with diameter 0.2m and height 1.5m. The challenge consists of two tracks, RGB (only RGB input) and RGBD (RGB and Depth inputs).
We create a set of PointGoal navigation episodes for the Gibson  3D scenes as the main dataset for the challenge. Gibson was preferred over SUNCG because unlike SUNCG it contains scans of real-world indoor environments. Gibson was chosen over Matterport3D because unlike Matterport3D Gibson's raw meshes are not publicly available allowing us to sequester a test set. We use the splits provided by the Gibson dataset, retaining the train, and val sets, and separating the test set into test-standard and test-challenge. The train and val scenes are provided to participants. The test scenes are used for the official challenge evaluation and are not provided to participants.
After calling the STOP action, the agent is evaluated using the "Success weighted by Path Length" (SPL) metric .
An episode is deemed successful if on calling the STOP action, the agent is within 0.2m of the goal position. The evaluation will be carried out in completely new houses which are not present in training and validation splits.
The Habitat challenge would not have been possible without the infrastructure and support of EvalAI team and the data of Gibson team. We are especially grateful to Rishabh Jain, Deshraj Yadav, Fei Xia and Amir Zamir.
Habitat-Challenge is MIT licensed. See the LICENSE file for details.
 P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir, "On evaluation of embodied navigation agents," arXiv:1807.06757, 2018
 F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, "Gibson env: Real-world perception for embodied agents," in CVPR, 2018