Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.
json_2.1.0 adding dataset jsons Dec 12, 2019
splits model and data gen init Dec 15, 2019 updating Dec 19, 2019 download link fix Dec 18, 2019 updating Dec 16, 2019


The ALFRED dataset contains 8k+ expert demostrations with 3 or more language annotations each. A trajectory consists of a sequence of expert actions, the corresponding image observations, and language annotations describing segments of the trajectory.


We provide a download script with three options:

1. Lite (93MB): Trajectory JSONs. Images need to be generated with and Resnet feats with

$ cd $ALFRED_ROOT/data
$ sh json

2. Modeling Quickstart (~198GB) - Recommended: Trajectory JSONs and Resnet Features

$ sh json_feat

3. Full Dataset (~410GB) - Trajectory JSONs, Raw Images, PDDL States, Videos, Resnet Features

$ sh full

Alternatively, if prefer to use something other than wget, you can directly use the download links in

File Structure

data/train/task_type-object-movableReceptacle-receptacle-sceneNum/trial_ID/                  (trajectory root)
data/train/task_type-object-movableReceptacle-receptacle-sceneNum/trial_ID/traj_data.json    (trajectory metadata)
data/train/task_type-object-movableReceptacle-receptacle-sceneNum/trial_ID/      (Resnet18 features)
data/train/task_type-object-movableReceptacle-receptacle-sceneNum/trial_ID/problem_x.pddl    (pddl state)
data/train/task_type-object-movableReceptacle-receptacle-sceneNum/trial_ID/video.mp4         (video sequence)
data/train/task_type-object-movableReceptacle-receptacle-sceneNum/trial_ID/raw_images/       (images from trajectory)

JSON Structure

Dictionary sturcture of traj_data.json:

Task Info:

['task_id'] = "trial_00003_T20190312_234237"        (unique trajectory ID)
['task_type'] = "pick_heat_then_place_in_recep"     (one of 7 task types)
['pddl_params'] = {'object_target': "AlarmClock",   (object)
                   'parent_target': "DeskLamp",     (receptacle)
                   'mrecep_target': "",             (movable receptacle)
                   "toggle_target": "",             (toggle object)
                   "object_sliced": false}          (should the object be sliced?)

Scene Info:

['scene'] =  {'floor_plan': "FloorPlan7",           (THOR scene name)
              'scene_num' : 7,                      (THOR scene number)
              'random_seed': 3810970210,            (seed for initializing object placements)
              'init_action' : <API_CMD>,            (called to set the starting position of the agent)
              'object_poses': <LIST_OBJS>,          (initial 6DOF poses of objects in the scene)
              'object_toggles': <LIST_OBJS>}        (initial states of togglable objects)

Language Annotations:

['turk_annotations']['anns'] =  
             [{'task_desc': "Examine a clock using the light of a lamp.",                 (goal instruction) 
               'high_descs': ["Turn to the left and move forward to the window ledge.",   (list of step-by-step instructions)
                              "Pick up the alarm clock on the table", ...],               
               'votes': [1, 1, 1]                                                         (AMTurk languauge quality votes)

Expert Demonstration:

['plan'] = {'high_pddl':
                ["high_idx": 4,                          (high-level subgoal index)
                     {"action": "PutObject",             (discrete high-level action)
                      "args": ["bread", "microwave"],    (discrete params)
                 "planner_action": <PDDL_ACTION> ],      (PDDL action)
                ["high_idx": 1,                          (high-level subgoal index)
                     {"action": "PickupObject",          (discrete low-level action)
                          {"bbox": [180, 346, 332, 421]} (bounding box for interact action)
                           "mask": [0, 0, ... 1, 1]},    (compressed pixel mask for interact action)
                 "api_action": <API_CMD> ],              (THOR API command for replay)


['images'] = [{"low_idx": 0,                    (low-level action index)
               "high_idx": 0,                   (high-level action index)
               "image_name": "000000000.png"}   (image filename)

Extracting Resnet Features

To extract Resnet features from raw image sequences:

$ python models/utils/ --data data/full_2.1.0 --batch 32 --gpu --visual_model resnet18 --filename

This will save files insides each trajectory root folder.

You can’t perform that action at this time.