Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
run
 
 
 
 
 
 
 
 
 
 

Entity-Graph-VLN

Code of the NeurIPS 2020 paper: Language and Visual Entity Relationship Graph for Agent Navigation
Yicong Hong, Cristian Rodriguez-Opazo, Yuankai Qi, Qi Wu, Stephen Gould

[Paper] [Supplemental] [GitHub]

Prerequisites

Installation

Install the Matterport3D Simulator.

Please find the versions of packages in our environment here. In particular, we use:

  • Python 3.6.9
  • NumPy 1.18.1
  • OpenCV 3.4.2
  • PyTorch 1.3.0
  • Torchvision 0.4.1

Data Preparation

Please follow the instructions below to prepare the data in directories:

Trained Network Weights

R2R Navigation

Please read Peter Anderson's VLN paper for the R2R Navigation task.

Our code is based on the code structure of the EnvDrop.

Reproduce Testing Results

To replicate the performance reported in our paper, load the trained network weights and run validation:

bash run/agent.bash

Training

Navigator

To train the network from scratch, first train a Navigator on the R2R training split:

Modify run/agent.bash, remove the argument for --load and set --train listener. Then,

bash run/agent.bash

The trained Navigator will be saved under snap/.

Speaker

You also need to train a Speaker for augmented training:

bash run/speak.bash

The trained Speaker will be saved under snap/.

Augmented Navigator

Finally, keep training the Navigator with the mixture of original data and augmented data:

bash run/bt_envdrop.bash

We apply a one-step learning rate decay to 1e-5 when training saturates.

Citation

If you use or discuss our Entity Relationship Graph, please cite our paper:

@article{hong2020language,
  title={Language and Visual Entity Relationship Graph for Agent Navigation},
  author={Hong, Yicong and Rodriguez, Cristian and Qi, Yuankai and Wu, Qi and Gould, Stephen},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

About

Code of paper: Language and Visual Entity Relationship Graph for Agent Navigation

Topics

Resources

Releases

No releases published

Packages

No packages published