Skip to content
PyTorch Code for NAACL 2019 paper: Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Branch: master
Clone or download
Latest commit 4c11585 Jun 13, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cmake update May 28, 2019
connectivity start env May 27, 2019
include update May 28, 2019
pybind11 @ 97784da start env May 27, 2019
r2r_src Add BT + EnvDrop May 29, 2019
run Update running files Jun 13, 2019
semantic_views Updating semantic views. May 26, 2019
src add r2r src Jun 2, 2019
tasks/R2R/data update May 28, 2019
.gitignore start env May 27, 2019
.gitmodules start env May 27, 2019
CMakeLists.txt update May 28, 2019
LICENSE Add code and licenses May 27, 2019
LICENSE_Matterport3DSimulator Add code and licenses May 27, 2019
README.md add r2r src Jun 2, 2019
python_requirements.txt update May 28, 2019

README.md

Code and Data for Paper "Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout"

Environment Installation

Download Room-to-Room navigation data:

bash ./tasks/R2R/data/download.sh

Download image features for environments:

mkdir img_features
wget https://www.dropbox.com/s/715bbj8yjz32ekf/ResNet-152-imagenet.zip -P img_features/
cd img_features
unzip ResNet-152-imagenet.zip

Python requirements: Need python3.6 (python 3.5 should be OK since I removed the allennlp dependencies)

pip install -r python_requirements.txt

Install Matterport3D simulators:

git submodule update --init --recursive 
sudo apt-get install libjsoncpp-dev libepoxy-dev libglm-dev libosmesa6 libosmesa6-dev libglew-dev
mkdir build && cd build
cmake -DEGL_RENDERING=ON ..
make -j8

Code

Speaker

bash run/speaker.bash 0

0 is the id of GPU. It will train the speaker and save the snapshot under snap/speaker/

Agent

bash run/agent.bash 0

0 is the id of GPU. It will train the agent and save the snapshot under snap/agent/. Unseen success rate would be around 46%.

Agent + Speaker (Back Translation)

After pre-training the speaker and the agnet,

bash run/bt_envdrop.bash 0

0 is the id of GPU. It will load the pre-trained agent and run back translation with environmental dropout.

Currently, the result with PyTorch 1.1 is a little bit lower than my NAACL reported number. It still easily reaches a success rate of 50% (+4% from w/o back translation).

Implementation Details

  1. When training the speaker and listener, we drop out features as much as we can. It means that the image feature are dropped randomly (with a smaller dropout rate), which has been seen used in multiple vision papers.
  2. The ml_weight is increased in using back translation, since the quality of generated sentence is not high and RL would be misled.
  3. Instead of training with augmented data and fine-tuning with training data, we trained them together.

Semantic Views

As shown in Fig.6 of our paper which is the same to

semantic_views/17DRP5sb8fy/10c252c90fa24ef3b698c6f54d984c5c/14.png 

semantic rgb

in this repo, we rendered semantic views from Matterport3D dataset. We provide a preview of semantic views and rgb views under the forder semantic_views.

To access the full rendered data, please first sign the Terms of Use agreement form in https://github.com/niessner/Matterport and cc' the email to us haotan@cs.unc.edu. And we would share a download link.

Thanks to the one who teaches me how to calibrate camera. Note that there would be a small pixel-level disagreement between the RGB view and semantic view, since the semantic view are rendered from 3D annotations while the RGB view are rendered from skyboxes. We are still aiming in solving it.

TODO's

  1. Provide test script for beam search. (Code is in train.py and agent.py)
  2. Release pre-trained snapshots.
  3. Check PyTorch 1.1 configurations.
  4. Update pip requirement with version specifications.
You can’t perform that action at this time.