VLN-Trans: Translator for the Vision and Language Navigation Agent

Paper (ACL 2023)

VLN-Trans (https://arxiv.org/pdf/2302.09230.pdf)

Installation

Install the Matterport3D Simulator.

Data Preparation

Please follow the instructions below to prepare the data in directories:

• MP3D navigability graphs: connectivity maps.

• Processed fine-grained R2R data and augmented: Fine-grained r2r data.

• Processed fine-grained R4R data and augmented: Fine-grained r4r data.

• R2R-Last: R2R-Last.

• Translator pre-train data (SyFiS): Translator pre-train data.

• MP3D image features: img features (ResNet-152-Places365).

Initial weights for VLN-trans and Translator

• VLN-trans: Download the pytorch_model.bin from here.
• Translator: Download the pre-train weights from here.

Trained Network Weights

• VLN-trans trained-weights

R2R Navigation

Please read Peter Anderson's VLN paper for the R2R Navigation task.

Test Navigator

To replicate the performance reported in our paper, load the trained network weights and run validation:

bash run_translator/test_agent.bash

Train Navigator

To train the network, run:

bash run_translator/train_agent.bash