Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 

README.md

Diagnosing-EnvBias-VLN

Feature resources of IJCAI 2020 paper "Diagnosing the Environment Bias in Vision-and-Language Navigation" and code snippet to adapt VLN codebase for using semantic features.

Download semantic features

Download semantic features by

bash ./img_features/download.sh

Several kinds of semantic features will be downloaded:

  1. ImageNet.tsv ImageNet-1000 features
  2. Detection.tsv detected object features
  3. GT-Seg.tsv ground-truth semantic segmentation features
  4. Learned-Seg.tsv predicted semantic features

To use the semantic features for your own VLN model

Put the downloaded image features in img_features into the image feature directory of your VLN codebase.

Put modify.py into your VLN codebase and run

python modify.py --CODE_ROOT $CODE_SRC --REPLACE_FEAT $SEMANTIC_FEAT

where $CODE_SRC is the path of your code source, $SEMANTIC_FEAT is the type of semantic feature you would like to use (same notations as in our paper).

This process should make a copy of your original code and create a new version with the modifications for semantic features.

E.g., if you want to run the offical "Back Translation with Environmental Dropout" model using semantic features, after installing their repo from https://github.com/airsplay/R2R-EnvDrop, copy the downloaded semantic features into their img_features folder. Put modify.py into the repo, and run:

python modify.py --CODE_ROOT r2r_src --REPLACE_FEAT GT-Seg

After that, when you follow their instructions for training, the model will be trained with our ground-truth semantic segmentation features.

Result samples

When running the "EnvDrop" agent (without back translation) using semantic features, we can get the following results:

Feature type Val seen (%) Val unseen (%) |gap|
ResNet 58.4 45.8 12.6
ImageNet 45.6 45.4 0.2
Detection 51.2 49.9 1.3
GT-Seg 48.3 48.9 0.6
Learned-Seg 49.4 46.0 3.4

About

Feature resources of "Diagnosing the Environment Bias in Vision-and-Language Navigation"

Resources

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.