Feature resources of IJCAI 2020 paper "Diagnosing the Environment Bias in Vision-and-Language Navigation" and code snippet to adapt VLN codebase for using semantic features.
Download semantic features
Download semantic features by
Several kinds of semantic features will be downloaded:
Detection.tsvdetected object features
GT-Seg.tsvground-truth semantic segmentation features
Learned-Seg.tsvpredicted semantic features
To use the semantic features for your own VLN model
Put the downloaded image features in
img_features into the image feature directory of your VLN codebase.
modify.py into your VLN codebase and run
python modify.py --CODE_ROOT $CODE_SRC --REPLACE_FEAT $SEMANTIC_FEAT
$CODE_SRC is the path of your code source,
$SEMANTIC_FEAT is the type of semantic feature you would like to use (same notations as in our paper).
This process should make a copy of your original code and create a new version with the modifications for semantic features.
E.g., if you want to run the offical "Back Translation with Environmental Dropout" model using semantic features, after installing their repo from https://github.com/airsplay/R2R-EnvDrop, copy the downloaded semantic features into their
img_features folder. Put
modify.py into the repo, and run:
python modify.py --CODE_ROOT r2r_src --REPLACE_FEAT GT-Seg
After that, when you follow their instructions for training, the model will be trained with our ground-truth semantic segmentation features.
When running the "EnvDrop" agent (without back translation) using semantic features, we can get the following results:
|Feature type||Val seen (%)||Val unseen (%)|||gap||