Skip to content

yeezhu/CMN.pytorch

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 

PyTorch implementation of Cross-modal Memory Network

Vision-Dialog Navigation by Exploring Cross-modal Memory, CVPR 2020.

Demo

Requirements

We recommend using the mattersim Dockerfile to install the simulator.

Dataset Download

Download the train, val_seen, val_unseen, and test splits of the CVDN and NDH dataset by executing:

sh tasks/CVDN/data/download.sh
sh tasks/NDH/data/download.sh

Installation

Build the docker image:

docker build -t mattersim .

Run the docker container, mounting your project path:

nvidia-docker run -it --shm-size 64G -v /User/home/Path_To_Project/:/Workspace/ mattersim

Compile the codebase:

mkdir build && cd build
cmake -DEGL_RENDERING=ON ..
make

Install python dependencies by running:

pip install -r tasks/NDH/requirements.txt

Train and Evaluate

To train and evaluate with trusted supervision, sample feedback, and all dialog history:

python tasks/NDH/train.py \
    --path_type=trusted_path \
    --history=all \
    --feedback=sample \
    --eval_type=val \
    --prefix=v1

Train and test with trusted supervision, sample feedback, and all dialog history:

python tasks/NDH/train.py \
    --path_type=trusted_path \
    --history=all \
    --feedback=sample \
    --eval_type=test \
    --prefix=v1

To generate a summary of the agent performance:

python tasks/NDH/summarize_perf.py

Citation

If you use the code in your research, please cite:

@inproceedings{zhu2020vision,
  title={Vision-Dialog Navigation by Exploring Cross-modal Memory},
  author={Zhu, Yi and Zhu, Fengda and Zhan, Zhaohuan and Lin, Bingqian and Jiao, Jianbin and Chang, Xiaojun and Liang, Xiaodan},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10730--10739},
  year={2020}
}

Acknowledgements

This repository is built upon the Matterport3DSimulator, CVDN and DAN-VisDial codebase.

About

PyTorch implementation of "Vision-Dialog Navigation by Exploring Cross-modal Memory", CVPR 2020.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published