The Code is pytorch implementation of Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser
.
- Install python3.6 and pytorch=1.4.0.
- Install requirements
pip install -r requirements.txt
.
- Download preprocessed dataset and extracted features:
sh scripts/download_preprocessed.sh
- Download the retrieved entity information (entity_{split}_100.pt) and candidate image information ( candidate_{split}_100.json) from https://drive.google.com/drive/folders/1s93pcmxHkPYMx9XXycFuZklC_oGeCqEJ?usp=sharing and palce them at the directory
./data
.
We first train a Guesser agumented with negative samples and download the A-Bot provided in Visdial-Diversity. Since it is difficulty to independently investigate the QGen's performance, we indirectly evaluate QGen by observing the image-guessing performance (Guesser and A-Bot are fixed). Then, we train the QGen and A-Bot in RL.
sh scripts/train_guesser.sh <work_path> <save_name>
- <work_path>: the project path (default to .)
- <save_name>: the saved path
sh scripts/train_qgen.sh <work_path> <save_name> <guesser_path> <abot_path>
- <guesser_path>: the pre-trained guesser
- <abot_path>: the pre-trained abot
sh scripts/train_rl.sh <work_path> <save_name> <guesser_path> <abot_path> <qbot_path>
We evaluate the retrieval result under the image-guessing setting.
sh scripts/eval_sys <work_path> <guesser_path> <abot_path> <qbot_path>
- <work_path>: the project path (default to .)
- <guesser_path>: the path of Q-Bot
- <abot_path>: the path of A-Bot
- <qbot_path>: the path of Q-Bot
sh scripts/eval_abot.sh <work_path> <abot_path>
We report the accuracy of guesser given the 10-round human dialogue.
sh scripts/eval_guesser.sh <work_path> <guesser_path>
If you find this data is useful or use the data in your work, please cite our paper as well as Murahari et. al.,Improving Generative Visual Dialog by Answering Diverse Questions.
@misc{zheng2021enhancing,
title={Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser},
author={Duo Zheng and Zipeng Xu and Fandong Meng and Xiaojie Wang and Jiaan Wang and Jie Zhou},
year={2021},
eprint={2109.02297},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@inproceedings{murahari2019visdialdiversity,
title={Improving Generative Visual Dialog by Answering Diverse Questions},
author={Vishvak Murahari, Prithvijit Chattopadhyay, Dhruv Batra, Devi Parikh, Abhishek Das},
booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
year={2019}
}
Please contact Duo Zheng (zd[at].bupt.edu.cn) for questions and suggestions.