The impact of answers in visual dialog

Base code for the paper The impact of answers in visual dialog at ReInAct2021.

Based on the code of Ravi Shekhar for Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat.

You can get the visual features of the images and target objects here.

You can get the GuessWhat?! dataset here.

Training Oracle Models

Configure experiment in config/Oracle/config.json or create your own config file.

python -m train.Oracle.train \
    -exp_name <EXPERIMENT_NAME> \
    -config <CONFIG_PATH> \
    -img_feat 'rss' \ % used if you need visual features
    -bin_name <BIN_MODEL_WEIGHTS_NAME> 

LXMERT Oracle

Follow this repository for the LXMERT Oracle implementation.

Training Guesser Model

Configure the experiment in config/SL/config.json or create your own.

python -m train.SL.train \
    -exp_name <EXPERIMENT_NAME>

Generate answers

python -m utils.predict_n_save \
    -config <CONFIG_PATH> \
    -split <SPLIT> \
    -model_name <MODEL_NAME> \
    -out_fname <NAME>.csv.gz \
    -img_feat 'rss' \ % used if you need visual features
    -bin_path <BIN_MODEL_WEIGHTS_NAME> 

Running performance report

Configure the experiment in config/GamePlay/config.json. Make sure that ensemble.json and oracle.json have the same hyperparams set as in the training guesser model and the training oracle model steps.

Precomputed answers for each model are available in data/model_answers.csv.gz.

python -m train.GamePlay.human_eval \                                                                       
    -model_filename <ENSEMBLE_PATH> \
    -model_ans_path <PATH_TO_ANSWERS> \
    -batch_size <BATCH_SIZE> \
    -exp_name <EXPERIMENT_NAME> \
    -use_model <MODEL_NAME>

where <PATH_TO_ANSWERS> is the same file generated by the utils.predict_n_save.