This is the accompanying repository of the following publication:
Madureira & Schlangen (2022). Can Visual Dialogue Models do Scorekeeping? Exploring how Dialogue Representations Incrementally Encode Shared Knowledge. Short paper presented at ACL 2022 in Dublin, Ireland.
The axis labels are swapped in Table 3 (and the corresponding figures in the Jupyter notebook).
The paper proposes an evaluation method to assess how visual dialogue models keep track of information about an image that is private/shared at a given turn. It implements a probing classifier based on a neural network whose input is a proposition embedding and a dialogue state representations and the output is a probability over classes (private or shared, believed to be true or false by the answerer).
The set up involves three main steps, each depending on one of the three subdirectories here:
generating_propositions/: turn VisDial QA pairs into propositions and get their embeddings.retrieving_dialogue_representations/: extract the dialogue state representations of the original visual dialogue encoders.main_task/: run experiments with the probing classifier.
Due to the different dependencies we used different Python environments for each part. You can re-create the environments with conda using the ```yml```` files:
python_envs/environmentcoref.yml: to replace pronouns on VisDial.python_envs/environment.yml: to generate propositions.python_envs/main_task_environment.ym: to get proposition embeddings and run the main experiments.
To retrieve dialogue representations, follow the instructions on the original repository to build the environment.
- Follow the instructions on
retrieving_dialogue_representations. - Follow the instruction on
generating_propositions. - Follow the instructions on
main_task.
This version of the code was used for the final version submitted on March 30, after fixing a problem with the extraction of the dialogue representations.
If you use this work, please cite:
@inproceedings{madureira-schlangen-2022-visual,
title = "Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge",
author = "Madureira, Brielen and
Schlangen, David",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-short.73",
doi = "10.18653/v1/2022.acl-short.73",
pages = "651--664",
}
This work is licensed mainly under two licences:
- Code deriving from Murahari et al. (2019) is licensed under BSD.
- Other source code is licensed under MIT.
We use many Python libraries. See credits on each repository for details.