- [Aug. 28th] Fixed an issue on incomplete context and achieved better average scores! Details in the following table:
Metrics | Interesting | Fluent | Engaging | Specific | Relevant | Correct | Appro. | Und. | Avg. |
---|---|---|---|---|---|---|---|---|---|
FED + C-PMI-SYM | 48.4 | 16.6 | 36.9 | 28.0 | 10.5 | 14.8 | 17.9 | 10.7 | 23.0 |
FED + C-PMI | 48.2 | 17.6 | 37.0 | 28.7 | 12.8 | 17.6 | 18.1 | 11.1 | 23.9 |
We propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative loglikelihood-based scorer with our proposed CPMI scorer, we achieve a relative 60.5% higher Spearman correlation on average for the FED evaluation metric.
The implementation of C-PMI is quite simple and only needs a few lines of code. Running the jupyter notebook, c-pmi.ipynb, will reproduce the experiment results in our paper. Our C-PMI and C-PMI-SYM scorer are defined as the function MI_score_turn_pmi
and the function MI_score_turn_sympmi
respectively in the notebook.
If you find our work useful, please consider citing:
@article{ren2023cpmi,
title = {C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation},
author = {Liliang Ren and Mankeerat Sidhu and Qi Zeng and Revanth Gangi Reddy and Heng Ji and ChengXiang Zhai},
year = {2023},
journal = {arXiv preprint arXiv: 2306.15245}
}
Liliang Ren (liliang3@illinois.edu)