Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement.
This is the code related with our study on : Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement.
Conversational agents participating in multi-party interactions face significant challenges in dialogue state tracking, since the identity of the speaker adds significant contextual meaning. It is common to utilise diarisation models to identify the speaker. However, it is not clear if these are accurate enough to correctly identify specific conversational events such as agreement or disagreement during a real-time interaction. This study uses a cooperative quiz, where the conversational agent acts as quiz-show host, to determine whether diarisation or a frequency-and-proximity-based method is more accurate at determining agreement, and whether this translates to feelings of engagement from the players.Experimental results show that our proposed method was preferred by players, and was more accurate at detecting agreement, reaching an accuracy of 0.44, compared to 0.28 for the diarised system.
You can consult all the data collected during our evaluation in the section Evaluation Folder
Install all the libraries:
pip install -r requirement.txtAdd your API credentials in the file google_api_credentials for STT and TTS of google
On windows:
.\start_servers_diarisation.batOn linux:
./start_servers_diarisation.bashOn windows:
.\start_servers.batOn linux:
./start_servers.bashThe branch "nlu_test" is dedicated to test new version of nlu
- Alexandre Kha: apk2002@hw.ac.uk
- Andy Edmondson: ae2016@hw.ac.uk
- Daniel Denley: dad2001@hw.ac.uk
- James Ndubuisi: jn2033@hw.ac.uk
- Lia Perochaud: lfrp2000@hw.ac.uk
- Miebaka Worika: mw2037@hw.ac.uk
- Neil O’Reilly: no2003@hw.ac.uk
- Raphaël Valeri: rv2018@hw.ac.uk
School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh