Dialogue_level.md

Annotation Details

We invite 20 participants to assess dialogue quality from the sampled dialogues resorting to four overall goals of CRS at the dialogue level. We design a questionnaire containing 5 questions covering the main overall goals of CRS, which refers to the effectiveness, efficiency, dialogue quality, satisfaction and overall. The annotators are asked to answer these questions for all dialogues only based on conversation. Two different annotators annotate each response, and if there is a disagreement between them, the third annotator will involve.

Questionnaire

Dialogue level

1. Throughout the conversation, are you interested in the movies system recommended?
   1. 1 point: I'm not interested in most movies.
   2. 2 point: Half and half.
   3. 3 point: I'm interested in most movies.
2. Throughout the conversation, is the system's recommendation efficient?
   1. 1 point: I need more interactions to determine whether to accept recommendation most of the time.
   2. 2 point: I need few interactions to determine whether to accept recommendation most of the time.
   3. 3 point: I can determine whether to accept recommendation without additional interactions most of the time.
3. Throughout the conversation, what's the dialogue quality of the system?
   1. 1 point: Incoherent or problematic, single style.
   2. 2 point: Coherent and somewhat diverse.
   3. 3 point: Coherent and diverse.
4. Throughout the conversation, how satisfied are you with system?
   1. 1 point: I am not satisfied.
   2. 2 point: I am more satisfied.
   3. 3 point: I am very satisfied.
5. What is your overall rating of the system?
   1. 1 point: Very bad.
   2. 2 point: Bad.
   3. 3 point: Average.
   4. 4 point: Good.
   5. 5 point: Perfect.