About evaluate_response.py

[heyzude]

Hi,

In evaluate_response.py, I see the following snippet

def parse_response_from_file(input_path):
"""Parses the response from a flattened file.
Args:
input_path: Path to read the responses from.
"""
lines = []
with open(input_path, "r") as file_id:
for ii in file_id.readlines():
split_line = ii.split("<SOR>", 1)
lines.append(
(split_line[0].strip("\n"), split_line[1].strip("\n").strip(""))
)
return lines

Here we have <SOR>, but this is only used at noblief mode, while the baseline also uses belief.
Is it allowed to fix evaluation code a little for cases like this? or should I conform to this eval script?

[satwikkottur]

Hello @heyzude ,

Thanks for your interest.

The official evaluation script for response generation is given here model/utils/response_evaluation.py.
The file you mention is derived from the official script, to be used only for the baseline since it produces output in a flattened form. Please use the official script to evaluate the performance numbers.

Feel free to re-open the issue if you have further questions.

[heyzude]

Thanks for solving my question!