Skip to content
/ usr Public

Code for ACL 2020 paper: USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (https://arxiv.org/pdf/2005.00456)

License

Notifications You must be signed in to change notification settings

Shikib/usr

Repository files navigation

Setup

  • Install dependencies.
  • You might have to run python3 setup.py develop
  • Download all model files from Google Drive
  • Unzip model folders into the examples/ directory. You should end up with the following model folders: roberta_ft/, uk/ and ctx/
  • Setup your custom data in the same way that shown in undr/, fct/ and both/. Note that in the latter two folders, the first line is skipped by the file (it can be arbitrary, so I've set it to just be a copy of the second line).

Running

From the examples/ directory:

Full USR metric (might require some editing of the file): dstc9_eval.py

Specific sub-metrics (harder):

MLM: sh mlm_scores.sh

DR-c: sh dr_c.sh

DR-f: sh dr_f.sh

Regression: python3 regression.py

Dataset

The USR data can be found here

About

Code for ACL 2020 paper: USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (https://arxiv.org/pdf/2005.00456)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published