Skip to content

Referenceless quality estimation for natural language generation – predicting human ratings


Notifications You must be signed in to change notification settings


Repository files navigation

RatPred 🐀

Rat(ing) Pred(ictor) – A Referenceless NLG Quality Estimation Tool

RatPred predicts NLG quality ratings/ranks using a recurrent neural network. It is trained using a set of human-rated/ranked NLG outputs along with the corresponing source meaning representations (MRs). It can then estimate human ratings/ranks given a MR and an NLG system output only. Unlike most automated metrics used for NLG, such as BLEU or NIST, RatPred does not need human-authored reference texts.

For details on the system architecture and its performance, please refer to our ICML-LGNL paper.

Note that RatPred is highly experimental and only tested, so bugs are inevitable. If you find a bug, feel free to contact me or open an issue.

Installation and Usage

See Note that currently RatPred is using Python 2. Python 3 version is in the works.


If you use or refer to RatPred, please cite or INLG 2019 paper:

Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser (2019): Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking). In INLG 2019, Tokyo, Japan.

You can also refer to our ICML-LGNL paper:

Ondřej Dušek, Jekaterina Novikova, and Verena Rieser (2017): Referenceless Quality Estimation for Natural Language Generation. In Proceedings of the 1st Workshop on Learning to Generate Natural Language, Sydney, Australia.


Author: Ondřej Dušek

Copyright © 2017-2018 Interaction Lab, Heriot-Watt University, Edinburgh.
Copyright © 2019 Institute of Formal and Applied Linguistics, Charles University, Prague.

Licensed under the Apache License, Version 2.0 (see LICENSE.txt).


This research received funding from the EPSRC projects DILiGENt (EP/M005429/1) and MaDrIgAL (EP/N017536/1) and Charles University project PRIMUS/19/SCI/10. The Titan Xp used for this research was donated by the NVIDIA Corporation.


Referenceless quality estimation for natural language generation – predicting human ratings







No packages published