Model evaluation

[sanagno]

We need a better evaluation pipeline to better quantify model performance and compare models with each other.
Some ideas include

• Evaluating on dataset for which we already have the chatGPT reference, e.g. HC3.
• Using a fine-tuned reward model.

[sanagno]

If we want to test knowledge/retrieval etc, there are some good evaluations on things like 'wizard of wikipedia' or 'wizard of the internet.
If we want to test conversational skills there is 'blender skill talk' or 'convai2'.
There are also things like 'Empathetic dialogue' to test empathy for instance

[pruksmhc]

Adding instructions evaluation dataset to this thread:
https://arxiv.org/pdf/2204.07705.pdf

[pruksmhc]

I'm interested to help with this, but have my hands full for the next week or so.

[totuta]

Is this task being worked on right now?

[sanagno]

So as far as I know :)

[adendek]

Do you still need a contributor?

[sanagno]

yes, please, If anyone is interested I can assign you to it

[adendek]

I'll start working on this tomorrow, but I don't want to stop or delay anyone who wants to contribute as well.

[shahules786]

For using RM to evaluate LLM results I have pushed a framework to do this in a sister repo https://github.com/Open-Assistant/oasst-model-eval/tree/main/model_eval/scoring. I can replicate the same here if that's something we consider useful. @sanagno