Initial skeleton for Evaluator classes and exceptions

[nathan-weinberg]

This PR introduces the Evaluator parent class and the MMLUEvaluator and MTBenchEvaluator child classes

It also introduces the EvalError parent exception class for all eval custom exceptions and the ModelNotFoundError child exception class

[alinaryan]

This LGTM - are PR-MMLU and PR-Bench going to live in the MMLU and MT-Bench child classes respectively or is the idea to add them in later as their own child classes?

[nathan-weinberg]

This LGTM - are PR-MMLU and PR-Bench going to live in the MMLU and MT-Bench child classes respectively or is the idea to add them in later as their own child classes?

Good question! I think we could go with either approach - either extending the current child classes or adding new child classes but in the same files.

Could you do an explicit approval? I've set the repo so two are required for merge. @alimaredia same for you.

@danmcp and @xukai92 feel free to weigh-in as well - this foundation will define subsequent work.

[alinaryan]

I lean towards new child classes since the inputs/dependencies differ across PR benchmarks. You could add them in this PR or it can be handled in a follow-up PR by anyone.
Thanks for working on this!!

[danmcp]

This LGTM - are PR-MMLU and PR-Bench going to live in the MMLU and MT-Bench child classes respectively or is the idea to add them in later as their own child classes?
@danmcp and @xukai92 feel free to weigh-in as well - this foundation will define subsequent work.

It would be good to understand the signatures before deciding. But I'm pretty sure they should be new classes. The other thing to consider whether the modeling makes more sense as a is-a or has-a relationship. Has-a might give some useful flexibility.

[nathan-weinberg]

This LGTM - are PR-MMLU and PR-Bench going to live in the MMLU and MT-Bench child classes respectively or is the idea to add them in later as their own child classes?
@danmcp and @xukai92 feel free to weigh-in as well - this foundation will define subsequent work.

It would be good to understand the signatures before deciding. But I'm pretty sure they should be new classes. The other thing to consider whether the modeling makes more sense as a is-a or has-a relationship. Has-a might give some useful flexibility.

I pushed up two additional child classes based on the specs in the doc - lmk your thoughts!