v0.3.1.5: Multi-Eval Node

[ianarawjo]

This is the first release adding the MultiEval node to ChainForge proper, alongside:

• improvements to response inspector table view to display multi-criteria scoring in column view
• table view is now default when multiple evaluators are detected

Voilà:

As you can see, Multi-Eval allows you to define multiple per-response evaluators inside the same node. You can use this to evaluate responses across multiple criteria. Evaluators can be a mix of code or LLM evaluators, as you see fit, and you can change the LLM scorer model on a per-evaluator basis.

This is a "beta" version of the MultiEval node, for two reasons:

• The output handle of MultiEval is disabled, since it doesn't yet work with VisNodes to plot data across multiple criteria. That is a separate issue that I didn't want holding up this push. It is coming.
• There are no genAI features in MultiEval, yet, like there are in Code Evaluator nodes. I want to do this right (beyond EvalGen, which is another matter). The idea is that you can describe the criteria in a prompt and the AI will add an evaluator to the list that it thinks is the best, on a per-criteria basis. For now as a workaround, you can use the genAI feature to generate code inside single Code Evaluators and port that code over.

The EvalGen wizard is also coming, to help users automatically generate evaluation metrics with human supervision. We have a version of this on the multi-eval branch (which due to the TypeScript front-end rewrite, we cannot directly merge into main), but it doesn't integrate Shreya's fixes.

---

This discussion was created from the release v0.3.1.5: Multi-Eval Node.

[dmaniloff]

Hi @ianarawjo and team! Thank you for ChainForge 🙏 . I am working through the EvalGen paper and would like to try it out.

I see that the work in this discussion was a prior requirement to getting EvalGen into ChainForge, and also noticed that
#299 was recently merged.

Is there a guide somewhere on how to setup an EvalGen flow? I went to https://chainforge.ai/play/ and added a Multi-Evaluator node but not really sure how to proceed from there.

Any pointers are greatly appreciated! Thank you.

[ianarawjo]

Hi there! Thanks for the question. Actually the right branch to try out EvalGen v1 is “multi-eval”. That is the version from the paper and I’ve checked in the past that it functions. You add a Multi-Eval node, attach it to Prompt Node output and then click the EvalGen button to access the popup. Hope this helps!

We have been working on a second version of EvalGen for ChainForge main, but it’s been slow, unfortunately, as all of us have many other obligations now.