Conversation Evaluations

[AndresPrez]

Is your feature request related to a problem? Please describe.
I have an application with user/bot conversations. I would like to somehow apply some of the Q&A evaluations to entire conversations.

Describe the solution you'd like
I'd like to be able to measure conversations with LLM evaluations.

Describe alternatives you've considered
One idea would be to add a top layer to all the existing metrics which job would be to leverage an LLM to somehow condense, "distil" or extract a few main topics or questions from the messages in the conversation. And also the answers to those questions given by the bot. After that, we can use the traditional Q&A metrics for each of them.