[AI Evaluation] How should the AI.Eval library be used for runtime AI evaluation?

[BrennanConroy]

All of the samples I've seen are just one-off test methods. Can we have an explicit sample showing how a real app might use the library?

For example, if I have a chat agent that a customer is talking to, how should I evaluate the conversation? Run evaluate after every AI response with the same iteration id so the full conversation will eventually be evaluated and previous sub-conversation evaluations will be overwritten?

[shyamnamboodiripad]

Thanks for the feedback @BrennanConroy. Yes, this has come up before and creating samples to demonstrate 'online' evaluation (alongside the current 'offline' samples) is a great idea. For example, we have discussed creating a sample that shows how to log online evaluation scores produced (during app run time) to telemetry and how to monitor results over time (using tools like App Insights). I have added this issue to our backlog tracking - hopefully we can create such a sample soon.

While offline evaluation has been one of the primary use cases for the libraries, the libraries are designed to be modular so that you can pick and choose the components that you need for orchestrating your evals in different settings.

For example, when doing online (runtime) evals, the response caching functionality available as part of the Microsoft.Extensions.AI.Evaluation.Reporting package is probably not going to be useful (since it is unlikely that the same user request and response will be seen multiple times). Similarly, if you decide to log scores to telemetry and use dashboarding functionality available as part of tools like App Insights to monitor the scores over time, you may not need the result storage and report generation functionality available as part of Microsoft.Extensions.AI.Evaluation.Reporting either. In this case, it should be possible to orchestrate online evaluation using just the core evaluation abstractions in the Microsoft.Extensions.AI.Evaluation package and the evaluators themselves (e.g., Microsoft.Extensions.AI.Evaluation.Quality and Microsoft.Extensions.AI.Evaluation.Safety packages).

[shyamnamboodiripad]

For example, if I have a chat agent that a customer is talking to, how should I evaluate the conversation? Run evaluate after every AI response with the same iteration id so the full conversation will eventually be evaluated and previous sub-conversation evaluations will be overwritten?

Most of the evaluations that ship as part of the library (such as the ones in the Quality package) only evaluate the final conversation turn. Some evaluators (such as RelevanceTruthAndCompletenessEvaluator) do consider the entire conversation history, however, the score produced is still generally for the final conversation turn only.

(Note that this is not a limitation of the eval API but rather just a reflection of how the existing evaluators that ship as part of the library have been implemented. It would still be possible to use the API to author your own custom evaluators that evaluate every conversation turn in a multi-turn conversation and produce a single score for the entire conversation.)

With this in mind, there are a few options to consider:

• One option would be to score individual conversation turns and log these scores to telemetry. To control cost, you may want to implement some sampling logic where you periodically / randomly evaluate a subset of responses (for a given user / for a particular conversation) that your chat agent is producing over time. You could then use standard mechanisms available as part of tools like App Insights for score aggregation (on a per-user and / or per-conversation basis), for monitoring on dashboards etc.

• If you don't want to log individual scores to telemetry, another option may be to implement your own custom score aggregator that aggregates the scores (for a particular conversation / user) in memory and periodically logs the aggregated scores to your telemetry. Again, you may still want to implement some sampling logic to control cost.

• One idea to implement this sampling and aggregation could be to author a wrapper IEvaluator that you can wrap around any IEvaluator (or around the CompositeEvaluator) that you want to use. This wrapper evaluator could implement the sampling logic (i.e., score some responses while ignoring others), as well as the aggregation logic (i.e., store the aggregated value for scores in memory and log the aggregated scores to telemetry periodically).

As discussed above, we can hopefully explore some of these ideas more elaborately in the sample that we create to demonstrate online (runtime) evaluation.