Objective:
The overall objective is to help non profit users to get more actionable, data driven insights from Eval runs that can help them to further optimise the prompt. Few initiatives listed below
Initiative 1: Enable users to analyze eval performance across thematic categories (e.g PII handling, Hinglish handling, corner case, hallucination checks, Facts handling etc) to identify weak areas and improve prompts/models faster accordingly. This was specific request from Sneha, VG etc.
Github ticket
Initiative 2 (long term): Use AI model to recommend prompt changes. One of the input for this can be category wise scoring as part of initiative 1
PRD Document: https://docs.google.com/document/d/1Z5mRQIS4rli-vWGT31-O93uFn1o_CFrWOA4oMO6uw9M/edit?tab=t.h9hid11svst#heading=h.joj9q8ysj80v
Objective:
The overall objective is to help non profit users to get more actionable, data driven insights from Eval runs that can help them to further optimise the prompt. Few initiatives listed below
Initiative 1: Enable users to analyze eval performance across thematic categories (e.g PII handling, Hinglish handling, corner case, hallucination checks, Facts handling etc) to identify weak areas and improve prompts/models faster accordingly. This was specific request from Sneha, VG etc.
Github ticket
Initiative 2 (long term): Use AI model to recommend prompt changes. One of the input for this can be category wise scoring as part of initiative 1
PRD Document: https://docs.google.com/document/d/1Z5mRQIS4rli-vWGT31-O93uFn1o_CFrWOA4oMO6uw9M/edit?tab=t.h9hid11svst#heading=h.joj9q8ysj80v