Skip to content

Evaluation: Automate prompt improvement #1159

Description

@AkhileshNegi

Is your feature request related to a problem?
Improving a config's prompt requires a manual evaluation loop, which is time-consuming and inefficient. Currently, there's no automated way to handle this cycle and receive a report upon convergence.

Describe the solution you'd like

  • Implement POST /api/v2/evaluations/iterations to automate the evaluation and prompt improvement process via a LangGraph state machine.
  • Define stopping criteria based on mean scores of Adherence to Ground Truth and Adherence to Prompt, with visibility tracking for Adherence to Knowledge Base.
  • Establish conditions for stopping after 3 consecutive rounds under the ceiling-delta threshold, reaching a maximum round cap, or encountering a hard failure.
  • Store round-by-round state in a Postgres LangGraph checkpoint, with minimal bookkeeping for the evaluation_iteration_run table.
  • Utilize the existing /cron/evaluations tick for resumption instead of a new scheduler.
  • Deliver a final report and best round results to the caller's callback_url following the same convention as prompt-improvement callbacks.
Original issue

Problem

Improving a config's prompt today is a manual loop: run an eval, read scores, call improve-prompt, wait, kick off another eval, repeat. There's no way to hand the whole cycle to Kaapi and get back a report once it converges.

Proposal

Add POST /api/v2/evaluations/iterations to run a self-driving loop that chains fast-eval and v2 prompt improvement via a LangGraph state machine:

start_evalwait_eval → (conditional) start_improvewait_improve → loop back to start_eval, or → finalize.

  • Stop score = mean of Adherence to Ground Truth + Adherence to Prompt; Adherence to Knowledge Base is tracked per round for visibility only and never gates stopping.
  • Stops on 3 consecutive rounds under the ceiling-delta threshold (ceiling_reached), a max_rounds cap (max_rounds_reached), or a hard round failure (round_failed).
  • Round-by-round state lives in a Postgres LangGraph checkpoint (thread_id = str(iteration_run.id)); the evaluation_iteration_run table is thin bookkeeping only (status/stop_reason/dataset/config/callback).
  • Resumption piggybacks on the existing /cron/evaluations tick rather than a new scheduler — no orchestrator polls a provider directly.
  • Final round-by-round report + best round delivered to the caller's callback_url, same best-effort convention as prompt-improvement callbacks.

Notes

Full design detail: docs/wiki/modules/evaluations.md (Async section) and docs/architecture/kaapi-evaluations-ARCHITECTURE.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions