Is your feature request related to a problem?
Improving a config's prompt requires a manual evaluation loop, which is time-consuming and inefficient. Currently, there's no automated way to handle this cycle and receive a report upon convergence.
Describe the solution you'd like
- Implement
POST /api/v2/evaluations/iterations to automate the evaluation and prompt improvement process via a LangGraph state machine.
- Define stopping criteria based on mean scores of
Adherence to Ground Truth and Adherence to Prompt, with visibility tracking for Adherence to Knowledge Base.
- Establish conditions for stopping after 3 consecutive rounds under the ceiling-delta threshold, reaching a maximum round cap, or encountering a hard failure.
- Store round-by-round state in a Postgres LangGraph checkpoint, with minimal bookkeeping for the
evaluation_iteration_run table.
- Utilize the existing
/cron/evaluations tick for resumption instead of a new scheduler.
- Deliver a final report and best round results to the caller's
callback_url following the same convention as prompt-improvement callbacks.
Original issue
Problem
Improving a config's prompt today is a manual loop: run an eval, read scores, call improve-prompt, wait, kick off another eval, repeat. There's no way to hand the whole cycle to Kaapi and get back a report once it converges.
Proposal
Add POST /api/v2/evaluations/iterations to run a self-driving loop that chains fast-eval and v2 prompt improvement via a LangGraph state machine:
start_eval → wait_eval → (conditional) start_improve → wait_improve → loop back to start_eval, or → finalize.
- Stop score = mean of
Adherence to Ground Truth + Adherence to Prompt; Adherence to Knowledge Base is tracked per round for visibility only and never gates stopping.
- Stops on 3 consecutive rounds under the ceiling-delta threshold (
ceiling_reached), a max_rounds cap (max_rounds_reached), or a hard round failure (round_failed).
- Round-by-round state lives in a Postgres LangGraph checkpoint (
thread_id = str(iteration_run.id)); the evaluation_iteration_run table is thin bookkeeping only (status/stop_reason/dataset/config/callback).
- Resumption piggybacks on the existing
/cron/evaluations tick rather than a new scheduler — no orchestrator polls a provider directly.
- Final round-by-round report + best round delivered to the caller's
callback_url, same best-effort convention as prompt-improvement callbacks.
Notes
Full design detail: docs/wiki/modules/evaluations.md (Async section) and docs/architecture/kaapi-evaluations-ARCHITECTURE.md.
Is your feature request related to a problem?
Improving a config's prompt requires a manual evaluation loop, which is time-consuming and inefficient. Currently, there's no automated way to handle this cycle and receive a report upon convergence.
Describe the solution you'd like
POST /api/v2/evaluations/iterationsto automate the evaluation and prompt improvement process via a LangGraph state machine.Adherence to Ground TruthandAdherence to Prompt, with visibility tracking forAdherence to Knowledge Base.evaluation_iteration_runtable./cron/evaluationstick for resumption instead of a new scheduler.callback_urlfollowing the same convention as prompt-improvement callbacks.Original issue
Problem
Improving a config's prompt today is a manual loop: run an eval, read scores, call
improve-prompt, wait, kick off another eval, repeat. There's no way to hand the whole cycle to Kaapi and get back a report once it converges.Proposal
Add
POST /api/v2/evaluations/iterationsto run a self-driving loop that chains fast-eval and v2 prompt improvement via a LangGraph state machine:start_eval→wait_eval→ (conditional)start_improve→wait_improve→ loop back tostart_eval, or →finalize.Adherence to Ground Truth+Adherence to Prompt;Adherence to Knowledge Baseis tracked per round for visibility only and never gates stopping.ceiling_reached), amax_roundscap (max_rounds_reached), or a hard round failure (round_failed).thread_id = str(iteration_run.id)); theevaluation_iteration_runtable is thin bookkeeping only (status/stop_reason/dataset/config/callback)./cron/evaluationstick rather than a new scheduler — no orchestrator polls a provider directly.callback_url, same best-effort convention as prompt-improvement callbacks.Notes
Full design detail:
docs/wiki/modules/evaluations.md(Async section) anddocs/architecture/kaapi-evaluations-ARCHITECTURE.md.