Repository navigation
Accounting for unknown-start requests and retries in a complete evaluation run #3650
Replies: 1 comment 4 replies
|
The manifest and the attempt ledger are easiest to keep outside DeepEval: one row per intended case with a stable case id, plus an append-only attempt log (attempt id, state, timestamps, result or error). Build the Retries inside a provider SDK are the ones that hide unknown-start states. Most SDKs let you turn them off (the OpenAI Python client has If the call happens inside a metric (LLM-as-judge), the same applies to the judge model, so a custom model wrapper that logs attempts is the place to hook it in. I don't know of a built-in DeepEval hook for an unknown-start state, so this is a pattern around the library, not a feature of it. |
Uh oh!
There was an error while loading. Please reload this page.
I am considering an evaluation protocol around model calls that keeps execution completeness separate from semantic scores. After an interruption, a request may be known not to have started, known to have started but have no recoverable result, have an unknown start status, or have a recovered result. Retrying these states indiscriminately can introduce unrecorded repeated trials.
What is the recommended DeepEval pattern for preserving the original case manifest and reporting those states, attempt identifiers and missing results? Is there a supported hook or result representation for an unknown-start attempt, especially when retries happen inside a provider SDK?
I would report completion coverage over all intended cases separately from scores on valid outputs, rather than converting failures into a semantic label or silently restricting the denominator to returned results. For a protocol that only permits recovery of requests known not to have started, are there existing examples or adapter patterns that preserve this distinction?
I found issue #3380 (#3380), which discusses batch cancellation and queued cases. My question is broader accounting/recovery guidance, not a claim that I reproduced that issue or that it remains in every version.
All reactions