feat(eval-harness): agentic retrieval via configurable Zep tools - #575
Conversation
Adds a tool-calling mode to the evaluation harness: the response model is given retrieval tools and decides what to fetch from Zep before answering, alongside (or instead of) the deterministic context block. - Tools are defined as ToolSpec entries in config/evaluation_config/tools.py (name, description, JSON Schema, async executor, enabled, requires_doc_graph) - --tools/--no-tools is the only switch that decides whether they are used; --context-block/--no-context-block toggles deterministic retrieval, and the two combine - Budgets: --max-tool-calls and --max-tool-iterations are enforced independently (parallel calls in one turn cost one iteration), with a forced answer turn once either is spent; --require-tool-call stops the model answering before it retrieves - Completeness is graded on the union of every successful tool output, so it still measures retrieval while accuracy measures generation - Latency reports overall user-facing latency plus tool wall-clock vs non-tool LLM time, with per-tool counts, failures, and cap/downgrade counters Search knobs are now applied only where Zep honours them: max_characters for scope="auto" (which ignores limit and reranker), limit and reranker for the other scopes. Judge rubrics move to config/evaluation_config/judge_prompts.py so every run snapshots the rubric that graded it; the completeness rubric gained tool-awareness, so baselines should be re-run rather than compared against older runs. Co-Authored-By: Claude <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Findings
-
Critical —
zep-eval-harness/retry.py,is_retryable()hunk: treating every404as permanently non-retryable changes retry behavior repository-wide. Ingestion creates threads and immediately adds messages; an eventually consistent404will now abort that conversation instead of retrying, causing incomplete evaluation graphs. Restrict the new policy to the tool-agent request that needs it, or keep transient Zep404s retryable. -
Warning —
zep-eval-harness/tool_agent.py, new agent loop: no tests cover budget enforcement, parallel/refused calls, malformed arguments, providertool_choicefallback, or final-answer retries. These paths directly determine retrieval results and metrics. Add focused async unit tests with mocked completion responses and tool executors.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
… answer The forced answer turn only refused ignored tool calls when the response had no text. A provider that rejected tool_choice="none" and was retried with "auto" can return both a "let me search..." preamble and tool calls, so that preamble was graded as the final answer and the requested calls never reached tool_trace. Tool calls now take precedence over text on the final turn, exactly as they do in the loop above: the calls are recorded as refused and the model is asked once more. The first turn's text is kept only as a fallback when the re-ask returns nothing, so a model that genuinely answers while also calling a tool doesn't lose its answer. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning — zep-eval-harness/tool_agent.py (new agent-loop implementation): No automated tests cover the new retrieval behavior. Budget boundaries, parallel calls, malformed/unknown calls, provider tool_choice downgrades, and forced-answer retries are complex and directly affect evaluation validity. Add unit tests with mocked completion/tool responses covering these branches before relying on reported completeness and tool metrics.
No other actionable correctness, security, or convention issues found.
…erankers
Two issues from review.
The forced-answer retry accepted its text without checking for tool calls, so a
provider that stalled twice ("still searching" plus another call) had that text
graded as the answer while the call went missing from tool_trace. The forced
answer is now a bounded loop where every attempt is validated the same way:
tool calls take precedence over text, each ignored call is recorded as refused
with its own round number, and if no attempt answers cleanly the run is flagged
with forced_answer_failed and counted as tests_forced_answer_failed. Text is
still kept as a last resort so a model that answers and calls a tool in the
same turn doesn't lose a real answer — the flag, not a blank answer, is what
records the failure.
Making the reranker configurable also advertised mmr and node_distance, which
cannot work as harness-wide defaults: Zep needs mmr_lambda for MMR and a
center_node_uuid to rerank around for node distance, and neither retrieval path
sends them. SUPPORTED_RERANKERS now lists only the argument-free rerankers
(rrf, cross_encoder, episode_mentions), a misconfiguration is rejected at
startup instead of failing on every search, and the README explains how to
evaluate the excluded two.
Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning
zep-eval-harness/tool_agent.py(new agent loop): No automated tests cover tool-call parsing, parallel calls, invalid arguments, providertool_choicefallback, call/iteration caps, or forced-answer behavior. These branches directly affect answers and evaluation metrics. Add focused async unit tests with mocked completion responses and tool executors, including exact-budget and over-budget cases.
The retry guard swept the whole 4xx range, which changed behavior for every script that uses retry_with_backoff, not just the tool path it was added for. A 404 from a resource that isn't visible yet — a thread or graph just created by the ingestion scripts — used to retry through and now would fail on the first attempt. Only 400 and 422 are terminal now: those are the malformed-request statuses the guard exists to stop retrying (model-supplied search arguments), and no working pipeline can depend on retrying them. Everything else, 404 included, retries exactly as it did before. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning — zep-eval-harness/tool_agent.py, new-file hunk @@ -0,0 +1,611: The complex agent loop has no automated tests. Budget enforcement, parallel calls, invalid arguments, provider tool_choice fallback, forced-answer retries, message/tool-call ID pairing, and metrics can regress silently. Add mocked async tests covering these paths, particularly call/iteration caps and valid follow-up message construction.
If one forced-answer attempt returned text plus tool calls and the next
returned neither, `text or preamble` promoted the earlier preamble ("Let me
search...") to the answer via the clean-turn break, so neither
forced_answer_failed nor answer_empty was set and the run looked like it had
answered.
The invariant is now explicit: the published answer either came from a turn that
answered cleanly with non-empty text, or forced_answer_failed is set. Falling
back to a preamble sets the flag wherever it happens, not only when every
attempt requested tools. The text is still published so a model that answers and
calls a tool in the same turn keeps its answer, and the counter is now documented
alongside the other tool-run counters.
Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning — zep-eval-harness/tool_agent.py:393: The new multi-turn agent and budget-enforcement state machine has no automated tests. Regressions in parallel-call ordering, refused-call pairing, cap accounting, malformed arguments, tool_choice fallback, or forced-answer retries could silently invalidate evaluation results. Add mocked async tests covering normal completion and each budget/provider edge case.
Warning — zep-eval-harness/retry.py:8: HTTP 400/422 errors are now immediately non-retryable for every harness workflow, not only the new tool path, but this behavioral change is untested. Add tests proving transient statuses still retry, 400/422 fail immediately, and _chat performs its intended one-time tool_choice="auto" fallback.
…aths The check ran over both rerankers unconditionally, so a tools-only run could be rejected because CONTEXT_BLOCK_RERANKER was unusable even though no context block search would happen, and a context-block run could be rejected over TOOL_SEARCH_RERANKER. That coupled two modes whose whole point is to toggle independently. Validation now covers only the paths that will run, via unsupported_rerankers(), which also makes the rule testable without the CLI. Confirmed both directions end to end: with CONTEXT_BLOCK_RERANKER unusable, --tools --no-context-block runs through to its first Zep call while the default context-block run is still rejected with the same clear message. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning — zep-eval-harness/tool_agent.py, new agent loop
The substantial tool-calling state machine has no automated tests. Budget enforcement, parallel calls, malformed arguments, missing call IDs, provider tool_choice downgrades, and forced-answer retries can silently corrupt evaluation results if they regress. Add unit tests with mocked completion responses and tool executors, especially for exact/overflowing call caps and repeated tool calls during forced-answer turns.
Warning — zep-eval-harness/config/evaluation_config/tools.py, _render_search_results()
Tool results tell the response model that facts with past end dates are “NO LONGER VALID,” while the completeness rubric explicitly says historical facts remain valid context. For historical questions, the agent may incorrectly discard the retrieved answer even though completeness marks retrieval successful. Reword the note to distinguish “not currently true” from “invalid for historical questions,” consistent with the judge prompt.
- Warming a document graph no longer sends CONTEXT_BLOCK_RERANKER, so a tools-only run can't fail on a constant its paths never use. Ranking is irrelevant to a limit=1 cache-priming search. - Context block searches truncate the query like the tool searches already did. The limit is Zep's, not a tool policy, so TOOL_SEARCH_MAX_QUERY_CHARS is now SEARCH_MAX_QUERY_CHARS and both paths honour it; a long test question used to fail the whole search. - scope="auto" tool output now carries the same temporal-validity note as every other scope and the context block. It was the default scope and the only path that never told the model a past end date means the fact no longer holds. - require_tool_call is enforced, not just requested: if the provider ignores tool_choice="required" and answers on the first turn, the model is told to retrieve first and given one more turn (once, so a model that refuses can still finish). --no-require-tool-call still allows an immediate answer. - Text from a turn that also requested tools now seeds the forced-answer fallback instead of being dropped, so a real answer isn't lost when the forced turns come back empty. It is still published only under forced_answer_failed. - retrieval_configuration records tools_requested alongside use_tools, so --tools with every spec inactive is legible as "asked for, none active" rather than looking like a non-tool run. - tests_with_no_calls is zero when tools were never offered, instead of counting every test in a context-block-only run. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Findings
-
Warning — zep-eval-harness/tool_agent.py,
run_tool_agentloop hunk: The retry used to enforcerequire_tool_callconsumes one ofmax_iterations, even though that turn made no tool call. With--max-tool-iterations 1, a provider that ignores/downgradestool_choice="required"is prompted to retrieve again but immediately exits the loop and is forced to answer without retrieval. It also reports the next successful tool-call turn as iteration 2, contrary to the documented definition that an iteration is a turn requesting tools. Track tool-call rounds separately from LLM attempts, and allow the one enforcement retry without spending the retrieval-iteration budget. -
Warning — zep-eval-harness/tool_agent.py,
_chathunk: Every HTTP 400 from a constrained turn is treated as unsupportedtool_choiceand retried with"auto". A malformed message, excessive context, invalid tool schema, or other request error will therefore be misreported as a tool-choice downgrade and may relax required/forbidden tool use if the provider happens to accept the altered request. Only downgrade when the error specifically identifiestool_choiceas unsupported; otherwise re-raise the original error. -
Suggestion — The patch adds a complex provider-dependent agent loop without automated tests. Add focused tests covering parallel calls, invalid/refused calls, both budgets, providers ignoring or rejecting each
tool_choicevalue, missing response content, forced-answer retries, and the--max-tool-iterations 1case.
The first-turn nudge added in f156339 called _assistant_message() on the message from _turn(), which is None when the provider returns no choices at all. That raised AttributeError out of run_tool_agent and through the un-guarded gather in evaluate_all_questions, aborting every remaining test case for the user. The nudge now requires a message: with no choices there is no assistant turn to replay and nothing to argue with, so it falls through to the forced-answer turn. Also attributes tool_choice downgrades instead of only counting them. A 400 that isn't about tool_choice fails the "auto" retry too and propagates, so it never reached the counter — but the counter still mixed first-turn rejections with forced-answer-turn ones. require_tool_call_unenforced now records specifically that require_tool_call was asked for and not applied, whether the provider rejected it or accepted it and answered without retrieving. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Critical
- tool_agent.py,
run_tool_agent()hunk: Thefor iteration in range(...)loop consumes an iteration even when the provider ignorestool_choice="required"and returns no tool call. With--max-tool-iterations 1, the corrective “retrieve first” message is appended, but the loop immediately ends and forces an answer without ever permitting retrieval. Larger budgets similarly lose one advertised retrieval round and misreport iteration-cap metrics. Retry the corrective turn without advancing the tool-iteration counter, and count only turns that actually request tools.
Warning
- No tests accompany the new 668-line agent loop and its budget/provider-fallback behavior. Add focused async tests covering ignored/rejected
tool_choice, parallel calls exceeding the call cap, malformed calls, forced-answer retries, and the one-iteration case above.
The first-turn nudge used `continue`, which advanced the iteration counter, so a provider that ignores tool_choice="required" cost the agent one of its retrieval rounds. With --max-tool-iterations 1 that was the whole budget: the nudge fired, the loop ended, and the answer came from the forced turn with no retrieval at all — the exact outcome the nudge exists to prevent. The loop now counts rounds explicitly and the nudge gives its round back, since it compensates for provider misbehavior rather than being retrieval the run asked for. The nudge turn also drops tool_choice="required", which the provider has already shown it won't honour. hit_iteration_cap is now measured against rounds actually offered to the model rather than result.iterations, which counts only rounds that requested tools — a round the model spent answering still consumed the budget. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning
zep-eval-harness/tool_agent.py:443— The new 676-line tool-calling state machine has no automated tests. Budget exhaustion, parallel calls, malformed arguments,tool_choicedowngrades, and forced-answer retries materially affect correctness. Add unit tests using fake completion/tool responses, especially verifying message pairing and call/iteration caps.
No additional actionable correctness or security defects found.
… one The forced-answer loop treated a response with no choices as a clean turn: text was "", tool_calls was empty, so it took the accept-and-break path, published an empty answer, and skipped its remaining attempt. The retrieval loop already treats a missing message as nothing to accept; the forced loop now applies the same rule and spends another attempt, which is worth doing because an empty candidate is often transient. If no attempt ever answers, forced_answer_failed is set as before (answer_empty covers the case where there was no text at all), and the log line no longer claims the model "kept requesting tools" when the provider simply returned nothing. The instruction is also not re-appended when the previous attempt produced no message, so the history doesn't repeat it back to back. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Warning — zep-eval-harness/tool_agent.py (new agent loop): No automated tests cover the new 686-line tool-calling state machine. Budget enforcement, parallel calls, malformed arguments, provider tool_choice downgrades, and forced-answer retries can silently alter evaluation results. Add unit tests with mocked OpenAI responses and tool executors covering these branches before relying on the reported metrics.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e865e02. Configure here.
… a block The first-turn nudge asserted "you have not retrieved anything yet, and you have no knowledge of this user" — false in "both" mode (--tools with the context block on), where the system prompt already carries a full context block and the completeness judge grades that block. It also spent a turn pushing the model toward a tool call it did not need, on the default tool-mode configuration. run_tool_agent now takes has_context_block. With a block injected, answering the first turn without a tool call is legitimate and the answer stands; without one, nothing else grounds the answer and the nudge fires as before, so the instruction is only ever sent when its claim is true. require_tool_call_unenforced is still recorded in both modes — it is a fact about the provider, not a claim about groundedness — and the docs now say so, since in "both" mode the context block did the grounding. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
-
Warning —
zep-eval-harness/tool_agent.py(new agent loop, especiallyrun_tool_agent): The 699-line provider-facing state machine has no automated tests. Budget exhaustion, malformed/unknown calls, parallel calls, ignoredtool_choice, missing choices, and forced-answer retries can silently alter answers and evaluation metrics. Add mocked async tests covering these branches and verifying message history, counters, and retrieved context. -
Suggestion —
zep-eval-harness/README.md, results JSON example around therequire_tool_call_unenforcedfields: The example is invalid JSON: it contains a trailing comma and an apparently duplicated/mis-indentedtests_require_tool_call_unenforcedentry. Correct the example so users can parse or copy it reliably.
1 participant
Adds a tool-calling mode to the eval harness: the response model is given retrieval tools (defined as
ToolSpecentries inconfig/evaluation_config/tools.py— name, description, JSON Schema, async executor,enabled,requires_doc_graph) and decides what to fetch from Zep before answering, either instead of or alongside the deterministic context block.--tools/--no-toolsis the only switch that decides whether tools are used,--context-block/--no-context-blocktoggles deterministic retrieval, and--max-tool-calls/--max-tool-iterationsare enforced independently (parallel calls in one turn cost one iteration) with a forced answer turn once either is spent, while--require-tool-callstops the model answering before it retrieves. Context completeness is graded on the union of every successful tool output so it still measures retrieval rather than generation, and latency now reports overall user-facing latency split into tool wall-clock vs non-tool LLM time, plus per-tool counts, failures, and cap/downgrade counters.Search knobs are now sent only where Zep honours them —
max_charactersforscope="auto"(which ignoreslimitandreranker) andlimit/rerankerfor the other scopes — since the previous behavior would have silently capped tool retrieval at 2500 characters under a different reranker than the context block, invalidating the cross-mode comparison. Judge rubrics move intoconfig/evaluation_config/judge_prompts.pyso each run snapshots the rubric that graded it; the completeness rubric gained tool-awareness, so re-run baselines rather than comparing completeness against older runs.Verified offline against fake Zep/LLM clients using real SDK response types (99 checks: budget enforcement, graded-context assembly, latency attribution, provider-quirk handling, statistics and summary rendering); nothing has been exercised against live Gemini or Zep.
🤖 Generated with Claude Code
Note
Medium Risk
Large change to the core evaluation path (retrieval, judging, metrics, and CLI), with intentional breaking comparability for completeness scores and
response_*latency versus pre-change runs; default behavior remains context-block-only (USE_TOOLS = False).Overview
Adds agentic retrieval to the eval harness: the response model can call Zep retrieval tools (
search_memory,search_documents,get_user_profile) instead of—or on top of—the deterministic context block.zep_evaluate.pygains--tools,--context-block, budgets, and--require-tool-call;tool_agent.pyruns the multi-turn loop with caps, forced final answer, andtool_tracerecording.Completeness grading now uses the union of successful tool outputs (plus any injected context block), with tool arguments and harness error text excluded so retrieval stays measurable separately from generation. Judge rubrics move to
judge_prompts.py(snapshotted per run); shared search rendering lives informatting.py.Search behavior is aligned across paths: query truncation,
CONTEXT_BLOCK_RERANKER/ scope-specific tool bounds (max_charactersforauto,limit/rerankerotherwise), and startup validation for unsupported rerankers.retry.pystops retrying 400/422 immediately.Results add
retrieval_configuration, richer timing (answer_latency_*, tool vs answer-LLM split), tool aggregates, and per-test tool fields;response_*timing meaning changed and the completeness rubric is tool-aware—baselines should be re-run before comparing to older runs. README and SKILL docs describe modes, fair comparisons, and how to read the new metrics.Reviewed by Cursor Bugbot for commit 2cff0ed. Bugbot is set up for automated code reviews on this repo. Configure here.