perf(harness): cache tool schemas per run + incremental cache_key fold#14
Conversation
The agent loop rebuilt the full tool-schema vector on every iteration via self.tools.schemas() — cloning each ToolSchema (name/description/parameters) and re-sorting by name — even though the registered tool set is fixed for a run. Hoist the computation to run start and clone the cached vector into each ModelRequest. Dynamic tool-selection middleware still operates on request.tools afterward (copy-on-filter), so per-turn behavior is unchanged; the build + sort now happen once instead of once per model call. The per-iteration Vec clone remains while ModelRequest.tools owns its Vec<ToolSchema>; eliminating it entirely is the coordinated Arc trait-break tracked separately (doc 04 §1). Adds a regression test (ToolCapturingModel) asserting a two-turn run presents the identical, non-empty schema set on every turn. Ref: docs/tinyagents-vendor-improvement-plan/04-performance.md §2
cache_key serialized the entire ModelRequest to a serde_json::Value, rebuilt the whole tree in canonical (key-sorted) form, then re-serialized it to bytes before hashing — three simultaneous whole-transcript allocations on every cached model call. OpenHuman transcripts routinely carry large tool results, so this O(history) round-trip is paid per call. Fold the request into the SHA-256 hasher one component at a time instead: each message and each tool schema becomes its own length-prefixed, canonicalized frame (tagged m/t with a preceding count), and the remaining scalar/parameter fields fold as a single envelope frame (tag E). Peak memory is now bounded by the largest single component rather than the whole request, and the length prefixes keep the frame stream unambiguous so distinct requests keep distinct keys. The envelope is taken as 'whatever remains after removing messages/tools', so any future ModelRequest field automatically participates in the key — no field can silently drop out and cause a false cache hit. The digest widened from the old 16-char note in the layout docs to the full 64-char SHA-256 hex (response-cache keys are in-memory only; no persisted format changes). Adds tests for message content/count/order sensitivity, tool-schema sensitivity, scalar-envelope sensitivity, and message-vs-tool disambiguation. Ref: docs/tinyagents-vendor-improvement-plan/04-performance.md §3
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 9 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
Comment |
1 participant
Summary
Two non-breaking agent-loop performance fixes (no public API change).
1. Cache tool schemas per run
AgentLoop::run_looprebuilt the fullVec<ToolSchema>on every iterationvia
self.tools.schemas()— cloning each schema's name/description/parametersand re-sorting by name — even though the registered tool set is fixed for a run.
Hoisted the computation to run start; the cached vector is cloned into each
ModelRequest(dynamic tool-selection middleware still filtersrequest.toolsafterward, copy-on-filter). Build + sort now happen once instead of per model
call.
2. Fold
cache_keyincrementally per componentcache_keyserialized the entireModelRequestto aserde_json::Value,rebuilt the whole tree in canonical (key-sorted) form, then re-serialized to
bytes — three simultaneous whole-transcript allocations on every cached call.
Now the request is folded into the SHA-256 hasher one component at a time: each
message and tool schema is its own length-prefixed, canonicalized frame, and the
remaining scalar fields fold as a single "envelope" frame (taken as whatever
remains after removing messages/tools, so any future field automatically
participates — no silent false cache hits). Peak memory is bounded by the largest
single component rather than the whole request.
Tests
tool_schemas_are_stable_across_turns— a two-turn run presents the identical,non-empty schema set on every turn.
cache_keysensitivity: message content/count/order, tool schemas, scalarenvelope, and message-vs-tool disambiguation.
Commands run
cargo fmt --checkcargo clippy --all-targets -- -D warningscargo test(786 lib + integration, green)Ref: OpenHuman
docs/tinyagents-vendor-improvement-plan/04-performance.md§2–3 (V1).