feat: opt-in exact-match LLM response cache (stage 1/5)

[zhongxuanwang-nv]

Overview

First stage of a five-stage series (exact match → logical keys → tool-result cache → semantic → streaming semantic), each stage a single reviewable commit with a matching docs PR. This stage is fully standalone.

Adds an opt-in, exact-match response cache for managed LLM calls as a response_cache section of the adaptive plugin config (no new plugin kind, no call-site changes). Repeated identical calls are served from an in-memory or Redis store instead of re-running the provider — instant repeats in dev loops and evals, and reproducible runs via the bypass_rate trust dial. Off by default; every cache error fails open to a live call.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

• Keying (crates/adaptive/src/response_cache/key.rs): requests are keyed by a SHA-256 fingerprint of the RFC 8785-canonicalized, normalized request. The provider surface is auto-detected from the request shape (nothing to configure); the decode is trusted only where faithful — known-lossy shapes and decodes that fail a structural round-trip (tools, conversation, tool_choice, generation params) fall back to raw-body fingerprinting, which can only cost a miss, never a wrong reuse. Tool-call IDs are normalized; only allowlisted headers are keyed; namespace/provider/schema-version are folded in.
• Storage rules (response_cache/mod.rs): only complete, replayable answers are stored — non-null error bodies, non-final status, truncated streams (no protocol-terminal event), and lossily-collected aggregates (thinking blocks, refusal-only choices) are never cached. Stateful calls (Responses persist-by-default without store: false, conversation, container) and nondeterministic calls under cache_nondeterministic: false bypass entirely.
• Streaming: a miss tees the live stream and stores the codec-assembled aggregate (buffered and streaming share one keyspace); a hit replays provider-native chunks with the SSE content type.
• Store (response_cache/store.rs): in-memory backend bounded by an honest resident-size budget with oldest-first eviction; Redis backend (feature-gated) with hard per-op deadlines, native expiry plus an embedded-expiry re-check.
• Gateway integration (crates/cli/src/gateway.rs): non-2xx / non-JSON upstream replies flow back as Err so nothing downstream can cache them, while the client still receives the upstream response verbatim; short-circuited hits carry correct content types.
• Config/validation: rejects zero/mistyped caps, answer-determining skip_keys, and auth headers in header_allowlist. Same typed config in Rust, Python, Node, and Go; nemo-relay doctor reports configuration and backend reachability.
• Observability: response_cache hit/miss/bypass marks carry key fingerprints and savings, never bodies. Entries are stored unredacted (PII sanitize guardrails rewrite telemetry only), so shared Redis deployments must be trusted and namespaced — documented.

Validation run on this branch: cargo test -p nemo-relay-adaptive --lib --tests --features nemo-relay/openinference (all targets green, ~530 tests incl. integration + benchmark suites), gateway HTTP cache tests (server::tests::gateway_*, real mock upstream), cargo fmt --all --check, cargo clippy -- -D warnings on both crates. Python/Go surfaces are config mirrors validated by the shared Rust validator; their suites cover round-trip + validation parity. Known pre-existing sandbox failures (plugin-shim sidecar tests) are unrelated.

Where should the reviewer start?

1. crates/adaptive/src/response_cache/key.rs — the key-derivation contract and the lossy-decode guards (the core correctness decisions: when to trust the normalized decode vs fall back to raw keying).
2. crates/adaptive/tests/integration/response_cache_tests.rs — the behavioral contract as tests: exact repeats hit, guards force misses instead of wrong reuses, streams store only on terminal events.
3. crates/cli/src/gateway.rs — the upstream-failure boundary (design decision: failures relayed verbatim but invisible to the execution chain).

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to feat(codec): add a provider codec factory and route built-in consumers through it #396 — keying auto-detects the provider surface via codec::resolve; the local surface-name helpers in response_cache/key.rs are slated to migrate onto the provider codec factory once it lands.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ddaa2acd-2f43-48d5-ae10-26a2be084e46

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-404.docs.buildwithfern.com/nemo/relay (​https://nvidia-preview-pull-request-404.docs.buildwithfern.com/nemo/relay​)

[github-actions]

License Diff

Compared against origin/main.

Lockfile license changes

Lockfile License Changes

Rust

Added

• None

Removed

• None

Updated/Changed

• None

Node

Added

• None

Removed

• None

Updated/Changed

• None

Python

Added

• None

Removed

• None

Updated/Changed

• None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (378 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (363 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (105 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/main into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (378 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (363 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (105 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done