feat(adapters): caching interceptor#2
Closed
Glorf wants to merge 2 commits into
Closed
Conversation
Introduces an interceptor-based middleware framework that runs at every
in-tree server's request/response boundary. PR 1 of 4 — base framework
only. Follow-on PRs add observability / caching / request-rewriting
interceptor families.
Framework
nemo_gym/adapters/
pipeline.py AdapterPipeline + stage-order validation
(REQUEST → REQUEST_TO_RESPONSE → RESPONSE)
middleware.py install_middleware(app, specs) — FastAPI middleware
that wraps call_next; body replay, multi-Set-Cookie
preservation, /s/<hex>/... session-prefix routing,
GracefulError → 429
proxy.py start_adapter_proxy(upstream_url, adapters) —
localhost uvicorn host mode for external-inference
agents whose SDK respects *_BASE_URL env vars
registry.py short-name → Interceptor class + runtime register()
types.py AdapterRequest/Response, three Interceptor ABCs,
Stage enum, GracefulError, ContextVar-backed
per-request context, InterceptorSpec and
AdapterProxyConfig pydantic models
interceptors/
endpoint.py Drives upstream HTTP call (required by proxy mode,
forbidden inside install_middleware)
request_logging.py
Canonical "did the chain fire?" probe (logs body
keys + response status/latency)
Server wire-up
adapters: list[dict] | None = None on three base configs:
BaseResponsesAPIModelConfig
BaseResponsesAPIAgentConfig
BaseResourcesServerConfig
install_middleware(app, self.config.adapters) at the tail of each
base's setup_webserver. Every in-tree server inheriting from
SimpleResponsesAPI{Model,Agent} or SimpleResourcesServer picks up
the adapters knob automatically.
adapter_proxy: AdapterProxyConfig | None = None on
BaseResponsesAPIAgentConfig. When set, SimpleResponsesAPIAgent.
setup_webserver starts a localhost uvicorn proxy in a daemon thread,
stores the ProxyHandle on self._proxy_handle, registers atexit cleanup.
Per-server override fixes
harbor_agent and mini_swe_agent override setup_webserver without
calling super(). Inline install_middleware(app, self.config.adapters)
calls added to both so the adapters field still applies.
claude_code_agent integration
Reads self._proxy_handle.url when adapter_proxy is set; threads the
proxy URL into the claude CLI subprocess via ANTHROPIC_BASE_URL +
ANTHROPIC_AUTH_TOKEN. Preserves the full model-name prefix in proxy
mode (no .split("/")[-1] stripping).
Safety
- Stage ordering validated at startup
- Unknown interceptor name raises at config-validation time
- install_middleware rejects `endpoint` in chain (host already forwards)
- start_adapter_proxy rejects user-supplied `endpoint` AND refuses
host="0.0.0.0" unless unsafe_allow_remote=True (otherwise leaks
the upstream API key to any caller on the network)
- best_effort=True interceptors swallow exceptions; strict ones
propagate
Tests
42 tests, framework-level coverage. Follow-on PRs add per-interceptor
test suites.
Docs
fern/versions/latest/pages/model-server/adapters.mdx new
fern/versions/latest/pages/model-server/index.mdx link added
Signed-off-by: Michal Bien <mbien@nvidia.com>
PR 3 of 4 — caching family. Built on feat/adapter-base (independent of
the observability and request-rewriting PRs).
Adds the `caching` interceptor and its supporting sqlite disk-cache
module:
- caching (request -> response)
SHA-256-keyed disk cache. On cache hit, short-circuits the chain
as a RequestToResponseInterceptor; on miss, writes the upstream
response back into the cache via the matching ResponseInterceptor.
Cache keys are session-scoped when ctx.extra["session_id"] is
present.
- nemo_gym/adapters/cache/disk_cache.py
sqlite-backed cache with canonical-body hashing.
- nemo_gym/adapters/cache/__init__.py
Cache subpackage exports.
Tests
test_adapter_cache_keys.py golden SHA-256 keys
test_adapter_disk_cache.py sqlite round-trip
test_adapter_interceptors.py TestCachingInterceptor behaviour
test_adapter_parity_replay.py cache hit returns byte-equal response
58 tests pass (42 from feat/adapter-base + 16 added).
Docs
fern/.../model-server/adapters.mdx appends Caching section with config
example and per-replica-cache-race
caveat.
Signed-off-by: Michal Bien <mbien@nvidia.com>
2 tasks
This was referenced Jun 22, 2026
Owner
Author
|
Superseded by NVIDIA-NeMo#1649 — moved upstream and rebased clean onto upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 3 of 4 — caching family. Built on feat/adapter-base (independent of the observability and request-rewriting PRs).
Adds the
cachinginterceptor and its supporting sqlite disk-cache module:caching (request -> response) SHA-256-keyed disk cache. On cache hit, short-circuits the chain as a RequestToResponseInterceptor; on miss, writes the upstream response back into the cache via the matching ResponseInterceptor. Cache keys are session-scoped when ctx.extra["session_id"] is present.
nemo_gym/adapters/cache/disk_cache.py sqlite-backed cache with canonical-body hashing.
nemo_gym/adapters/cache/init.py Cache subpackage exports.
Tests
test_adapter_cache_keys.py golden SHA-256 keys
test_adapter_disk_cache.py sqlite round-trip
test_adapter_interceptors.py TestCachingInterceptor behaviour
test_adapter_parity_replay.py cache hit returns byte-equal response
58 tests pass (42 from feat/adapter-base + 16 added).
Docs
fern/.../model-server/adapters.mdx appends Caching section with config
example and per-replica-cache-race
caveat.