feat(adapters): observability interceptors#3
Closed
Glorf wants to merge 2 commits into
Closed
Conversation
Introduces an interceptor-based middleware framework that runs at every
in-tree server's request/response boundary. PR 1 of 4 — base framework
only. Follow-on PRs add observability / caching / request-rewriting
interceptor families.
Framework
nemo_gym/adapters/
pipeline.py AdapterPipeline + stage-order validation
(REQUEST → REQUEST_TO_RESPONSE → RESPONSE)
middleware.py install_middleware(app, specs) — FastAPI middleware
that wraps call_next; body replay, multi-Set-Cookie
preservation, /s/<hex>/... session-prefix routing,
GracefulError → 429
proxy.py start_adapter_proxy(upstream_url, adapters) —
localhost uvicorn host mode for external-inference
agents whose SDK respects *_BASE_URL env vars
registry.py short-name → Interceptor class + runtime register()
types.py AdapterRequest/Response, three Interceptor ABCs,
Stage enum, GracefulError, ContextVar-backed
per-request context, InterceptorSpec and
AdapterProxyConfig pydantic models
interceptors/
endpoint.py Drives upstream HTTP call (required by proxy mode,
forbidden inside install_middleware)
request_logging.py
Canonical "did the chain fire?" probe (logs body
keys + response status/latency)
Server wire-up
adapters: list[dict] | None = None on three base configs:
BaseResponsesAPIModelConfig
BaseResponsesAPIAgentConfig
BaseResourcesServerConfig
install_middleware(app, self.config.adapters) at the tail of each
base's setup_webserver. Every in-tree server inheriting from
SimpleResponsesAPI{Model,Agent} or SimpleResourcesServer picks up
the adapters knob automatically.
adapter_proxy: AdapterProxyConfig | None = None on
BaseResponsesAPIAgentConfig. When set, SimpleResponsesAPIAgent.
setup_webserver starts a localhost uvicorn proxy in a daemon thread,
stores the ProxyHandle on self._proxy_handle, registers atexit cleanup.
Per-server override fixes
harbor_agent and mini_swe_agent override setup_webserver without
calling super(). Inline install_middleware(app, self.config.adapters)
calls added to both so the adapters field still applies.
claude_code_agent integration
Reads self._proxy_handle.url when adapter_proxy is set; threads the
proxy URL into the claude CLI subprocess via ANTHROPIC_BASE_URL +
ANTHROPIC_AUTH_TOKEN. Preserves the full model-name prefix in proxy
mode (no .split("/")[-1] stripping).
Safety
- Stage ordering validated at startup
- Unknown interceptor name raises at config-validation time
- install_middleware rejects `endpoint` in chain (host already forwards)
- start_adapter_proxy rejects user-supplied `endpoint` AND refuses
host="0.0.0.0" unless unsafe_allow_remote=True (otherwise leaks
the upstream API key to any caller on the network)
- best_effort=True interceptors swallow exceptions; strict ones
propagate
Tests
42 tests, framework-level coverage. Follow-on PRs add per-interceptor
test suites.
Docs
fern/versions/latest/pages/model-server/adapters.mdx new
fern/versions/latest/pages/model-server/index.mdx link added
Signed-off-by: Michal Bien <mbien@nvidia.com>
PR 2 of 4 — observability family. Built on feat/adapter-base.
Adds 5 interceptors for inspecting traffic without changing semantics:
- log_tokens response log usage token counts + latency
- response_stats response accumulate counts / tokens / latency
- progress_tracking response optional webhook ping every N responses
- reasoning response normalize <think>...</think> OR reasoning
field into reasoning_content
- raise_client_errors response non-retriable 4xx -> RuntimeError;
429/408 pass through (retriable)
All five are registered as builtins in InterceptorRegistry._BUILTIN.
Tests
test_adapter_interceptors.py per-interceptor behaviour
test_adapter_parity_replay.py captured-fixture parity (5 scenarios)
adapter_fixtures/ logging, log_tokens, response_stats,
progress_tracking, raise_client_errors,
reasoning JSON fixtures
60 tests pass (42 from feat/adapter-base + 18 added).
Docs
fern/.../model-server/adapters.mdx appends Observability section with
per-interceptor YAML config snippets
Signed-off-by: Michal Bien <mbien@nvidia.com>
2 tasks
This was referenced Jun 22, 2026
Owner
Author
|
Superseded by NVIDIA-NeMo#1650 — moved upstream and rebased clean onto upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 2 of 4 — observability family. Built on feat/adapter-base.
Adds 5 interceptors for inspecting traffic without changing semantics:
field into reasoning_content
429/408 pass through (retriable)
All five are registered as builtins in InterceptorRegistry._BUILTIN.
Tests
test_adapter_interceptors.py per-interceptor behaviour
test_adapter_parity_replay.py captured-fixture parity (5 scenarios)
adapter_fixtures/ logging, log_tokens, response_stats,
progress_tracking, raise_client_errors,
reasoning JSON fixtures
60 tests pass (42 from feat/adapter-base + 18 added).
Docs
fern/.../model-server/adapters.mdx appends Observability section with
per-interceptor YAML config snippets