feat(adapters): request-rewriting interceptors#4
Closed
Glorf wants to merge 2 commits into
Closed
Conversation
Introduces an interceptor-based middleware framework that runs at every
in-tree server's request/response boundary. PR 1 of 4 — base framework
only. Follow-on PRs add observability / caching / request-rewriting
interceptor families.
Framework
nemo_gym/adapters/
pipeline.py AdapterPipeline + stage-order validation
(REQUEST → REQUEST_TO_RESPONSE → RESPONSE)
middleware.py install_middleware(app, specs) — FastAPI middleware
that wraps call_next; body replay, multi-Set-Cookie
preservation, /s/<hex>/... session-prefix routing,
GracefulError → 429
proxy.py start_adapter_proxy(upstream_url, adapters) —
localhost uvicorn host mode for external-inference
agents whose SDK respects *_BASE_URL env vars
registry.py short-name → Interceptor class + runtime register()
types.py AdapterRequest/Response, three Interceptor ABCs,
Stage enum, GracefulError, ContextVar-backed
per-request context, InterceptorSpec and
AdapterProxyConfig pydantic models
interceptors/
endpoint.py Drives upstream HTTP call (required by proxy mode,
forbidden inside install_middleware)
request_logging.py
Canonical "did the chain fire?" probe (logs body
keys + response status/latency)
Server wire-up
adapters: list[dict] | None = None on three base configs:
BaseResponsesAPIModelConfig
BaseResponsesAPIAgentConfig
BaseResourcesServerConfig
install_middleware(app, self.config.adapters) at the tail of each
base's setup_webserver. Every in-tree server inheriting from
SimpleResponsesAPI{Model,Agent} or SimpleResourcesServer picks up
the adapters knob automatically.
adapter_proxy: AdapterProxyConfig | None = None on
BaseResponsesAPIAgentConfig. When set, SimpleResponsesAPIAgent.
setup_webserver starts a localhost uvicorn proxy in a daemon thread,
stores the ProxyHandle on self._proxy_handle, registers atexit cleanup.
Per-server override fixes
harbor_agent and mini_swe_agent override setup_webserver without
calling super(). Inline install_middleware(app, self.config.adapters)
calls added to both so the adapters field still applies.
claude_code_agent integration
Reads self._proxy_handle.url when adapter_proxy is set; threads the
proxy URL into the claude CLI subprocess via ANTHROPIC_BASE_URL +
ANTHROPIC_AUTH_TOKEN. Preserves the full model-name prefix in proxy
mode (no .split("/")[-1] stripping).
Safety
- Stage ordering validated at startup
- Unknown interceptor name raises at config-validation time
- install_middleware rejects `endpoint` in chain (host already forwards)
- start_adapter_proxy rejects user-supplied `endpoint` AND refuses
host="0.0.0.0" unless unsafe_allow_remote=True (otherwise leaks
the upstream API key to any caller on the network)
- best_effort=True interceptors swallow exceptions; strict ones
propagate
Tests
42 tests, framework-level coverage. Follow-on PRs add per-interceptor
test suites.
Docs
fern/versions/latest/pages/model-server/adapters.mdx new
fern/versions/latest/pages/model-server/index.mdx link added
Signed-off-by: Michal Bien <mbien@nvidia.com>
PR 4 of 4 — request-rewriting family. Built on feat/adapter-base
(independent of the observability and caching PRs).
Adds 6 interceptors that mutate the outbound request body before it
reaches the upstream:
- drop_params remove named parameters from outbound body
- payload_modifier add / remove / rename body fields
- system_message inject a system message (prepend / append /
replace)
- consolidate_system merge displaced system messages into one at
position 0 (Qwen3 requires this; other models
tolerate it)
- modify_tools strip or add properties on
tools[].function.parameters
- turn_counter per-session turn budget; raises GracefulError
(-> 429) on exhaustion; injects warn/urgent
system messages at 80%/95% of budget
All six are registered as builtins in InterceptorRegistry._BUILTIN.
Tests
test_adapter_interceptors.py drop_params, modify_tools,
system_message (prepend / replace),
payload_modifier (add / remove),
turn_counter (basic + session
isolation)
test_adapter_consolidate_system.py displaced-system-message merging
edge cases
test_adapter_parity_replay.py 6 captured-fixture scenarios
81 tests pass (42 from feat/adapter-base + 39 added).
Docs
fern/.../model-server/adapters.mdx appends Request-Rewriting section
with per-interceptor YAML config
snippets.
Signed-off-by: Michal Bien <mbien@nvidia.com>
2 tasks
Owner
Author
|
Superseded by NVIDIA-NeMo#1651 — moved upstream and rebased clean onto upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 4 of 4 — request-rewriting family. Built on feat/adapter-base (independent of the observability and caching PRs).
Adds 6 interceptors that mutate the outbound request body before it reaches the upstream:
replace)
position 0 (Qwen3 requires this; other models
tolerate it)
tools[].function.parameters
(-> 429) on exhaustion; injects warn/urgent
system messages at 80%/95% of budget
All six are registered as builtins in InterceptorRegistry._BUILTIN.
Tests
test_adapter_interceptors.py drop_params, modify_tools,
system_message (prepend / replace),
payload_modifier (add / remove),
turn_counter (basic + session
isolation)
test_adapter_consolidate_system.py displaced-system-message merging
edge cases
test_adapter_parity_replay.py 6 captured-fixture scenarios
81 tests pass (42 from feat/adapter-base + 39 added).
Docs
fern/.../model-server/adapters.mdx appends Request-Rewriting section
with per-interceptor YAML config
snippets.