[RFC] IOChain: request/response filters for OpenAI-compatible serving

[mukeshbaphna]

RFC: IOChain request/response filters for SGLang inference serving

Author: Mukesh Baphna (@mukeshbaphna)
Status: Draft / seeking maintainer feedback
Related:

• [Feature] Add a Plugin System to SGLang for Extensible Runtime Customization #13825
• [Feature] support add custom middlewares #6621

Summary

SGLang should provide a small, explicit request/response filter pipeline at the OpenAI-compatible serving layer.

The goal is to let deployments inspect, reject, annotate, or observe inference requests and responses without forking SGLang or monkey-patching internal code. This proposal is intentionally narrower than a general plugin system: it focuses only on request/response lifecycle hooks around OpenAI-compatible serving.

Motivation

Today, customization is possible at the HTTP middleware layer, but HTTP middleware only sees raw transport-level request/response data. It does not have structured access to:

• Parsed OpenAI-compatible request objects.
• SGLang internal request metadata.
• Generated responses.
• Usage data, latency, and completion metadata.
• A shared per-request context across ingress and egress.

This leaves production users with weak extension points for common serving needs such as:

• Request auditing.
• Usage and latency export.
• Request-level policy checks.
• Guardrail integration.
• Request annotation.
• Lightweight response inspection.

The broader need overlaps with the plugin-system request in #13825, while #6621 covers custom HTTP middleware. IOChain would complement those efforts by adding a structured inference-serving hook point rather than another raw HTTP middleware layer.

Goals

• Add a minimal async filter interface for request and response lifecycle hooks.
• Support blocking filters for policy and guardrail use cases.
• Support non-blocking filters for telemetry and audit use cases.
• Preserve the existing inference path unless filters are configured.
• Keep the no-filter and no-op-filter overhead negligible and measurable.
• Provide one clear registration path for server operators.
• Support streaming and non-streaming OpenAI-compatible endpoints.

Non-goals

• This is not a full SGLang plugin system.
• This does not add scheduler-level, engine-level, model-loading, tokenizer, quantization, or sampling plugins.
• This does not require user filters to run inside the scheduler process.
• This does not replace HTTP middleware.
• This does not define a distributed tracing standard.
• This does not expose unstable internal objects as a long-term public API unless maintainers explicitly want that.

Proposed API

from sglang.srt.iochain import IOContext, IOFilter


class MyFilter(IOFilter):
    blocking = True

    async def on_request(self, ctx: IOContext) -> None:
        ...

    async def on_response(self, ctx: IOContext) -> None:
        ...

blocking = True means the request waits for the hook to finish.

blocking = False means SGLang schedules the hook as a background task. This is intended for telemetry, audit logging, and best-effort export.

Proposed context object

@dataclass(frozen=True)
class IOContext:
    request_id: str
    raw_request: Any | None
    adapted_request: Any | None
    response: Any | None
    metadata: Mapping[str, Any]
    start_time: float

Open question: metadata may need to be mutable if filters are expected to share state between ingress and egress. If SGLang prefers immutable context objects, hooks can return an updated context instead.

Lifecycle

For non-streaming requests:

1. Build IOContext after the OpenAI-compatible request is parsed.
2. Run on_request filters in configured order.
3. Continue with existing SGLang serving flow.
4. Attach response and usage data to context.
5. Run on_response filters in configured order.
6. Return the response to the client.

For streaming requests:

1. Run on_request before stream generation starts.
2. Stream chunks as SGLang does today.
3. Build final response/usage metadata when the stream completes.
4. Run on_response after completion.

Open question: if a streaming request is disconnected or cancelled, should on_response run with cancellation/error metadata?

Initial registration proposal

Add a CLI/config option similar in spirit to middleware registration:

python -m sglang.launch_server \
  --io-filter my_package.filters.AuditFilter \
  --io-filter my_package.filters.PolicyFilter

Each value resolves to a Python class or factory. Filters run in the order provided.

Open question: maintainers may prefer config-file registration instead of, or in addition to, CLI registration.

Error handling

For blocking filters:

• Exceptions from on_request should fail the request with a clear server-side error path.
• A filter should be able to reject a request intentionally, ideally with an explicit exception type such as IOFilterReject.
• Exceptions from on_response should be logged and should not corrupt an already generated response unless maintainers prefer strict failure semantics.

For non-blocking filters:

• Exceptions should be captured and logged.
• Background tasks should not fail the client request after the response path has continued.

Request mutation

This RFC proposes starting with observation plus explicit rejection.

Request mutation can be added later if maintainers want prompt rewriting or request normalization. Starting without mutation keeps the first version easier to reason about and avoids unclear ownership of validation, tokenization, and scheduler invariants.

Open question: should v1 allow mutation of safe fields, or should mutation be deferred?

Security considerations

• Filters are server-side trusted code loaded by the operator.
• Filter import paths should not be accepted from untrusted user input.
• Errors should not leak sensitive request data to clients.
• Logs should avoid printing full prompts or completions by default.
• Non-blocking filters should not retain unbounded request/response data in memory.

Performance considerations

• No filters configured: no meaningful behavior change.
• No-op filter chain: target overhead should be measured and reported.
• Blocking filters are explicitly on the request critical path.
• Non-blocking filters should use bounded background execution or clear documentation to avoid unbounded task growth under load.

Compatibility

This should be additive and disabled by default.

Existing HTTP middleware behavior should remain unchanged. IOChain should be compatible with existing OpenAI-compatible endpoints and should not require users to modify model code.

Proposed implementation phases

Phase 1: minimal observability hooks

• Add IOFilter, IOContext, and filter-chain execution.
• Register filters through CLI/config.
• Support on_request and on_response.
• Support non-streaming endpoints.
• Add unit tests for ordering, blocking/background behavior, and error handling.

Phase 2: streaming completion hooks

• Add final-stream egress hook support.
• Include cancellation/error metadata where available.
• Add tests for completed streams and interrupted streams.

Phase 3: built-in examples

• Add a simple audit/logging example.
• Add a token/usage counting example if maintainers want one.
• Document performance expectations and production cautions.

Phase 4: optional policy/rejection support

• Add explicit rejection exception and response mapping.
• Decide whether limited request mutation belongs in IOChain.

Questions for maintainers

1. Is the OpenAI-compatible serving layer the right place for this extension point?
2. Should the first version be observation/rejection only, or should request mutation be included?
3. What registration mechanism is preferred: CLI, config file, Python API, or multiple?
4. What should happen when streaming requests are cancelled before final usage metadata exists?
5. Should this live as a narrow IOChain feature or be shaped as the first slice of the broader plugin system in [Feature] Add a Plugin System to SGLang for Extensible Runtime Customization #13825?
6. Would maintainers prefer a design-only RFC first, or a small draft PR behind a disabled-by-default flag?

Suggested next step

If maintainers are open to this direction, I can open a draft implementation PR for Phase 1 with tests and benchmark numbers for no-filter and no-op-filter overhead.