Skip to content

fix(translation): reject lossy buffered responses - #274

Draft
bbednarski9 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
bbednarski9:fix/translation-reject-lossy-responses
Draft

fix(translation): reject lossy buffered responses#274
bbednarski9 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
bbednarski9:fix/translation-reject-lossy-responses

Conversation

@bbednarski9

Copy link
Copy Markdown
Contributor

What

Makes LossyConversionPolicy effective when encoding buffered LLM responses:

  • OpenAI Chat and Anthropic encoders report multiple normalized outputs instead
    of silently selecting one.
  • All three response encoders reject unsupported content under Reject, or emit
    a diagnostic and retain their existing best-effort result under
    AllowWithDiagnostics.
  • OpenAI Chat decoding retains every choice so the encoder can detect that a
    target format cannot represent all of them.
  • Unknown top-level OpenAI Responses output items remain in the neutral response
    long enough for target encoders to diagnose the loss.

Exact same-format replay remains the first path and returns the preserved
provider body unchanged.

Why

Before this change, strict loss detection applied to requests but several
buffered response paths silently discarded extra outputs or unsupported content.
For a routing integration such as #270, that could return a weakened
cross-protocol response instead of surfacing a translation failure while the
trusted fallback was still available.

Related: #192, #270

How tested

  • uv run ruff check . clean (N/A: Rust-only change)
  • uv run mypy switchyard clean (N/A: Rust-only change)
  • uv run pytest tests/ green (N/A: Rust-only change)
  • cargo test -p switchyard-translation --all-targets
  • cargo test --workspace --all-targets
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

Checklist

  • One class per file; filename = snake_case of the primary class. (N/A: no Python classes added.)
  • New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use. (N/A: no Python symbols added.)
  • Unit tests added for new components / bug fixes.
  • README / --help updated if customer-facing surface changed. (N/A: behavior is governed by the existing translation policy.)
  • Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

  • This PR is limited to buffered response translation. feat(translation): preserve raw stream events #192 supplies the
    separate preserved-event contract for same-protocol streaming.
  • No protocol type or public API is added. AllowWithDiagnostics remains the
    explicit opt-in for best-effort conversion.
  • Unknown Responses output items replay exactly through normal same-format
    in-memory preservation. If preservation is disabled, strict encoding rejects
    them because the current neutral Unknown block cannot prove whether raw JSON
    originated at the output-item level or inside message content.
  • This PR is independent of feat(relay): add Switchyard-owned HTTP dynamic plugin #270: translation tests exercise the policy directly,
    while feat(relay): add Switchyard-owned HTTP dynamic plugin #270 converts the resulting finalization error into its existing
    exactly-once trusted-fallback flow.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant