Skip to content

AISG v1.1.0 — Streaming Chat Completions

Latest
Latest

Choose a tag to compare

View all tags
@alpha-cen alpha-cen released this 08 May 23:15
· 18 commits to main since this release
v1.1.0
6455abd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's New

Streaming Chat Completions (SSE)

Requests with stream: true are now fully supported. The gateway scans the full request through DLP before forwarding, then streams the LLM response back as Server-Sent Events — matching the OpenAI streaming spec.

  • Drop-in compatible — set stream: true in your existing OpenAI SDK calls, no other changes needed
  • DLP on every request — PII redaction and prompt injection blocking run on the full request before streaming begins
  • Token usage included — stream_options.include_usage is automatically enabled; usage appears in the final chunk
  • Graceful error handling — mid-stream errors return a structured error chunk followed by data: [DONE]

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="your-key")

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

What Changed

Added

  • Full SSE streaming support for chat completions
  • OpenAI-compatible streamed responses
  • Streaming passthrough while preserving request-side enforcement

What Does NOT Change

  • Request-side DLP (PII redaction, prompt injection blocking) — identical behavior
  • Provider resolution and routing — same logic
  • API key auth, rate limiting, fail-closed security — all unchanged
  • Response-side DLP scanning — not in scope (planned for a future release)

Files Changed

File Change
proxy-api/app/providers.py Added forward_chat_completion_stream()
proxy-api/app/routers/proxy.py Removed stream: true rejection, added SSE streaming path
proxy-api/tests/test_proxy.py Updated test to verify streaming is accepted

Links

Documentation

  • Managed Cloud — 1M free credits, no credit card

PR

Full Changelog

v1.0.0...v1.1.0

Release Notes

When publishing:

  • Set as Latest release
  • Enable Create a discussion for this release
Assets 2
Loading