Skip to content

Broker: stream responses through instead of buffering the full body #81

Description

@adamcohenhillel

The model broker buffers the entire upstream response before returning: response.read() at services/model-broker/openphone_model_broker.py:311. For stream:true Responses requests (the assistant's realtime path), this means the connection stalls for up to 120s (_proxy_to_openai timeout) and then dumps the buffered body — the assistant cannot stream tokens through broker mode.

Fix

  • Detect Accept: text/event-stream or stream:true in the payload.
  • Pass through the upstream response body as SSE chunks (urllib.response.HTTPResponse.read1 in a loop, flushing per chunk).
  • Preserve Content-Type: text/event-stream, Transfer-Encoding: chunked, and disable buffering in the response wfile.
  • Add a smoke test using a synthetic SSE upstream.

Size: M

Metadata

Metadata

Assignees

No one assigned

    Labels

    M4-runtimeM4: Provider & Runtime MaturityenhancementNew feature or request

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions