feat(translate): eager SSE Prelude — TTFB decouples from upstream prefill

[steventohme]

Stacked on #$(gh pr list --head makosblade/inject-model-choice-non-anthropic --json number --jq '.[0].number' 2>/dev/null || echo '?') (the routing-marker-injection PR). Assume that merges first.

Why

Cross-format SSE clients (and Codex via /v1/responses) waited the full upstream prefill + first-decode for any visible byte, because no frame flushed until upstream's first `Write()`. Even after the routing decision was made we sat on it.

What

Add a `Prelude(streaming bool)` method on each outermost SSE writer. Proxy entry points call it right after the routing decision, which commits HTTP 200 and emits the writer's format-specific first frame immediately.

Path	Wrapper	Prelude emits
Claude Code → non-Anthropic upstream	`AnthropicSSETranslator`	`message_start` + routing-marker text block
Cursor / OpenAI chat completions	`OpenAIRoutingMarkerWriter`	marker `chat.completion.chunk`
Codex /v1/responses	`ResponsesWriter`	`response.created`
Gemini same-format	`GeminiRoutingMarkerWriter`	marker candidate chunk

The marker-wrap is skipped when the outer sink is already `ResponsesWriter` (Codex injects its own badge on the first text delta — wrapping would produce a duplicate).

Trade-off

Prelude locks the response to HTTP 200. Any later upstream error must surface as an in-stream error event rather than an HTTP status. Clients already handle this; middle-boxes that look at HTTP status won't see 5xx.

Test plan

• `go test ./internal/translate/... ./internal/proxy/...` green
• New: `TestOpenAIRoutingMarkerWriter_PreludeFiresBeforeUpstream` — proves marker flushes without any upstream Write, and the upstream-triggered fallback path skips re-emitting.
• New: `TestOpenAIRoutingMarkerWriter_PreludeNoOpWhenNonStreaming` — non-streaming requests untouched.
• Manual: hit /v1/messages with Claude Code, /v1/chat/completions with Cursor, /v1/responses with Codex; confirm visible first-frame latency drops to ~routing time.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7c7dbac. Configure here.}

[cursor]

Gemini routing marker lacks ingress stripping

Medium Severity

The new GeminiRoutingMarkerWriter injects routing markers into Gemini-format SSE responses as real content (text parts in candidates). StripRoutingMarkerFromMessages is called on the Anthropic and OpenAI ingress paths but not on the Gemini path (ProxyGemini), and it only handles messages[] format — not Gemini's contents[].parts[].text. Clients that echo back conversation history will re-send the marker every turn, compounding token waste and destabilizing prefix caching at Google.

Additional Locations (1)

• internal/translate/gemini_marker.go#L97-L109

 

^{Triggered by learned rule: Router-injected response decorations must be stripped on ingress}

^{Reviewed by Cursor Bugbot for commit 7c7dbac. Configure here.}

[cursor]

Prelude skips headersEmitted guard unlike other implementations

Low Severity

AnthropicSSETranslator.Prelude calls t.inner.WriteHeader(http.StatusOK) unconditionally without checking t.headersEmitted first. All three other Prelude implementations (GeminiRoutingMarkerWriter, OpenAIRoutingMarkerWriter, ResponsesWriter) guard this call with a headersEmitted/httpHeadersSent check. This inconsistency could cause a double WriteHeader call on the inner writer if the control flow ever changes so that WriteHeader is called before Prelude.

 

^{Reviewed by Cursor Bugbot for commit 7c7dbac. Configure here.}