Codex fails against custom OpenAI-compatible (Responses API) provider with `upstream_error`, while identical curl requests succeed

[justusaugust]

Summary

With a custom [model_providers.*] pointing at an OpenAI-compatible Responses API endpoint (Pioneer, https://api.pioneer.ai/v1), every Codex request fails:

ERROR: stream disconnected before completion: response.failed event received

The upstream SSE stream Codex receives is:

{"type": "response.failed", "sequence_number": 3,
 "response": {"id": "resp_...", "status": "failed", "error": null,
              "model": "gpt-5.5", "output": []},
 "error": {"message": "Upstream provider error",
           "type": "server_error", "code": "upstream_error"}}

The same provider works perfectly from curl and other clients. I captured Codex's exact request body + headers and replayed them with curl — they succeed. So the failure is specific to how the Codex client itself drives the connection, not the provider, the key, or billing.

Environment

• codex-cli 0.134.0
• macOS 26.2 (arm64), zsh
• Provider: Pioneer (OpenAI-compatible, Responses API), wire_api = "responses", Bearer auth via env_key
• Models tried: gpt-5.5, claude-opus-4-7, claude-sonnet-4-6 (all fail identically)

Config

model = "gpt-5.5"
model_provider = "pioneerx"

[model_providers.pioneerx]
name = "Pioneer"
base_url = "https://api.pioneer.ai/v1"
wire_api = "responses"
env_key = "PIONEER_API_KEY"

Steps to reproduce

CODEX_HOME=~/.codex-pioneer codex exec --skip-git-repo-check 'say hi'
# -> ERROR: stream disconnected before completion: response.failed event received  (x retries)

Evidence it is NOT the provider

All tests below were run within the same minute, same key, same account.

1. Direct curl to the provider — succeeds (6/6):

curl -sS -X POST "https://api.pioneer.ai/v1/responses" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","input":"hi","stream":true}'
# -> event: response.created ... event: response.completed   HTTP 200, output "OK"

Works streaming and non-streaming, for OpenAI-backed (gpt-5.5) and Anthropic-backed (claude-*) models.

2. curl replaying Codex's EXACT captured body + headers — succeeds.
I captured the full request body Codex sends (via RUST_LOG=codex_client::transport=trace) and POSTed it verbatim with curl --data @body.json plus Codex's headers (openai-beta, originator, session_id, user-agent, Accept: text/event-stream). Result: response.completed, HTTP 200.

3. Codex direct — fails, same minute as the passing curls.

So: identical endpoint + key + payload → curl completed, Codex response.failed.

Captured request shape (what Codex sends)

Top-level keys in the POST body to /v1/responses:

model, instructions (~21KB), input, tools (21), tool_choice="auto",
parallel_tool_calls=true, reasoning={"effort":"xhigh"}, store=false,
stream=true, include=["reasoning.encrypted_content"], service_tier,
prompt_cache_key, text={"verbosity":"low"}, client_metadata

Things ruled out

• Not billing/auth — a 401 would never reach response.created; auth succeeds and a resp_... id is issued before response.failed.
• Not wire_api — wire_api = "chat" is rejected by 0.134.0 ("no longer supported"); responses is the only option and is what the working curl uses.
• Not the provider id — same failure with the id pioneer and a custom id pioneerx.
• Not a transient outage — deterministic across dozens of attempts over ~24h; curl is simultaneously 100% green.
• A local HTTP/1.1 forwarding proxy — curl through the proxy succeeds, but Codex through the same proxy still fails, which points back at the Codex client.

Sample response IDs (provider-issued, for upstream tracing)

resp_bc3c68835f6d4d57aa9aad4a
resp_66bdf8575aa341e8a3b3eb1e
resp_606dc8f86e4f4a66baf57ac6
resp_54ff6cd3d7964f7f9c0f4f80

Ask

What does Codex do differently from a raw curl of the same body/headers against a wire_api = "responses" provider that would cause the upstream to emit response.failed? Happy to capture additional traces (full RUST_LOG, the exact serialized body) on request.