`POST /v1/realtime/calls` doesn't accept the OpenAI GA WebRTC request shape (multipart in, `application/sdp` out)

[mudler]

Summary

POST /v1/realtime/calls accepts only a JSON request body and answers with a JSON body. OpenAI's GA endpoint accepts multipart/form-data or application/sdp, and answers with the SDP as plain text. A client written against the documented OpenAI contract therefore cannot complete the WebRTC handshake with LocalAI — it fails at signaling with 400 sdp is required.

What OpenAI's endpoint does

Per the Realtime WebRTC guide, POST https://api.openai.com/v1/realtime/calls accepts two request shapes:

1. Unified interface — multipart/form-data with two fields:
   • sdp — the SDP offer (application/sdp)
   • session — the session config as JSON, which is where model lives, alongside audio.output.voice, instructions, turn_detection, and so on
2. Ephemeral token — Content-Type: application/sdp, the raw SDP offer as the body

In both cases the response is the SDP answer as plain text, fed straight to pc.setRemoteDescription(answer). It is not wrapped in JSON.

What LocalAI does

core/http/endpoints/openai/realtime_webrtc.go:

type RealtimeCallRequest struct {
	SDP   string `json:"sdp"`
	Model string `json:"model"`
	LocalAIAssistant bool `json:"localai_assistant,omitempty"`
}

type RealtimeCallResponse struct {
	SDP       string `json:"sdp"`
	SessionID string `json:"session_id"`
}

var req RealtimeCallRequest
if err := c.Bind(&req); err != nil { ... }
if req.SDP == "" {
	return c.JSON(http.StatusBadRequest, map[string]string{"error": "sdp is required"})
}
if req.Model == "" {
	return c.JSON(http.StatusBadRequest, map[string]string{"error": "model is required"})
}
...
return c.JSON(http.StatusCreated, RealtimeCallResponse{SDP: localDesc.SDP, SessionID: sessionID})

Three divergences:

1. Request encoding. The structs carry only json tags, so Echo's binder extracts nothing from a multipart body — every field stays empty and the handler returns 400 sdp is required. application/sdp isn't handled either.
2. Response encoding. A GA client reads the response body as the SDP answer. LocalAI returns {"sdp": ..., "session_id": ...}, so the client hands a JSON document to setRemoteDescription and the handshake fails.
3. Where model lives. LocalAI requires it at the top level. In the multipart flow OpenAI carries it inside the session object, so even a client that guessed the JSON shape would have to restructure its session config.

Reproducing

A real client that hits this: ESP-Brookesia's OpenAI agent (espressif/brookesia_agent_openai, v0.8.1), which implements the GA multipart flow in openai/https_client.c → https_post_realtime_call(). Pointed at a LocalAI instance it gets 400 sdp is required before any media flows.

By hand:

# Fails against LocalAI, works against api.openai.com
curl -X POST "$LOCALAI/v1/realtime/calls" \
  -H "Authorization: Bearer $KEY" \
  -F "sdp=<offer.sdp;type=application/sdp" \
  -F 'session={"type":"realtime","model":"my-model"};type=application/json'
# -> {"error":"sdp is required"}

Suggested fix

In RealtimeCalls(), branch on the request Content-Type before binding:

• multipart/form-data — read the sdp part as the offer and the session part as JSON, taking model from inside it (and ideally honouring the rest of the session config: voice, instructions, turn_detection).
• application/sdp — the body is the offer; take model from the ?model= query parameter, as the ephemeral-token flow does.
• application/json — current behaviour, kept for compatibility.

For the response, return Content-Type: application/sdp with the bare answer when the request came in either GA shape, and keep the JSON response for JSON requests. Content negotiation on the request encoding keeps every existing client working while making GA clients work too.

Why it's worth doing

The whole point of the OpenAI-compatible surface is that clients written for OpenAI work unmodified. Right now every embedded and browser client built on the documented GA WebRTC flow needs a patched fork to reach LocalAI. Accepting the documented shapes removes that.

[mudler]

@localai-org-maint-bot pick this up and open up a PR