Skip to content

Feature/server multipart transcriptions - #25

Merged
0xShug0 merged 3 commits into
0xShug0:release-0.1from
dkruyt:feature/server-multipart-transcriptions
Jul 3, 2026
Merged

Feature/server multipart transcriptions#25
0xShug0 merged 3 commits into
0xShug0:release-0.1from
dkruyt:feature/server-multipart-transcriptions

Conversation

@dkruyt

@dkruyt dkruyt commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Two small additions to audiocpp_server that make it a better drop-in for OpenAI-API-compatible clients (Open WebUI, llama-swap's playground, etc.):

  • Accept multipart/form-data uploads on POST /v1/audio/transcriptions. The real OpenAI Whisper API (and every client built against it — Open WebUI included) uploads audio as a multipart/form-data file part. audiocpp_server currently only accepts a JSON body with a server-local path (audio/audio_path/file as a string). That's fine for local scripting, but it means the endpoint can't be used as-is behind a client that does real file uploads. This adds a small multipart parser (app/server/multipart.{h,cpp}) and routes requests with a multipart/form-data Content-Type through it: the uploaded bytes are spooled to a temp file and handed to the existing JSON-based request builder, so the original JSON/path request shape keeps working unchanged.

  • Add GET /v1/audio/voices?model=<id>. Clients that build a voice picker (llama-swap's playground does this) have no way to discover which cached_voice_id values a TTS model actually supports, so they fall back to generic placeholder names (alloy, nova, ...) that don't exist as real presets and 500 at inference time. For families that keep voice presets under model_root/embeddings/*.safetensors (pocket_tts today — see assets.cpp), this lists the available ids. Families without an embeddings directory, or an unknown/missing model query param, just report an empty list, matching how the frontend already falls back gracefully. This also required capturing the raw query string on HttpRequest, which read_http_request was previously discarding entirely.

Neither change touches the existing JSON-only request shapes — both are purely additive.

Testing

  • New unit test exercising the multipart parser directly: boundary extraction (unquoted/quoted), multi-part splitting, and byte-exact handling of binary payloads with embedded NUL bytes.
  • Built audiocpp_server with CUDA (-DENGINE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DGGML_NATIVE=ON) and ran it against real weights (Qwen3-ASR-0.6B, kyutai/pocket-tts), on an actual GPU:
    • POST /v1/audio/transcriptions with a JSON {"model","audio"} body (existing behavior) — unchanged, verified no regression.
    • POST /v1/audio/transcriptions with a real multipart/form-data upload (-F file=@... -F model=..., with and without language) — new behavior, correct transcript returned, temp file cleaned up afterward.
    • Missing file/model multipart fields — errors out same as other malformed requests (500, consistent with existing error handling style).
    • GET /v1/audio/voices?model=pocket-tts — returns the real cached voice ids (alba, cosette, marius, ...).
    • GET /v1/audio/voices?model=qwen3-asr (no embeddings dir), ?model=<unknown>, and no model param — all return {"voices":[]}.
    • POST /v1/audio/speech — unchanged, verified no regression after both changes.
  • Deployed behind llama-swap in front of Open WebUI and confirmed both endpoints work end-to-end through a real client: STT via multipart upload, and TTS with the voice picker now populated from /v1/audio/voices instead of guessing.

Notes for reviewers

  • handle_transcription now dispatches on Content-Type (multipart/form-data → new path, otherwise → existing JSON path) rather than replacing the JSON handling.
  • The multipart parser is intentionally minimal (no external dependency) and mirrors the existing hand-rolled HTTP parsing style already used in http.cpp.
  • Voice discovery is scoped to "does model_root/embeddings exist" rather than hardcoding pocket_tts, so it should pick up other families automatically if they adopt the same layout.

@0xShug0

0xShug0 commented Jul 3, 2026

Copy link
Copy Markdown
Owner

@dkruyt Thank you for your contribution! I’ll test the PR, and if everything looks good, I’ll merge it.

@0xShug0

0xShug0 commented Jul 3, 2026

Copy link
Copy Markdown
Owner

@dkruyt Two small follow-ups before merge:

  1. Could you update app/server/README.md to document multipart transcription uploads and GET /v1/audio/voices?model=<id>?
  2. The PR description mentions a new multipart parser unit test, but I do not see a test file or CTest target in the diff. Could you add it?

dkruyt added 3 commits July 3, 2026 20:22
OpenAI's Whisper API (and clients built against it, e.g. Open WebUI) POST
the audio as a multipart/form-data "file" part rather than a JSON body
with a server-local path. Add a small multipart parser and route
transcription requests with a multipart Content-Type through it: the
uploaded bytes are spooled to a temp file and passed through the existing
JSON-based request builder, so the plain JSON path (audio/audio_path/file
as a string path) keeps working unchanged.
Clients that build a voice picker (e.g. llama-swap's playground) have no
way to learn which cached_voice_id values a TTS model actually supports,
so they fall back to generic placeholder names (alloy, nova, ...) that
don't exist as real presets and 500 at inference time.

For families that keep voice presets under model_root/embeddings/*.safetensors
(pocket_tts today, see assets.cpp), list the available ids. Families without
an embeddings directory just report no voices, same as an unknown or missing
model id.

Also required capturing the raw query string in HttpRequest, which read_http_request
was previously discarding entirely.
…t test

Addresses review feedback on the multipart-transcriptions PR: the
README didn't mention the new multipart upload path or the
/v1/audio/voices endpoint, and the multipart parser had no test
coverage or CTest target.
@dkruyt
dkruyt force-pushed the feature/server-multipart-transcriptions branch from cc55db4 to 1b9fc76 Compare July 3, 2026 18:22
@dkruyt

dkruyt commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Both addressed:

  • app/server/README.md now documents the multipart /v1/audio/transcriptions upload (with a curl -F example) alongside the existing JSON path, and adds a section for GET /v1/audio/voices?model=<id> with an example request/response.
  • Added tests/unittests/test_server_multipart.cpp plus a server_multipart_test CTest target (wired into CMakeLists.txt next to the other unit tests).

@0xShug0
0xShug0 merged commit 0243ca4 into 0xShug0:release-0.1 Jul 3, 2026
4 checks passed
@0xShug0

0xShug0 commented Jul 3, 2026

Copy link
Copy Markdown
Owner

@dkruyt merged. Thanks!

dleiferives pushed a commit to dleiferives/audio.cpp that referenced this pull request Jul 25, 2026
…anscriptions

Feature/server multipart transcriptions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants