Feature Request: Move ffmpeg conversion out of the upload request thread

[devkit415]

Description

When AUDIO_COMPRESS_UPLOADS is enabled, ingest_uploaded_recording probes the uploaded file and runs ffmpeg conversion synchronously inside the request thread.

Under concurrent uploads this conversion consumes CPU on the request path and adds latency variance, even though the upload itself has already been accepted.

The conversion result is needed before the recording is stored and enqueued, but the work does not have to block the HTTP response.

Moving this work off the request thread would keep upload responses fast and predictable under burst load.

Affected Version

v0.10.18-alpha

Requested Feature

Move the probe and ffmpeg conversion step in ingest_uploaded_recording (src/api/recordings.py:2323) off the HTTP request thread, so that uploads return promptly and conversion runs as a background or queued step before storage and enqueue.

Proposed Implementation

• Identify the conversion block in ingest_uploaded_recording (src/api/recordings.py:2560-2692), which calls get_codec_info, convert_if_needed (src/api/recordings.py:2659) and related probe/repair helpers, and extract it into a pre-processing step that can run outside the request thread.
• Store the uploaded file in staging first, then enqueue a conversion job (or run it in a background thread pool) that produces the final media path; only after conversion completes should the recording row be committed and the transcription job enqueued.
• Keep the empty-file refusal (src/api/recordings.py:2476-2480) and the duplicate-hash computation (src/api/recordings.py:2481-2522) in the request path, since both are cheap and the hash must be computed on the original bytes before any re-encoding.
• Preserve all current conversion outcomes and error codes: audio_converter_missing (500), audio_conversion_timeout (500), audio_decode_failed (400) and the video retention/passthrough branches.
• Keep AUDIO_COMPRESS_UPLOADS (src/config/app_config.py:92) and AUDIO_CODEC/AUDIO_BITRATE behavior unchanged; only the execution location changes.
• Ensure the staging file is cleaned up on conversion failure, as today (_discard_staging).

Expected Behavior

• Upload responses return without waiting for ffmpeg, so burst upload latency stays low and predictable.
• Conversion still happens before transcription is enqueued, so the ASR receives the same converted media as before.
• All existing conversion error codes and status codes remain unchanged.
• No change to the stored file format, hash, or duplicate detection.
• No orphaned staging files after failures.

Root Cause Analysis

ingest_uploaded_recording (src/api/recordings.py:2323) saves the uploaded file, computes its hash, then performs get_codec_info and convert_if_needed (src/api/recordings.py:2659) synchronously before committing the recording and enqueuing transcription.

convert_if_needed invokes ffmpeg, which is CPU-bound and can take seconds for large files. Under concurrent uploads this conversion runs on the request threads and competes with the ASR and embedding work for CPU, increasing upload latency variance.

The current behavior deviates from the desired one because conversion is coupled to the HTTP response even though its result is only needed before enqueue.

[murtaza-nasir]

I have implemented this for the next release. A file that ffprobe can read, with an audio track, is stored as it arrived, and the upload is answered at once. The file is then converted in a convert job on its own workers (CONVERT_QUEUE_WORKERS, default 2), and its transcription is queued once the conversion is done. Files that cannot be read, videos without an audio track, videos over the size limit whose extracted audio has to fit it, and parts of a join are still refused at once, with the same error codes. If a conversion fails later, the recording is marked as failed, with the error in place of the transcript, and the recording.transcription.failed webhook is sent. While the file waits or is converted, the recording is shown as Queued (conversion) or Converting, and stage in the status API is conversion. With CONVERT_QUEUE_WORKERS=0, files are converted in the upload request, as before.