Feature Request: Move embedding generation off the transcription critical path

[devkit415]

Description

Transcription jobs currently generate transcript chunk embeddings synchronously at the end of every transcription, inside the worker thread that just ran the ASR call.

On the instance measured for this request, a 30-second clip takes about 16.4 seconds of processing time, of which roughly 2.4 seconds is the ASR call and the remaining ~14 seconds is embedding generation. This makes the transcription queue drain roughly five times slower than the ASR alone would allow, and it blocks completion status until embeddings finish.

The recording is already marked COMPLETED before chunks are built, and an embedding failure does not fail the recording, so deferring this work is safe and would not change the user-visible success path.

Affected Version

v0.10.18-alpha

Requested Feature

Move process_recording_chunks (src/services/embeddings.py:547) off the transcription critical path so that a transcription job finishes as soon as the ASR result is stored, and embeddings are generated afterward without blocking the worker that is serving the queue.

The recording's COMPLETED status and transcript must be available to the user before embeddings finish.

Proposed Implementation

• Invoke process_recording_chunks asynchronously after finish_processing marks the recording COMPLETED (src/tasks/processing.py:346), instead of synchronously inside it. Options: enqueue it as a low-priority job in the existing FairJobQueue (src/services/job_queue.py:42), run it in a separate thread pool or process pool, or generate embeddings lazily on first search or detail access.
• Keep the existing safety property: embeddings are generated before old chunks are deleted, and a failure leaves existing chunks intact (src/services/embeddings.py:547-560). The recording stays COMPLETED regardless of embedding outcome.
• Preserve the per-user billing path: generate_embeddings bills the recording owner for API-mode usage, so the deferred path must still pass recording.user_id.
• If enqueuing as a job, add a job type (for example embed_chunks) to the transcription category so it reuses the existing worker pool and fair scheduling, or to a separate low-priority pool so it cannot starve transcription.
• Keep ENABLE_INQUIRE_MODE gating: when it is false, skip chunk processing entirely, as today.

Expected Behavior

• A 30-second clip reports COMPLETED within about 2-3 seconds instead of ~16 seconds.
• Queue drain for a 10-file burst of 30-second clips drops from over 5 minutes to roughly 25-30 seconds, because the ASR is the only serial step and it processes one clip in about 2.3 seconds.
• Embedding failures no longer delay COMPLETED status or block the transcription worker.
• Search and detail views that depend on chunks either wait for embeddings to finish or fall back gracefully, without showing stale or partial data.
• No change to the JSON payload, status codes, or authorization decisions.

Root Cause Analysis

finish_processing in src/tasks/processing.py:346 calls process_recording_chunks(recording_id) synchronously after setting recording.status = 'COMPLETED'. process_recording_chunks (src/services/embeddings.py:547) calls chunk_transcript_segments and then generate_embeddings, which runs a local all-MiniLM-L6-v2 model and takes ~14 seconds for a 30-second clip. This runs inside the transcription worker thread (src/services/job_queue.py:485 _run_transcription -> transcribe_audio_task -> finish_processing), so the worker cannot claim the next job until embeddings finish. With 2 workers and 10 jobs, the queue drains in over 5 minutes even though the ASR itself processes a clip in ~2.3 seconds.

The work is CPU-bound and blocks the GIL (src/services/job_queue.py:108-118 uses threading.Thread), so the second worker does not provide real parallelism for embeddings. The ASR service itself serializes requests, so it is not the bottleneck; embedding generation is.

Relevant files: src/tasks/processing.py:346 (finish_processing), src/services/embeddings.py:547 (process_recording_chunks), src/services/job_queue.py:24 (TRANSCRIPTION_WORKERS), src/services/job_queue.py:108-118 (worker threads), src/services/job_queue.py:485 (_run_transcription).

[devkit415]

Converted into PR #545.

[murtaza-nasir]

The measurements in PR #545 match what I saw here. In the next release, the Inquire chunks of a completed recording are built by an index job on its own worker (INDEX_QUEUE_WORKERS, default 1), so the next transcription or summary can start without waiting for the embedding model. A failed build is retried, and the recording stays completed. Rebuilds after transcript and speaker edits use the same job, as your PR proposed. This will be included in the next release.

[devkit415]

Great job! Thank you!