v0.10.0 - Concurrent transcriptions 🧵

What's new

Transcriptions now run side by side. Until now only the last step of the pipeline (the decoder) ran in parallel, while every request crowded a single shared model up front. That was fine on a quiet CPU box; under load, and on GPU, it raced and crashed. Parakeet now bounds the whole request, audio in to text out, to -workers concurrent transcriptions, each on its own resources. A busy server queues cleanly instead of allocating until it dies.

Multi-GPU from the start. The old -gpu-device picked one GPU and ignored the rest. It's now -gpu-devices "0,1,2": list the cards or MIG slices you have, and Parakeet spreads its workers across them, each with its own encoder and decoder. Nothing is shared between two requests anymore, which removes the GPU crash altogether.

Memory, realistically. On GPU every worker keeps its own model in VRAM: about 670 MB per worker with int8, 2.5 GB with fp32. Match -workers to your card.

How to use it

# run 4 transcriptions at once
./parakeet -workers 4

# GPU: spread 4 workers over two cards (or two MIG slices)
./parakeet -gpu cuda -gpu-devices 0,1 -workers 4

Flag	Default	Description
-workers	4	Concurrent transcriptions (the whole pipeline)
-gpu-devices	0	Comma-separated CUDA device ids to spread workers over