Repository navigation
Parakeet can now let go of its models and load them again, so an idle server is not holding 2 GB of RAM or VRAM for nothing.
✨ Features
- Free the models when idle.
-model-unload-after-idle-duration 5mreleases the ASR model, the VAD and diarization after that long without requests.0is the default and keeps them resident. - Or on demand.
POST /v1/models/unloadfrees them right away andPOST /v1/models/loadloads them back.GET /v1/modelsreports whether the weights are in memory. - The next request reloads them. Nothing else to trigger: the sessions are built again when they are needed, in about 3.5 s on CPU, paid by the first request after an unload.
- The memory really comes back. Destroying the ONNX sessions left the memory in the C allocator, so the server now hands it back to the OS: 1.08 GB of RSS down to 48 MB with int8 on CPU, 490 MiB of VRAM down to 262 MiB on CUDA.
⚠️ Notes
- On CUDA the process keeps its context, about 150 MiB of VRAM over an idle desktop. What the unload frees is the weights, not that.