Skip to content

v0.12.0 - Model unloading 🪶

Latest

Choose a tag to compare

@achetronic achetronic released this 17 Nov 12:33
· 1 commit to master since this release
0b3a0ef

Parakeet can now let go of its models and load them again, so an idle server is not holding 2 GB of RAM or VRAM for nothing.

✨ Features

  • Free the models when idle. -model-unload-after-idle-duration 5m releases the ASR model, the VAD and diarization after that long without requests. 0 is the default and keeps them resident.
  • Or on demand. POST /v1/models/unload frees them right away and POST /v1/models/load loads them back. GET /v1/models reports whether the weights are in memory.
  • The next request reloads them. Nothing else to trigger: the sessions are built again when they are needed, in about 3.5 s on CPU, paid by the first request after an unload.
  • The memory really comes back. Destroying the ONNX sessions left the memory in the C allocator, so the server now hands it back to the OS: 1.08 GB of RSS down to 48 MB with int8 on CPU, 490 MiB of VRAM down to 262 MiB on CUDA.

⚠️ Notes

  • On CUDA the process keeps its context, about 150 MiB of VRAM over an idle desktop. What the unload frees is the weights, not that.