kvad 0.20.0

Whisper on the GPU

Speech to text now runs on the GPU as well as the CPU (#244). On a Mac, NB-Whisper large writes down a minute and a half of speech in about three seconds.

On an M5 Pro, transcribing all 85 s of NB-Whisper's king.mp3 with nb-whisper-large and --timestamps:

backend	time	× real time
cpu-f32	48–52 s	1.7×
cpu-q8	33 s	2.5×
gpu-bf16 (runs as f16)	4.2 s	20×
gpu-q8	2.9 s	29×

All four gave the same transcript.

• Default backend: with none named, a server loads a Whisper at gpu-q8 where there is a GPU, and cpu-f32 where there is none. kvad transcribe --backend gpu-bf16 picks another.
• --local stays on the CPU. It runs in this process, and refuses a GPU backend with a message saying so.
• Checked against transformers. Run at f32, which is a test setting rather than a backend a server offers, the GPU model gives the same tokens as transformers on tiny, small and large. At two bytes or q8 a few tokens can differ: a comma, or a timestamp one step later. The CPU's own q8 differs at the same places.
• f16 rather than bf16: dense weights run as f16 even when gpu-bf16 is asked for. bf16 was eight times further from transformers and changed a word on large.

Transcribe in the web UI

A Transcribe page sits beside Images and Videos (#243).

• In: drop a recording or pick one, choose a Whisper (one not in memory is loaded first), and set a language, translation, timestamps or a prompt.
• As it runs: segments appear as each 30 seconds is heard, with a progress bar. Cancel keeps what was heard so far.
• Out: copy the text, or save it as .txt, .srt or .vtt. The files are byte-identical to what the API returns for the same formats.
• Chat: the Chat page still leaves speech models out.

What behaves differently

• On a machine with a GPU, a Whisper loaded without a backend named is now loaded at gpu-q8, not cpu-f32. Its transcripts can differ from cpu-f32's at the near ties described above. Name cpu-f32 to get the old behaviour, which is the one checked token for token against transformers.

Known

• Hallucinated text can show a stray �. It has been seen on nb-whisper-tiny after its first window. The model writes bytes, and in a window it is unsure of it can write bytes that are not UTF-8: a lone continuation byte after a whole character, or a segment that ends halfway through one. (An earlier version of these notes blamed a character split between two segments. Inside one segment is the usual case.) #245 fixes it on master by holding the decoder to valid UTF-8. The fix comes in the next release.

Install

curl -fsSL https://raw.githubusercontent.com/bisand/kvad/master/install.sh | sh

It asks before putting anything on your PATH or starting anything at
login, and if the server should start at login it asks what address to
listen on, offering 127.0.0.1:5823. Name one it cannot serve from — an
address something already holds, or a non-loopback one, which
kvad-serve refuses while no auth mode is configured — and it says why
and asks again, rather than installing a service that cannot start.

For a machine with nobody watching, --yes never opens a terminal and
answers every question the quiet way: binaries only, nothing added to
your shell and nothing started. Ask for the rest explicitly, --bind
included, since a run that cannot stop to ask would otherwise take 5823:

curl -fsSL https://raw.githubusercontent.com/bisand/kvad/master/install.sh | sh -s -- --yes --service --bind 127.0.0.1:8123

--prefix DIR, --version vX.Y.Z and --uninstall do what they look
like; --uninstall takes the binaries and the service and leaves your
models and conversations alone. sh install.sh --help lists the rest.

macOS ships kvad, kvad-serve, kvad-tui and kvad-gpu. Linux ships
kvad and a CPU-only kvad-serve; the other two link Metal.

Downloads can be checked against SHA256SUMS below.