Feature Requests: Custom local GGUF folders & Native on-board LLM for post-processing

[User20o2]

Hi Handy team,

First of all, I want to emphasize how much I love this tool. It is incredibly well-built, seamless, and genuinely one of the best local STT solutions out there. While running extensive local benchmarks for a post-production audit, I noted down two ideas that could make Handy the undisputed king of local STT.

Idea 1: Upstream Community GGUF Compatibility (Specifically Parakeet Status 4 Error) I saw in #1496 that you are already working on configurable storage directories. However, there seems to be a specific compatibility roadblock regarding the Parakeet models: I noticed Handy successfully uses Q5_K_M for models like Qwen3 and Granite. But when I tried to test the Parakeet TDT 0.6B v3 with a highly regarded community Q6_K quantization (e.g., from mudler) to squeeze out better WER than the provided 705 MB Q8_0 version, the backend crashed with a gguf load error (status 4).

Suggestion: Since the engine clearly handles K-quants perfectly fine for other models, could you please ensure the internal Parakeet/FastConformer implementation is aligned with upstream conversion standards? Right now, standard community GGUFs for Parakeet seem to be rejected due to a specific tensor/magic mismatch.

Idea 2: Native on-board LLMs for Experimental Post-Processing The experimental AI post-processing is a fantastic feature! However, it currently relies on commercial API keys or running a separate local backend server like Ollama.

Suggestion: It would be the ultimate game-changer if Handy could natively load and run small local text models (e.g. tiny LLMs in GGUF format via an embedded llama.cpp engine) directly within the app itself to handle punctuation and formatting. This would make Handy a 100% self-contained, zero-setup powerhouse, eliminating the need to spin up external Ollama servers entirely.

Please treat these purely as ideas for the future roadmap. You have built a truly fantastic piece of software!

[cjpais]

You must download the models from the handy-computer hf org, there is not necessarily support for people who generated .ggufs before transcribe.cpp's release. Many people made one off engines for one model, but that's not scalable for a project like handy. So we only support models which were converted using the converter in transcribe.cpp. I hope that the community will come to some standard for the asr models, and I hope that transcribe.cpp will be that home. If not transcribe.cpp will make sure we support the canonical form. But as of yet it does not exist. We do parse whisper.bin files correctly because whisper.cpp is canonical, but parakeet.cpp while used, is no where near as canonical in my view (though I can be wrong). If someone submits a PR to transcribe.cpp to parse them correctly I will pull it in if it is well done.

The Q8 version is basically as good as the FP32 version. See here: https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf. Q6 is slightly 'better' but this is within noise. In reality it will likely perform worse than q8.

We are working on llms locally, give it time. The user experience is the biggest thing and that means we must make it work on 99% of machines out of the box, and that's the hard part. This is squarely why it is an experimental feature today. Eventually we will remove support for third party providers entirely. It will be a local only app eventually. But we must have a model that is extraordinarily good that more or less runs on anything from the last 10 years. This is where the project is going. We know this is possible more or less, and I don't want to add more complexity until it's absolutely critical. Right now the power users get what they want but with some pain and we want to bring the power user functionality to everyone with 0 pain. The goal is to try to make the tool disappear as much as possible