Issue: Speed

[Fish-Owl]

At the speed that a normal user can type, almost no suggestions are made.

This seems to be the make-or-break quality of all these similar apps. Other attempts like CoTabby and Ghosty suffer from this issue as well.

I am using an M5 Max MacBook Pro with 128GB of Unified Memory, and basically the highest specs available. Even with that, only the smallest model (Qwen 3.5 0.8B) gives me suggestions in time for me to use them.

I do not know what secret approach CoTypist uses that means I can use Gemma 4 E4B (a 6.2GB model) and still get suggestions faster than I can type (and faster than Qwen 3.5 0.8B in KeyType for that matter), but I know that is going to be the deciding factor for most users.

[johnbean393]

@Fish-Owl

I have the same specs, so you're in luck here!

Reducing latency (without sacrificing accuracy) is definitely one of my top priorities. Via optimization tricks like batched branch decode, I have already cut latency by ~80% compared to base llama.cpp.

That being said, I agree that there's no harm in reducing latency further and that this is a make-or-break quality!

[johnbean393]

@Fish-Owl

Given these specs, I assume you're a developer yourself?

If you could make a PR, that'd be great too!

[johnbean393]

@Fish-Owl

I have come up with a few ideas, and Codex has already implemented some of them. I'm happy to report that perceived latency has improved significantly! Excited to create a new release soon.

Also, I actually recommend LFM2.5-8B-A1B-Base for our specs. You can see more details here

[salmanuc]

speed issue on m4 air 24gb as well though 0.8 also take a while to kick in
previously i was using cotypist with qwen 3 4b which worked well overall

[johnbean393]

New branch promotion cache optimization incoming.

On a side note, I'm really liking gpt-image-2 diagrams illustrations for changes!

[johnbean393]

Added a new tab in the app's settings to visualize latency!

Personally, most of my completions are displayed in under 200 milliseconds, or 0.2 seconds, but I can see this being in the range of 400 milliseconds for a M5 MacBook Air.

Feel free to post screenshots for what you observe in this app, or even better, you can export and send me latency data by pressing "Export data...". Try to post your Mac's specs as well.