Skip to content

Releases: TheLastOxygen/mafdet

Release list

MAFDET 0.1.0 — reads and thinks while you type

MAFDET 0.1.0 — reads and thinks while you type Latest
Latest

Choose a tag to compare

View all tags
@TheLastOxygen TheLastOxygen released this 22 Oct 14:38
v0.1.0
312e40f

You type. It already knows. While you write a message, MAFDET does two things ahead of Enter.

  • It reads. Your local LLM takes in your message as you write it, so when you press Enter the prefill is already done.
  • It thinks. A reasoning model starts working on your message while you are still typing, and keeps that reasoning when you change the text. On Enter, it finishes from where it got to.

The thinking is the part that matters most. On a mid-range GPU, a reasoning model can spend ten seconds or more thinking before the first word of its answer, and much longer when the model does not fit in VRAM. MAFDET moves most of that thinking into the time you spend typing.

Measured on an RTX 2060 (6 GB), word problems typed in real time Without MAFDET With MAFDET
Qwen3-4B, mean wait after Enter (24 problems, 24/24 correct both ways) 12.2 s 3.2 s
Qwen3-14B, partly on the CPU (6 problems, 6/6 correct both ways) 108 s 47 s

On the 4B, about two thirds of the reasoning happens before Enter, and the reasoning is shorter overall: the material gathered while you type saves the model from starting over.

This is the first release, built for the hardware most people own.

Get started

Windows: download the source (Source code (zip) below), unzip it and double-click start_mafdet.bat. The first time it sets up a Python environment and installs MAFDET, in about a minute. It needs Python 3.10 or later.

From a terminal:

git clone https://github.com/TheLastOxygen/mafdet.git
cd mafdet
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e .
mafdet

The first start guides you through the rest. It looks at your machine, installs the right build of llama.cpp, and downloads the model that runs best on your hardware. No PyTorch, no manual downloads.

What's in it

  • Thinking ahead on Qwen3, on by default. The model reasons in short chunks on the message as it is at that moment, and keeps its reasoning as notes when you edit. The notes start with a frame that keeps the model gathering facts and partial results, instead of remarking that the message is unfinished. With that frame, the share of off-topic notes fell from 86% to 17-26%.
  • Reading ahead for every model. The KV cache follows your text, whatever you do: typos, edits in the middle, pasted documents. The cache always matches a computation from scratch, and mafdet judge checks it with 0 errors.
  • The answer written ahead at your pauses, for every model. If you pause and then send the message unchanged, the answer is already on screen.
  • The app, in two forms. A window (mafdet gui) in English, Italian, French or German, or the same app in the terminal (mafdet tui). As you type, a status line shows what the model is doing with your message, and every answer ends with how much was read and reasoned before Enter.
  • A model catalog for your machine. 18 models in five families: Qwen3, Llama 3, Gemma 3, Mistral and Phi-4. Before you download a model, MAFDET tells you whether it fits in your VRAM, how fast it writes and how long the answer takes. On the reference PC the estimates are within 12% of the measured speeds, and Measure replaces them with real numbers in half a minute.
  • Automatic setup. MAFDET installs llama.cpp in the build for your machine: CUDA 12 or 13 for an NVIDIA card, CPU otherwise. Downloads resume where they stopped, and llama.cpp servers close with the app.
  • Measurement tools, for anyone checking the claims:
    • mafdet hunt simulates real typing and measures the wait after Enter;
    • mafdet race benchmarks prefill and decode on your hardware;
    • mafdet judge checks the incremental cache against a computation from scratch;
    • demo_live.py shows the cache following your text, step by step.

Tested on

  • Windows 11, RTX 2060 6 GB, i5-10400, 16 GB RAM, Python 3.12.
  • Linux has its llama.cpp builds and code paths in place, but has not been tested yet.
  • macOS (Metal) and AMD GPUs are not supported yet.

Known limitations

  • Thinking ahead works on Qwen3 only. It is the one family whose frame was measured. Llama 3, Gemma 3, Mistral and Phi-4 read while you type and write the answer ahead at your pauses.
  • Thinking ahead is not exact. The model reasons on versions of the message that are not final yet, so its answer can differ from one started from scratch. That is why it was measured on answers that can be checked. Reading ahead is exact.
  • The first start can be slow. The first time a new llama.cpp build or a new model family starts, it compiles GPU kernels, which can take up to a minute. After that it starts in seconds.
  • The gatherer is experimental. With --gatherer, a small model gathers material while you type for a large one to read. On the RTX 2060 it does not beat the large model thinking ahead by itself; the README has the measurements.
  • mafdet models needs about 115 terminal columns.

Open questions

Thinking ahead has the most room left. The README's Results section lists what we measured, including what did not work. We hope someone takes it further:

  • thinking ahead on models larger than the VRAM that beats the large model thinking by itself;
  • frames for other families, and for thinking models with their own formats (DeepSeek-R1, gpt-oss).
Assets 2
Loading