MAFDET 0.5.0 — reading ahead in Open WebUI

MAFDET makes a local model read and think while you type. 0.5.0 brings the reading to Open WebUI. mafdet proxy puts MAFDET between Open WebUI and llama.cpp: the answer starts as soon as you press Enter, in the chat you already use.

Open WebUI, Qwen3-4B on an RTX 2060, 8 conversations each way	Enter → first word, mean	median	slowest
on llama-server with its default options	5.4 s	3.3 s	20.7 s
through mafdet proxy	0.16 s	0.14 s	0.4 s

And the thinking, experimental: with mafdet proxy --think-ahead, a Qwen3 with thinking on reasons about your message while you type, as in the MAFDET app. In Open WebUI, the first word of the answer came 7.6 → 4.3 s after Enter (mean of 22 turns, against reading ahead only), faster in 20 of 22. It changes the answer (it is not exact), so it is off unless you ask for it.

Without MAFDET, your message waits in line behind the title, tags and follow-up suggestions Open WebUI asks the model for, and Open WebUI's 30 built-in tools (~5.8k tokens) sometimes get read again. MAFDET gives those requests a slot of their own, keeps the conversation in the other, and reads your message while you type: on Enter, one token is left to read. It changes nothing in the prompt the model answers.

Get started

pipx install mafdet            # or: pipx upgrade mafdet · Windows with python.org's Python: pip install -U mafdet
mafdet doctor --install        # llama.cpp for this machine, if you have not installed it yet
mafdet models --download qwen3-4b
mafdet proxy --model qwen3-4b

Then, in Open WebUI, add an OpenAI connection to http://127.0.0.1:8077/v1, and install the small user script mafdet proxy points to, in Tampermonkey. Every step, with Docker and your own llama-server: Open WebUI + MAFDET in 5 minutes.

What's new

• mafdet proxy starts llama-server the way reading ahead needs it (two slots sharing one cache, nothing cleared behind its back, the model named after its catalog id) and the proxy in front of it, an OpenAI-compatible server. --url uses a llama-server you already run instead.
• The proxy (in mafdet/proxy/, standard library only):
  • Drafts. A front end posts what is being typed to /v1/drafts; the proxy puts it after the conversation it has seen, with the same template options and tools as the real request, and has the engine read it.
  • The request on Enter goes through unchanged, matched to its drafts. A follow-up suggestion clicked (no drafts) stays in the conversation.
  • Side requests (titles, tags, follow-ups) go to a second slot.
  • A probe finds what the engine supports; an audit checks the reuse it predicts against what the engine reports (/proxy/stats).
  • --exact for answers bit-identical to no proxy (without the gain), --cors for web pages.
• Thinking ahead in the proxy (--think-ahead, experimental): the MAFDET app's frame, run on the draft while you type, and the answer streamed back with its reasoning (reasoning_content) and Qwen3's tool calls (tool_calls).
• mafdet-drafts.js: drafts from any web front end in a few lines.
• Open WebUI user script (mafdet/proxy/clients/open-webui.user.js, Tampermonkey or Violentmonkey). Open WebUI stays untouched.
• The chat keeps more turns. With thinking on, past turns keep only their answers, as Qwen3's template shows them: after 6 turns the history is 1.3k tokens instead of 8k, and the context no longer fills up (Qwen3-4B, mean wait 2.45 → 1.94 s).
• Models split with the CPU (larger than the VRAM) get a q8_0 KV cache and, for Qwen3-14B, the IQ4_XS quantization: on a 6 GB budget, wait after Enter 34.0 → 21.1 s, 24/24 right.
• The repository moved to github.com/Mafdet-AI/mafdet.

The proxy started as a pull request from That80-20 (#4), on a design discussed in #2. Thank you.

Tested on

• Windows 11, RTX 2060 6 GB, llama.cpp b11747 (CUDA), Qwen3-4B Q4_K_M, Open WebUI 0.13.0 in Chrome with Tampermonkey.
• The proxy's tests (57) run without a GPU, against a fake engine.

Known limitations

• Thinking ahead is experimental: Qwen3 only, measured for speed but not yet for the answers' accuracy in the proxy (the app's own benchmark: 24/24 on Qwen3-4B).
• llama.cpp only. Ollama, LM Studio, vLLM and SGLang are on the roadmap (PROXY.md); help welcome.
• Not read ahead: files and knowledge attached in Open WebUI (its RAG template changes the prompt), and long pastes when Open WebUI's "Paste Large Text as File" is on.
• Measured on one card and one model, with thinking off for the answers. The answer's writing speed was not measured: it shares the GPU with the side requests.
• Answers can differ in wording, rarely in substance, from a run without the proxy: llama.cpp computes a prefix read ahead in other batches. --exact avoids it.

Full changelog: v0.4.1...v0.5.0

MAFDET 0.4.1 — installs on python.org's Python on a Mac

A fix for installing llama.cpp on a Mac with Python from python.org.

With Python from python.org, 0.4.0 stopped right after pip install with CERTIFICATE_VERIFY_FAILED. This happened when MAFDET installed llama.cpp and when it downloaded models. That Python has no certificates of its own until you run its Install Certificates script.

MAFDET now checks certificates against the system's own store, as pip and huggingface_hub already do:

• the keychain on macOS;
• the certificate store on Windows.

Nothing to run by hand anymore. SSL_CERT_FILE and SSL_CERT_DIR still take precedence when they are set, which is common on company networks. Linux is unchanged.

Update

pip3 install -U mafdet        # or: pipx upgrade mafdet

If you downloaded the source, the double-click launchers (start_mafdet.command, start_mafdet.bat) install the update by themselves the next time you start MAFDET.

Everything else is as in 0.4.0: any GGUF, searched and downloaded from Hugging Face inside the app, or files of your own.

Full changelog: v0.4.0...v0.4.1

MAFDET 0.4.0 — any GGUF, straight from Hugging Face

MAFDET now runs any GGUF model, not only the ones in its catalog. Search Hugging Face from inside the app and download a model in one click, or use a GGUF file you already have.

Get started

From PyPI (Python 3.10 or later). With Homebrew's Python on macOS, or on recent Ubuntu and Debian, use pipx: plain pip install stops there with externally-managed-environment.

brew install pipx        # Ubuntu/Debian: sudo apt install pipx
pipx ensurepath          # then open a new terminal
pipx install mafdet      # Windows with Python from python.org: pip install mafdet
mafdet

To update from 0.3.0: pipx upgrade mafdet, or pip install -U mafdet.

Windows / macOS, double click: download the source (Source code (zip) below), unzip it and double-click start_mafdet.bat (Windows) or start_mafdet.command (Mac).

What's new

• Search Hugging Face (Models view).
  • Finding a model. Search the GGUF repositories on the Hub, most downloaded first.
  • Before you download. Each file shows what it does on your machine: whether it fits on the GPU, how fast it writes, and how long the answer takes after Enter. A quantization is suggested (Q4_K_M when there is one).
  • Downloading. An interrupted download resumes where it stopped. A model split into several files downloads all of them.
  • Gated models (Llama, Gemma) need their license accepted on huggingface.co and hf auth login on the computer.
• Your own GGUF files.
  • Put a .gguf in ~/.mafdet/models, or pick one anywhere on disk with Add a GGUF file.
  • These files appear under Your models, together with the ones from the search. Their numbers come from the file's own header: layers, parameters, active experts. Measure works as for the catalog.
• Nothing else to download. The tokenizer and the chat template come from the GGUF itself, through llama-server. On Qwen3-4B they match the Hugging Face tokenizer exactly, and each call takes 0.17 ms.
• Thinking ahead.
  • A Qwen3 with thinking mode, including fine-tunes, reasons while you type.
  • Every other model reads your message while you type and writes the answer ahead at your pauses.
• From the terminal.
  • mafdet models lists your models too.
  • mafdet models --download owner/repo:file.gguf downloads any GGUF on Hugging Face.

The model has to use an architecture that the installed llama.cpp (b11747) supports. If it doesn't, the app shows the error from llama.cpp.

Also in this release

New measurements of IQ4_XS against Q4_K_M, in research/LOG.md:

Model	Card	IQ4_XS against Q4_K_M
Qwen3-14B	RTX 4070 SUPER (entirely on the GPU)	31% faster, same answers
Qwen3-14B	RTX 2060	11% faster
Qwen3-4B		about 4% faster on the 4070 SUPER, no gain on the 2060, and one answer worse out of 24

Full changelog: v0.3.0...v0.4.0

MAFDET 0.3.0 — now on AMD Radeon (benchmarks wanted)

MAFDET now runs on AMD Radeon cards, through llama.cpp's Vulkan build. It finds the card, installs the Vulkan build by itself and runs the models on the GPU. Thinking ahead works the same way as on CUDA.

We need someone with a Radeon to measure it. The Vulkan path is checked on an RTX 2060, but nobody has run MAFDET on a Radeon yet. If you have a dedicated Radeon (RX 5000 or later, 8 GB or more is ideal), about half an hour of it would give the first numbers. The steps are below, under Help wanted.

The Vulkan build on the RTX 2060 (Qwen3-4B Q4_K_M)	CUDA	Vulkan
Reads (prompt of 512 tokens)	2,141 tok/s	2,103 tok/s
Writes	87 tok/s	83 tok/s
Word problems typed in real time, 3/3 correct: mean wait after Enter, without → with thinking ahead		9.1 s → 4.4 s

Get started

Windows: download the source (Source code (zip) below), unzip it and double-click start_mafdet.bat. The first time it sets up a Python environment and installs MAFDET, in about a minute. It needs Python 3.10 or later.

macOS (Apple Silicon): the same, with start_mafdet.command.

From a terminal:

git clone https://github.com/TheLastOxygen/mafdet.git
cd mafdet
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e .
mafdet

The first start installs the right build of llama.cpp by itself:

• CUDA 12 or 13 for an NVIDIA card
• Vulkan for an AMD Radeon
• Metal on a Mac
• CPU otherwise

What's new

• AMD Radeon through Vulkan.
  • Finding the card. On Windows, MAFDET reads the cards Windows has running (DXGI); the registry would also list cards removed long ago. On Linux, it reads the amdgpu driver.
  • Dedicated cards only. Integrated Radeons (Ryzen APUs) keep the CPU build for now.
  • Only the dedicated card is used. On a PC that also has an integrated GPU, llama.cpp uses only the dedicated card. A model larger than its VRAM is split between that card and the CPU, never onto the integrated GPU.
  • Speed estimates. The model catalog knows the memory bandwidth of the RX 400 to RX 9070 cards, so it can estimate speeds before you download a model.
• mafdet doctor --install --build 12|13|vulkan|cpu picks the llama.cpp build. vulkan works on any card with a Vulkan driver, NVIDIA and Intel included. --cuda still works.
• Clearer messages without a GPU. "No NVIDIA card found" is now "No GPU found (NVIDIA, AMD Radeon or Apple Silicon)", in all four languages and in the terminal app.

Help wanted: benchmarks on an AMD Radeon

1. Install MAFDET and open it once. It should find your card, install llama.cpp Windows · Vulkan (or Linux · Vulkan) and download Qwen3-4B. mafdet doctor shows what it found.
2. Measure the model. In the app, go to Models and press Measure on Qwen3-4B.
3. Run the benchmark. From the MAFDET folder, with its environment active, run python research/thinking_ahead.py radeon_4b.json. It types 24 word problems in real time, with and without thinking ahead, in about 20 minutes. Leave the machine otherwise idle while it runs.
4. Send the results. Open an issue with the output of mafdet doctor, the numbers from Measure and radeon_4b.json.

If something does not work, an issue helps just as much. For example the card is not found, llama.cpp does not start, or the model runs on the CPU. Attach the output of mafdet doctor and the server*.log files next to llama-server, in ~/.mafdet/llama.cpp.

Tested on

• Windows 11, RTX 2060 6 GB: with CUDA and with the Vulkan build. i5-10400, 16 GB RAM, Python 3.12.
• macOS 26.2, MacBook Pro M2 Pro 16 GB: Python 3.12.
• AMD Radeon: not yet. On Windows, finding the card was checked only with the NVIDIA and Intel adapters of the test PC.
• Linux: has its llama.cpp builds and code paths in place, AMD included, but has not been tested yet.

Known limitations

• No measurement on a Radeon yet. Speeds in the catalog are estimates calibrated on an NVIDIA card with CUDA.
• On Windows, the VRAM in use on a Radeon is not known. It counts as free in the estimates. llama.cpp still splits the model by the free memory Vulkan reports.
• Thinking ahead works on Qwen3 only, and it is not exact: the model reasons on versions of the message that are not final yet. Reading ahead is exact.

The full measurements, and what did not work, are in the README's Results and in research/LOG.md.

MAFDET 0.2.0 — now on Apple Silicon Macs

MAFDET now runs on Apple Silicon Macs. The models run on the GPU through Metal, in the unified memory, and thinking ahead works there as it does on the PC.

Measured on a MacBook Pro M2 Pro (16 GB), word problems typed in real time	Without MAFDET	With MAFDET
Qwen3-4B, mean wait after Enter (24 problems, 24/24 correct both ways)	18.1 s	5.6 s
Qwen3-4B, median wait	16.7 s	4.1 s
Qwen3-14B, all on the GPU at 18 tok/s (6 problems, 6/6 correct both ways)	32.7 s	12.1 s

On the Mac the 14B fits entirely in unified memory: Metal may use 12 GB of the 16. It writes at 18 tok/s, against 4.5-5 on the RTX 2060, where part of it runs on the CPU. So the wait after Enter is 12 s instead of 47 s. On the 4B, MAFDET was faster in all 24 problems.

Get started

macOS (Apple Silicon): download the source (Source code (zip) below), unzip it and double-click start_mafdet.command. The first time it sets up a Python environment and installs MAFDET, in about a minute. It needs Python 3.10 or later, from python.org or brew install python@3.12. If the Finder says it cannot verify the file, right-click it, Open, then Open again; this is needed only the first time.

Windows: the same, with start_mafdet.bat.

From a terminal:

git clone https://github.com/TheLastOxygen/mafdet.git
cd mafdet
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e .
mafdet

The first start installs the right build of llama.cpp by itself: Metal on a Mac, CUDA 12 or 13 for an NVIDIA card, CPU otherwise.

What's new

• macOS on Apple Silicon.
  • MAFDET finds the Apple GPU through Metal and uses the memory Metal may take as its VRAM.
  • The model catalog estimates speeds on the Mac too, counting the unified memory once. On the M2 Pro the estimate for Qwen3-4B was 53 tok/s; the measured speed was 57.
  • On macOS, llama.cpp servers close with the app even when it is killed.
• An answer every time, after thinking ahead. Now and then Qwen3-4B wrote its answer among its thoughts and ended its turn without leaving the reasoning. The app then showed only thoughts and an empty answer. This happened once in 24 problems on the Mac, and once before on the RTX 2060. The app now closes the reasoning as Qwen3's template does, and the model writes the answer.
• Guess the ending (experimental). In Settings, Experimental, or with --branches 1. The model also guesses how your message ends and, on a slot of its own, reasons on the guess; if it guessed the exact message, that reasoning is ready on Enter. On Qwen3-4B the mean wait went from 4.17 s to 3.51 s, 48/48 correct both ways. The gain is not yet significant, and the app offers the branch only to models that fit in VRAM.
• The terminal app as an amber phosphor screen.
  • The start is now a boot sequence, with a log of what was really found on your machine.
  • View names decrypt into place.
  • Settings, Cinematic effects, turns the animations off.
• Setup you can follow. The terminal shows every download as it goes: each library, llama.cpp and the models, with size, a progress bar, speed and time left.
• mafdet doctor --install --cuda 12|13|cpu picks the llama.cpp build when the automatic choice is not the one you want.

Fixes

• First start: picking another quantization on the model card works again; it used to jump back to the recommended one.
• The window: a formula written over several lines ($$ … $$, \[ … \]) now shows as a formula instead of raw LaTeX.

Tested on

• macOS 26.2, MacBook Pro M2 Pro 16 GB, Python 3.12.
• Windows 11, RTX 2060 6 GB, i5-10400, 16 GB RAM, Python 3.12.
• Linux has its llama.cpp builds and code paths in place, but has not been tested yet.
• Intel Macs run on the CPU: llama.cpp's Intel build for macOS has no Metal.

Known limitations

• Tested on one Mac only. Numbers from other chips are welcome, especially Macs with 8 GB: run mafdet doctor, then Measure in the app.
• Long runs make a MacBook hot. In about 25 minutes of nonstop benchmark the M2 Pro reached thermal pressure Heavy for a few minutes, and the 4B wrote at 51 tok/s instead of 54.
• Thinking ahead works on Qwen3 only, and it is not exact: the model reasons on versions of the message that are not final yet. Reading ahead is exact.

The full measurements, and what did not work, are in the README's Results and in research/LOG.md.

MAFDET 0.1.0 — reads and thinks while you type

You type. It already knows. While you write a message, MAFDET does two things ahead of Enter.

• It reads. Your local LLM takes in your message as you write it, so when you press Enter the prefill is already done.
• It thinks. A reasoning model starts working on your message while you are still typing, and keeps that reasoning when you change the text. On Enter, it finishes from where it got to.

The thinking is the part that matters most. On a mid-range GPU, a reasoning model can spend ten seconds or more thinking before the first word of its answer, and much longer when the model does not fit in VRAM. MAFDET moves most of that thinking into the time you spend typing.

Measured on an RTX 2060 (6 GB), word problems typed in real time	Without MAFDET	With MAFDET
Qwen3-4B, mean wait after Enter (24 problems, 24/24 correct both ways)	12.2 s	3.2 s
Qwen3-14B, partly on the CPU (6 problems, 6/6 correct both ways)	108 s	47 s

On the 4B, about two thirds of the reasoning happens before Enter, and the reasoning is shorter overall: the material gathered while you type saves the model from starting over.

This is the first release, built for the hardware most people own.

Get started

Windows: download the source (Source code (zip) below), unzip it and double-click start_mafdet.bat. The first time it sets up a Python environment and installs MAFDET, in about a minute. It needs Python 3.10 or later.

From a terminal:

git clone https://github.com/TheLastOxygen/mafdet.git
cd mafdet
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e .
mafdet

The first start guides you through the rest. It looks at your machine, installs the right build of llama.cpp, and downloads the model that runs best on your hardware. No PyTorch, no manual downloads.

What's in it

• Thinking ahead on Qwen3, on by default. The model reasons in short chunks on the message as it is at that moment, and keeps its reasoning as notes when you edit. The notes start with a frame that keeps the model gathering facts and partial results, instead of remarking that the message is unfinished. With that frame, the share of off-topic notes fell from 86% to 17-26%.
• Reading ahead for every model. The KV cache follows your text, whatever you do: typos, edits in the middle, pasted documents. The cache always matches a computation from scratch, and mafdet judge checks it with 0 errors.
• The answer written ahead at your pauses, for every model. If you pause and then send the message unchanged, the answer is already on screen.
• The app, in two forms. A window (mafdet gui) in English, Italian, French or German, or the same app in the terminal (mafdet tui). As you type, a status line shows what the model is doing with your message, and every answer ends with how much was read and reasoned before Enter.
• A model catalog for your machine. 18 models in five families: Qwen3, Llama 3, Gemma 3, Mistral and Phi-4. Before you download a model, MAFDET tells you whether it fits in your VRAM, how fast it writes and how long the answer takes. On the reference PC the estimates are within 12% of the measured speeds, and Measure replaces them with real numbers in half a minute.
• Automatic setup. MAFDET installs llama.cpp in the build for your machine: CUDA 12 or 13 for an NVIDIA card, CPU otherwise. Downloads resume where they stopped, and llama.cpp servers close with the app.
• Measurement tools, for anyone checking the claims:
  • mafdet hunt simulates real typing and measures the wait after Enter;
  • mafdet race benchmarks prefill and decode on your hardware;
  • mafdet judge checks the incremental cache against a computation from scratch;
  • demo_live.py shows the cache following your text, step by step.

Tested on

• Windows 11, RTX 2060 6 GB, i5-10400, 16 GB RAM, Python 3.12.
• Linux has its llama.cpp builds and code paths in place, but has not been tested yet.
• macOS (Metal) and AMD GPUs are not supported yet.

Known limitations

• Thinking ahead works on Qwen3 only. It is the one family whose frame was measured. Llama 3, Gemma 3, Mistral and Phi-4 read while you type and write the answer ahead at your pauses.
• Thinking ahead is not exact. The model reasons on versions of the message that are not final yet, so its answer can differ from one started from scratch. That is why it was measured on answers that can be checked. Reading ahead is exact.
• The first start can be slow. The first time a new llama.cpp build or a new model family starts, it compiles GPU kernels, which can take up to a minute. After that it starts in seconds.
• The gatherer is experimental. With --gatherer, a small model gathers material while you type for a large one to read. On the RTX 2060 it does not beat the large model thinking ahead by itself; the README has the measurements.
• mafdet models needs about 115 terminal columns.

Open questions

Thinking ahead has the most room left. The README's Results section lists what we measured, including what did not work. We hope someone takes it further:

• thinking ahead on models larger than the VRAM that beats the large model thinking by itself;
• frames for other families, and for thinking models with their own formats (DeepSeek-R1, gpt-oss).