Your docs have two readers now: people, and AI agents grabbing chunks to answer questions. Most docs are written for one and forget the other exists.
docfit tells you which one you're forgetting, and shows you what to fix.
You point it at a URL, a markdown file, or a whole folder, and it gives you a scorecard. Run it with --suggest and it hands you the rewrite too, ready to paste in.
Skip to: What it looks like · Why I built this · Install · The six scores · PostHog example · LLM setup · Batch mode · Save reports · CLI reference · Limitations
Before anything gets installed, here's a run against posthog.com/docs/product-analytics:
docfit report: https://posthog.com/docs/product-analytics
Overall: 78.0 / 100
Evaluator Score Bar Issues
chunkability 82.0 ████████████████░░░░ 2
retrievability 63.5 ████████████░░░░░░░░ 2
answerability 62.5 ████████████░░░░░░░░ 4
code_quality 100.0 ████████████████████ 0
ambiguity 75.0 ███████████████░░░░░ 11
task_orientation 85.0 █████████████████░░░ 0
Not a bad page. But the shape tells a story. Ambiguity found 11 issues, mostly words like funnels, stickiness, Self-driving used with no definition. If you ask an AI agent "what's stickiness in PostHog?" and it retrieves this chunk, it has to guess. Humans reading top to bottom can piece it together. Agents pulling one chunk at a time can't.
Run the same command with --suggest and docfit doesn't just complain about the jargon. It hands you a fix:
Before: Build trends, funnels, retention, paths, stickiness, and lifecycle insights on the events you already send to PostHog.
After: Build trends, funnels (a series of steps users take to complete a task), retention (the ability to keep users engaged over time), paths, stickiness (how often users return), and lifecycle insights on the events you already send to PostHog.
That's copy-and-paste ready. You didn't write it, docfit did.
Every LLM eval tool I looked at measures the output: did the model answer correctly, did it hallucinate, was it formatted right. That matters. But it's half the problem.
The other half is the input. A great RAG pipeline pointed at bad docs gives you bad answers, and there's nothing you can do about it in the pipeline. The fix has to happen upstream, in the docs themselves. And to fix docs at that layer, you first have to measure them.
That's what docfit does. It sits one step earlier than every other eval tool in the stack.
git clone https://github.com/bewanderer/docfit
cd docfit
python -m venv .venv
.\.venv\Scripts\Activate.ps1 # Windows
# source .venv/bin/activate # macOS/Linux
pip install -e ".[embeddings,dev]"You need one LLM key. The default is Groq because they have a free tier that's more than enough to try things out. Sign up at console.groq.com, then:
$env:GROQ_API_KEY = "gsk_your_key" # Windows
# export GROQ_API_KEY=gsk_your_key # macOS/LinuxNow try it on anything:
docfit report https://your-docs.example.com
docfit report ./README.md
docfit report ./docs/First run downloads a small embedding model (about 90MB, one-time). After that, everything's cached.
docfit runs six evaluators. Each one asks a different question about your docs.
Chunkability asks: can this doc even be split into sensible pieces? It looks at header structure, paragraph shape, whether sections have consistent nesting, whether chunks end mid-sentence. Fully deterministic, no LLM needed. If a doc has no headers, it can't be usefully retrieved, and humans can't skim it either.
Retrievability asks: if someone asks a real question, will search actually find the answer? docfit generates realistic questions from each chunk, embeds them, and checks whether the source chunk shows up at position 1, 2, or buried at 15. It's a direct measure of RAG quality.
Answerability asks: given only one chunk, can a reader actually answer a question about it, or would they have to invent context? This is the metric that catches "this page is great for humans reading top-to-bottom but useless as retrieval fodder."
Code quality asks: are your code samples complete, tagged with a language, and would they actually run? Humans copy-paste them. Agents parse them to decide what to call. docfit auto-recognises install commands (pip install, docker run, curl and 30+ others) and doesn't ding them for missing shebangs. Small thing, kills a whole class of false positives.
Ambiguity asks: are you using words the reader doesn't know? Undefined jargon, pronouns without antecedents, context you assumed. Humans figure it out from experience. Agents don't have experience. This is where the rewrite suggestions really shine.
Task orientation asks: is this doc structured around what users want to do, or around what things are? "How to install X" beats "The X architecture" every time, because users search for tasks. Agents match user intent to task language too.
Rough intuition from the pages I've tested:
- 90+ is a genuinely good page. Well structured, self-contained, task-shaped.
- 75 to 90 is solid docs with one or two dimensions worth improving.
- 60 to 75 works but has real gaps. Look at which dimension is dragging the number down.
- Under 60 usually means structural or content problems worth prioritising.
The single overall score is the least interesting number. The shape of the six is what tells you what to actually do. A page scoring 80 on chunkability and 50 on answerability tells you the content is well organised but too high level. A page at 50 chunkability and 90 answerability tells you rich content that's hard to retrieve. Different fixes.
I ran docfit against nine pages of posthog.com/docs while building this. Here's what it found.
| Page | Overall | Chunk | Retr | Answ | Code | Ambig | Task |
|---|---|---|---|---|---|---|---|
| /docs | 62.0 | 42.0 | 100.0 | 0.0 | 100.0 | 40.0 | 90.0 |
| /docs/product-analytics | 78.0 | 82.0 | 63.5 | 62.5 | 100.0 | 75.0 | 85.0 |
| /docs/session-replay | 77.9 | 82.0 | 68.8 | 68.8 | 100.0 | 80.0 | 67.5 |
| /docs/feature-flags | 78.3 | 82.0 | 72.9 | 62.5 | 100.0 | 70.0 | 82.5 |
| /docs/experiments | 63.1 | 53.5 | 100.0 | 0.0 | 100.0 | 80.0 | 45.0 |
| /docs/surveys | 78.3 | 82.0 | 47.9 | 75.0 | 100.0 | 80.0 | 85.0 |
| /docs/data-warehouse | 72.5 | 15.0 | 100.0 | 50.0 | 100.0 | 80.0 | 90.0 |
| /docs/max-ai | 71.4 | 57.0 | 84.1 | 10.9 | 100.0 | 78.8 | 97.5 |
| /docs/getting-started/install | 83.4 | 90.7 | 86.1 | 50.0 | 100.0 | 83.3 | 90.0 |
| Median | 77.9 | 82.0 | 84.1 | 50.0 | 100.0 | 80.0 | 85.0 |
A few things stood out.
The install guide scored highest overall (83.4). That's not surprising. Installation content wins because it's inherently task-oriented, well-structured, and concrete. When you have a good structural example, look for that shape elsewhere.
Answerability was the weakest dimension almost everywhere (median 50). Most pages describe what a feature IS at a high level but don't include the concrete step-by-step content that lets a single chunk answer a specific question. This is the biggest opportunity across the whole set.
The /docs landing page scored 62. That's the right number, not a bug. Landing pages are navigation grids, not narrative, and the tool correctly reflects that. If you're auditing your own docs and you see a landing page score high on this rubric, something's probably off.
The /docs/data-warehouse page got 15 on chunkability. It has almost no header structure, so it's a wall of text. Fixing headers would probably lift the score into the 70s without touching a single word of content.
One thing to flag honestly: three pages got 100 on retrievability. That happened because those pages produced very few chunks after filtering, so any semantic search trivially found the right one. Small chunk counts make MRR noisy. It's a real limitation of the eval, not a real strength of those pages.
docfit uses an LLM for the judgement work (answerability, ambiguity, task orientation, code completeness, question generation). Four backends are supported.
| Provider | Default model | Cost | Env var |
|---|---|---|---|
| Groq (default) | llama-3.3-70b-versatile | Free tier 100K/day | GROQ_API_KEY |
| OpenAI | gpt-4o-mini | Paid, cheap | OPENAI_API_KEY |
| Anthropic | claude-haiku-4-5 | Paid | ANTHROPIC_API_KEY |
| Ollama | llama3.2:3b | Free, local | OLLAMA_HOST |
Switch for the whole session with env vars:
export DOCFIT_LLM_PROVIDER=anthropic
export DOCFIT_LLM_MODEL=claude-haiku-4-5Or per run:
docfit report ./docs.md --provider ollama --model llama3.2:3bChunkability, the Python and JSON syntax checks, install-command detection, and all the embedding work run locally with zero API cost. Only the actual judgement calls hit the LLM.
If you have more than one doc, use batch:
docfit batch ./docs/ # print a summary table
docfit batch ./docs/ --recursive # walk subdirectories
docfit batch ./docs/ --only chunkability # skip the LLM, run local checks only
docfit batch ./docs/ --save-to markdown --name "sept-audit"That last command writes one full report per file into ./report/sept-audit/, plus a _summary.md with median, mean, best and worst pages, and per-evaluator stats across the folder. That summary is the artifact worth handing to a docs team.
Any run of report or batch can save:
docfit report ./doc.md --save-to markdown
docfit report ./doc.md --save-to html
docfit report ./doc.md --save-to all --name "my-audit"Markdown is portable and renders on GitHub. HTML is styled with proper CSS, dark-mode aware, works in any browser. JSON is for when you want to feed the results into something else.
Files land in ./report/. Custom names get cleaned up (no illegal characters) and are collision-safe: if my-audit.html already exists, the next save becomes my-audit-2.html. Nothing overwrites by accident.
If you use PostHog, docfit can send each run as a docfit_run event so you can track doc quality over time on a dashboard.
docfit report ./docs.md --posthog-key phc_your_project_keySet POSTHOG_API_KEY in your env to enable it globally.
docfit report <source> score a single URL, file, directory, or literal string
docfit batch <directory> score every .md file in a directory
docfit chunks <source> show how the doc would be chunked (useful for debugging)
docfit list-evaluators list all 6 evaluators
docfit list-providers list all 4 LLM backends
docfit version print the docfit version
Every command supports --help for its full options.
Being honest about the sharp edges.
English only. Every prompt is English, and non-English docs will run but the LLM judgement quality drops off.
Batch runs sequentially, one file at a time. A 20-page audit takes a while. Parallel execution is on the list.
Markdown files only for directory input. HTML, reStructuredText, and Jupyter notebooks are on the roadmap, not built yet.
--suggest roughly doubles token usage because every issue triggers a follow-up rewrite call. That's the tradeoff for actionable output, and it's opt-in for a reason.
Small local models (Ollama, Groq's 8B) sometimes return malformed JSON on the judge prompts. docfit handles this gracefully by skipping the affected chunk, but you'll see lower coverage compared to the 70B default.
On very short docs (under 5 chunks after filtering), retrievability becomes noisy. Not a bug, just a sample-size thing. Worth mentioning because you'll see it.
pip install -e ".[dev,embeddings]"
pytest # 106 tests
ruff check . # styleAll tests mock the LLM. You can run the whole suite offline without an API key.
MIT. See LICENSE.
docfit is v0.1. Feature-complete for what I set out to build, tested end to end on real docs, but the interface might still change. Issues and feedback are welcome at github.com/bewanderer/docfit/issues.