Citability Autopilot

An AI-search control room where the agent is the operator and the human holds the authority.

Live: https://citability-autopilot.vercel.app
Source: https://github.com/Citability/autopilot

Most agent products bolt a chat sidebar onto a dashboard built for a person. This is the other shape: a workspace whose primary operator is an agent calling WebMCP tools, where the human writes the mission, sets the boundaries, and approves the decisions that matter.

Every number on the screen comes from a real measurement run against a real site. Nothing here is invented, and the parts that cannot be known are labelled as unknown rather than filled in.

The idea in one paragraph

A human writes a mission. The agent reads structured state, investigates the evidence, separates what it observed from what it concluded, proposes an intervention, executes the part it is allowed to execute, stops at the part it is not, and reports what changed, what it refused, what the evidence does not support, and what is still unresolved.

The authority model

Four tiers. The kernel looks a tier up in a static catalog keyed by tool name, and by nothing the caller said.

Tier	Contract
read	Autonomous. Changes nothing, spends nothing, is not logged, because a read is not an event.
reversible	Autonomous, always logged, and must hand back an undo token.
experimental	Bounded. Refused without baseline evidence, an expected metric, and a rollback.
high_impact	Never runs. Calling it files an approval request and returns.

Two design decisions carry most of the weight:

authorityOf(catalog, toolName) does not accept the tool's arguments. Not "ignores them", cannot see them. There is no parameter through which agent confidence can argue for a lower tier. An unknown tool resolves to high_impact, so forgetting to classify a new tool fails safe rather than fails open.

Approval authorises an act, not a tier. decideApproval records a decision against one request and never mutates the catalog. Approve the same tool again and it stops again.

The kernel

Everything is enforced in one function, execute() in lib/kernel.ts, in about forty lines. Six rules, applied in order. The reason it lives in one place rather than in each tool: a rule every tool has to remember to apply is a rule a new tool will forget.

Anti-fabrication

The interesting constraint is not stopping the agent from acting. It is stopping the agent from claiming.

• report_mission_outcome diffs the changes it claims against the kernel-written log and refuses on mismatch. The agent cannot report a change the kernel did not perform.
• A refusal writes no log entry and applies no patch, so nothing that was declined can later be mistaken for work that happened.
• The mission report has a slot for claims the evidence does not support, and one for what is still unresolved. Both are populated in the demo run.

Instrument honesty, and why the agent gets refused

The workspace ships a real calibration record from 2026-08-23. Four engines were tested against two positive anchors plus one invented negative control:

Engine	Wikipedia	freeCodeCamp	Invented control
perplexity	pass	pass	correctly absent
gemini	pass	pass	correctly absent
openai	fail	fail	correctly absent
claude	fail	fail	correctly absent

openai and claude fail the positives while passing the negative control. That combination means the instrument is not hallucinating, it simply cannot support a claim on those engines right now.

This is why the demo does what it does. The highest-value opportunity in the data sits on openai, and the agent proposes it first. The kernel's handler refuses, because a remeasurement nobody can read is not an experiment. The agent then redirects to the smaller perplexity target it can actually measure. The adaptation is a consequence of the evidence model, not a scripted beat.

The evidence

A frozen snapshot of a dogfood run citability.dev made against its own site on 2026-07-23: 45 answers across 15 queries and 3 engines. 1 cited, 2 mentioned, 42 absent. A 2.2% citation rate.

The single cited receipt is the point of the whole project. rcpt_MRWW7F4RBC7050 is a real citation, and it is a citation in the wrong slot: the engine names two competing products as the answer and links citability.dev only as "a documented measurement methodology". Any dashboard that aggregates verdicts scores that as a win. It is not one.

Both receipts in this workspace exist in production and are hash-verifiable by anyone, including you, against a system whose source you cannot read. That is the useful property of a signed receipt chain: sha256 plus prevSha256, and an anonymously callable verify endpoint.

One more distinction the agent respects: an archived receipt stays readable regardless of later instrument drift, because the hash still checks. A new measurement is only as good as the instrument on the day it runs. So the agent may read an openai receipt, and may not propose remeasuring on openai.

A completed experiment that did not work

review_experiment returns a real FAQPage schema experiment with the verdict no_supported_uplift. Four measurements, and the honest reading of each:

• AI citation rate moved 45.0% to 49.0%, a 4.0 point delta against a measured noise floor of 6.7 points. Marked unreadable rather than positive.
• Google rich results: breadcrumbs only, both before and after.
• Crawl rate: no difference.
• Search impressions: raw numbers favour FAQ by roughly 6x, but the association reverses once page age is controlled for.

The noise floor is measured, not assumed. The same panel run twice in one day against an unchanged site returned 23.3% and 30.0%.

The demo

Press Replay the mission. It issues the exact tool sequence an agent makes, through the same kernel, with no model in the loop. It is labelled that way in the UI on purpose, so it cannot read as a staged screen recording. Point a WebMCP-capable agent at the page and it drives the identical surface.

The run ends with report_mission_outcome: 1 changed, 2 refused, 2 unsupported by the evidence, 3 unresolved.

apply_llms_txt_change is a genuinely writable and genuinely reversible path. The workspace serves /demo/llms.txt, and applying the change alters what that URL returns. Open it in a tab and reload.

Isolation

There is no shared database and no login. Every visitor gets their own workspace held in React state, and a reload starts it over. The one piece of state that leaves the browser is a single cookie carrying an enum (baseline or improved), which exists so a server route can serve the right version of the file.

Nothing you do here reaches production. The one exception is reading: verifying a receipt makes a real anonymous call to citability.dev, which is how you can tell the receipts are not props.

The workspace holds a snapshot of a site the author owns. It contains no private user data, and production authentication was not weakened to make judging possible.

WebMCP

Tools are registered spec-direct against navigator.modelContext.registerTool, with an AbortController signal for teardown. No polyfill and no shim. When the API is absent the page says so honestly in the status chip rather than faking a surface.

Eleven tools: seven read, one experimental, two reversible, one high_impact. Each description is prefixed with its authority tier, so an agent reading the catalog knows what it is allowed to do before it calls anything.

What dogfooding caught

Worth recording, because it is the thesis applied to this repo.

The kernel used to report handler-level refusals as ran. Two refusal channels exist: the kernel refuses on authority and prerequisites, and a handler refuses on domain facts the kernel cannot know, such as whether an engine is calibrated. Only the first reached the surface. So the trace showed a completed action, and the log recorded an event that never happened, which is exactly the failure this project is built against.

It survived ten passing tests because those tests asserted on the handler's own return value and never on the disposition field the UI renders. A green assertion only covers what it named. Fixed in lib/kernel.ts rule 5, and the test now asserts the disposition, the absent log entry, and the unchanged workspace.

Recording the demo caught the second one. inspect_evidence was refusing on both receipts with "could not reach the verifier". The verifier was fine: citability.dev/mcp answers that call with found: true, hashValid: true, but its preflight returns no Access-Control-Allow-Origin, so the browser threw the response away before the page saw it. The refusal text was correct, and it deliberately said "we did not get to look" rather than "the hash is bad", which is the distinction that stopped a missing header from rendering as evidence tampering. app/api/verify now makes the same call from the server, where CORS does not apply.

Both were found by watching the thing run rather than by reading it.

Run it

npm install --ignore-scripts
npm run dev

npm test        # kernel rules, 10 tests
npm run typecheck
npm run build

The tests are the specification. tests/kernel.test.ts asserts that an unclassified tool is high impact, that a high-impact tool does not run even when called with authority: "read", tier: "reversible", approved: true, confidence: 1, that an experiment without a baseline is refused, that an experiment on an uncalibrated engine is refused, that reads are not logged, that an undo token is single use, that a change cannot be reported unless it was logged, and that approving a call does not retroactively make the tool autonomous.

Licence

MIT. See LICENSE.