The open claim-history layer for public knowledge.
Refract reveals how claims change across public revision histories — where every claim came from, what supported it, what challenged it, when it stabilized, and what context altered its meaning.
Refract is the open claim-history layer for public knowledge — it ingests revision histories from Wikipedia, Fandom, and any MediaWiki instance, extracts a deterministic event stream of every claim, source, section, and dispute, and makes that stream queryable and reproducible. No model. No interpretation. Byte-for-byte identical on every run.
Built and maintained by NextConsensus. Refract is domain-neutral infrastructure for observing how public knowledge changes. Repository boundary.
Machines do not just need more retrieved text. They need provenance, instability, disagreement, and temporal change — six things that a current snapshot cannot provide:
- Where it appeared — when a claim first entered the public record
- How it changed — every addition, removal, reintroduction, and in-place modification
- What was tagged — policy templates, dispute signals, maintenance markers
- What was reverted — every revert, edit cluster, concentrated contestation
- What moved — section reorganization, lead promotion, category shifts, page moves
- What was discussed — correlated talk page activity, thread lifecycle, activity spikes
Refract makes that knowledge legible to machines by decomposing every statement into its history. That is more durable than search, monitoring, or summarization.
Given a MediaWiki page, the engine produces a structured event stream with 26 deterministic event types:
| Category | What Refract captures |
|---|---|
| Appearance | sentence_first_seen, sentence_removed, sentence_modified, sentence_reintroduced |
| Citations | citation_added, citation_removed, citation_replaced |
| Templates | template_added, template_removed, template_parameter_changed |
| Sections | section_reorganized, lead_promotion, lead_demotion |
| Reverts | revert_detected, edit_cluster_detected |
| Links & categories | wikilink_added, wikilink_removed, category_added, category_removed |
| Page metadata | page_moved, protection_changed |
| Talk page | talk_page_correlated, talk_thread_opened, talk_thread_archived, talk_reply_added, talk_activity_spike |
- Investigative journalists — trace how a claim about a public figure evolved across revision history: when it was added, who softened it, when sources appeared or disappeared
- Wikipedia editors — audit how policy templates (NPOV, BLP, due-weight) correlate with content changes over time
- Data scientists & researchers — deterministic features for edit-quality models, sourcing-behavior studies, content-drift measurement
- OSINT analysts — structured event streams from public editorial history, reproducible on request
- Fan wiki communities — canon disputes, headcanon drift, content-fork detection across MediaWiki instances
- AI/ML teams — training data curation (include stable, well-sourced claims; exclude contested or source-fragile ones), provenance-aware RAG, temporal leakage detection in training corpora
- Regulatory & compliance monitors — track changes to drug safety pages, guideline entries, and policy language for early signals of institutional shifts
- Knowledge graph engineers — extract entity and relationship changes across revision history for evolving ontologies
- Publishers & platform trust teams — monitor how claims spread, get cited, and stabilize across the public record
Each consumer brings their own interpretation layer on top of Refract's deterministic event stream:
| Consumer | They build | How they use Refract |
|---|---|---|
| Healthcare intelligence | Sourced briefs answering "does this claim still hold up?" | Feed structured events into a 4-lane measurement pipeline (clinical truth, ratification, economic stake, feasibility). Each event carries FactProvenance with the exact thresholds used. |
| AI training data curation | Training datasets filtered by claim stability | Score each claim by revert count, citation churn, talk page correlation, and template dispute history from the event stream. Include only claims above a stability threshold. |
| Provenance-aware RAG | Retrieval that weights results by claim stability | Enrich each retrieved chunk with its claim history — stable, recently changed, source-fragile, contested. The RAG system uses the signal to filter or demote low-confidence results. |
| Regulatory monitoring | Early-warning dashboards for policy changes | Run refract cron on drug pages, guideline entries, and regulatory topics. When new events fire (citation removal, template dispute, section reorganization), alert the monitoring team with the structured diff. |
| Competitive intelligence | Cross-jurisdiction claim divergence maps | Use refract diff to compare the same topic across wikis (English vs German Wikipedia, Fandom vs independent wiki). Track how framing differs and when it diverged. |
| Fact-checking | Claim provenance timelines | Given a claim text, query its lifecycle across the event stream — first appearance, source additions, revert history, talk page correlation, stabilization time. Return a verifiable timeline. |
| Academic research | Large-scale knowledge dynamics studies | Export ObservationReport with Merkle-verifiable claim histories. Run cohort analyses on claim stability across topics, time periods, and editorial environments. |
| Journalism forensics | Edit pattern analysis for public figures | Track how a specific claim about a person or topic evolved. Look for coordinated editing, source softening, or removal without replacement. |
| Fan wiki canon tracking | Canon divergence detection across competing wikis | Compare the same fictional universe's page across Fandom and independent wikis. Detect when one wiki retcons content while the other doesn't — and by how much. |
| Knowledge graph engineering | Evolving ontologies from category and link changes | Use refract analyze --depth forensic to capture category_added/removed and wikilink_added/removed events. Build an entity graph that evolves with the public record. |
The common architecture: Refract extracts the mechanical facts. The downstream system interprets what those facts mean for its domain. No interpretation enters Refract's pipeline; no consumer re-extracts from raw revision history.
Refract pairs naturally with modern tools. The event stream is standard NDJSON — anything that reads JSON or speaks HTTP can consume it.
| Category | Technology | How they fit |
|---|---|---|
| Vector databases | Pinecone, Weaviate, pgvector | Store claim embeddings alongside stability metadata. Query: "find claims like X that are stable and well-sourced." |
| RAG frameworks | LangChain, LlamaIndex | Use Refract's stability signals as retrieval filters or reranking features. Attach claim provenance to each retrieved chunk. |
| AI coding agents | Claude Code, Cline, Codex CLI, OpenClaw | Agents connect via Refract's built-in MCP server (refract mcp) to read claim history and cite provenance. |
| MCP (Model Context Protocol) | Any MCP client | refract mcp is a native MCP server with tools for analyze, claim, export, cron, and classify. |
| Data query | DuckDB, ClickHouse | Query NDJSON output with SQL: SELECT event_type, count(*) FROM 'events.jsonl' GROUP BY event_type; |
| Streaming | Kafka, Redpanda, Cloudflare Queues | Each EvidenceEvent is a message keyed by claimId for real-time monitoring. |
| Visualization | Observable Framework, Mermaid, D3 | refract visualize --format mermaid produces Mermaid diagrams. Observable has a @refract-org/observable data loader. |
| Model serving | OpenAI API, DeepSeek, Ollama, vLLM, Workers AI | Any OpenAI-compatible endpoint plugs into refract classify. Workers AI runs at the edge. |
| Local inference | WebGPU, MLX, llama.cpp | Run detection models on-device — no API key needed. Refract defaults are mechanical; any boundary can use a local model via Ollama or MCP sampling. |
| Notebooks | Jupyter, Marimo, Observable | Load events into a DataFrame: pd.read_json("events.jsonl", lines=True). Marimo's reactive runtime is ideal for live event stream analysis. |
| Serverless | Cloudflare Workers, D1, R2 | Run refract via npx in a Worker. Store events in D1, export to R2, queue re-observations. Entirely edge-deployable. |
# 1. Analyze a page (zero install, no config needed)
npx @refract-org/cli analyze "Bitcoin" --depth brief
# 2. View results in the web UI
refract explore "Bitcoin"
# 3. Export as an ObservationReport with claim lifecycle
refract analyze "Bitcoin" --report > bitcoin-report.json
# 4. Save as a signed evidence bundle
refract export "Bitcoin" --bundle > bitcoin-bundle.jsonWhat you're seeing: These are observed changes — deterministic facts extracted from revision history. Refract reports what changed, not whether a claim is true or false.
What you'll see:
Analysis of "Bitcoin" at depth brief found 330 events across 20 revisions.
[2009-03-08] wikilink_added (rev 275832581→275832690)
Section: body
target: cryptography
[2009-12-10] citation_added (rev 308164432→308164529)
Section: (lead)
ref: sourceforge.net/projects/bitcoin/
[2009-12-12] template_added (rev 308164529→308180771)
Section: body
template: primarysources
Full output (330 events): docs/example-output.md.
| Method | Command |
|---|---|
| Bun (if installed) | bunx @refract-org/cli analyze "Bitcoin" |
| Docker (prebuilt) | docker run ghcr.io/refract-org/cli analyze "Bitcoin" |
| Local install | bun add @refract-org/cli && refract analyze "Bitcoin" (or wikihistory) |
| Build from source | git clone https://github.com/refract-org/refract && cd refract && bun install && bun run build |
bun add @refract-org/evidence-graph @refract-org/analyzersimport type { EvidenceEvent, Revision } from "@refract-org/evidence-graph";
import { sectionDiffer, citationTracker } from "@refract-org/analyzers";| Package | npm | Description |
|---|---|---|
@refract-org/evidence-graph |
 |
Core types, schemas, BYO-inference boundaries |
@refract-org/ingestion |
 |
Wikimedia API adapters — fetching, diffing, rate limits |
@refract-org/analyzers |
 |
Deterministic analyzers — sections, citations, reverts, templates |
@refract-org/cli |
 |
CLI tool — refract / wikihistory commands, classify inference |
@refract-org/persistence |
— | Local SQLite persistence (bun:sqlite, not published) |
@refract-org/eval |
 |
Evaluation harness — ground truth validation and benchmarks |
@refract-org/observable |
— | Observable Framework data loader (not published) |
Refract tracks claim provenance — structured evidence linking a claim's lifecycle to specific revisions, sources, and policy signals. It complements existing tools:
| Tool | What it does | What Refract adds |
|---|---|---|
| WikiWho | Sentence-level authorship (who wrote which token) | Sentence lifecycle: when a sentence first appeared, was removed, rewrote, or was reintroduced |
| ORES | ML edit quality scores (likely damaging, good-faith) | Deterministic edit classification + policy-coded signals |
| XTools | Edit stats, page history summaries, top editors | Structured event stream: section changes, citation turnover, template diffs, page moves, category shifts |
| PetScan | Category intersection queries across pages | Category evolution per-page over time |
The engine follows a two-knowledge-split:
- Deterministic: Wikipedia API ingestion, diff computation, section extraction, citation tracking, template classification, revert detection — byte-reproducible, no model involved.
- Outcome labels: Independently sourced ground truth (talk page consensus, page protection events) — never redefined by the pipeline.
Every analyzer threshold is a typed function signature where a model can replace the default heuristic. The defaults work offline with no configuration required:
| Boundary | Default (mechanical) | Plug in a model to decide |
|---|---|---|
| Sentence similarity | Word-overlap ratio (0.8) | "Are these two sentences the same claim?" |
| Revert detection | 6 regex patterns | "Is this edit comment a revert?" |
| Template classification | Name-to-type lookup | "What policy signal does this template represent?" |
| Edit cluster detection | Time window + min size | "Are these edits semantically related?" |
| Heuristic classification | Size thresholds + comment patterns | "What kind of edit is this?" |
Pass overrides via --config file or inline CLI flags (--similarity, --spike-factor, --cluster-window, etc.). The effective parameters are recorded in each event's FactProvenance.parameters when non-default values are used.
# Domain-tuned: Fandom wikis have tighter edit clusters
refract analyze "Darth_Vader" --api https://starwars.fandom.com/api.php --cluster-window 30 --similarity 0.85Refract connects to any MediaWiki instance — corporate wikis, institutional
knowledge bases, private fan wikis. Use the --api flag with the wiki's
api.php URL.
| Method | CLI flags | Description |
|---|---|---|
| Bearer token | --api-key <token> |
Sends Authorization: Bearer <token> with every request |
| Basic auth | --api-user <user> --api-password <pass> |
Sends HTTP basic auth credentials |
| OAuth2 | OAUTH_CLIENT_ID + OAUTH_CLIENT_SECRET env vars |
Sends X-OAuth-Client-Id and X-OAuth-Client-Secret headers |
All three methods work with every command:
# Bearer token
refract analyze "Page" --api https://corp-wiki.example.com/w/api.php --api-key "sk-..."
# Basic auth
refract analyze "Page" --api https://corp-wiki.example.com/w/api.php --api-user "admin" --api-password "..."
# OAuth2 (via env vars)
OAUTH_CLIENT_ID="..." OAUTH_CLIENT_SECRET="..." \
refract analyze "Page" --api https://corp-wiki.example.com/w/api.phpCredentials are never logged or exposed in error messages.
A Docker Compose setup is available for testing against a local MediaWiki instance with auth:
cd docker
docker compose up -d
DOCKER_TESTS=true bun run testThis starts MediaWiki at http://localhost:8080 and an nginx proxy with basic
auth at http://localhost:8081. The auth integration tests validate bearer
token, basic auth, and OAuth2 paths.
Refract works on any public MediaWiki instance — Fandom.com, independent fan wikis, private wikis. Wikipedia's editorial norms suppress the most interesting dynamics; fandom wikis don't.
| Dynamic | What Refract captures |
|---|---|
| Canon disputes | category_removed: Canon characters → category_added: Legends characters after the 2014 Disney acquisition |
| Headcanon drift | "Vader turned because of fear of loss" vs "pride and ambition" — reversibly edited, both cite the same films |
| Warring wikis | Cross-wiki diff detects a Game of Thrones Fandom wiki fork vs parallel evolution on an independent ASOIAF wiki |
| Decade-spanning consensus | 2008 talk page consensus about what's canon, overturned in 2023 — L3 outcome labels with temporal validity windows |
If the engine handles fandom, it handles anything.
| Category | Why |
|---|---|
| Truth detector | Reports what changed, not whether the change is accurate |
| Model interpreter | No LLM in the pipeline — interpretation lives downstream in consumers |
| Editor quality judge | No scoring, ranking, or profiling of individual editors |
| Prediction engine | No forecasting, no sentiment analysis, no trend extrapolation |
| Live monitor | Polling-based, not real-time — use cron mode for scheduled observation |
| Healthcare scorer | Domain-agnostic by design — no clinical, regulatory, or payer logic |
AGPL-3.0. See LICENSE.
If you modify this software and deploy it as a network service, you must release your modifications.
Commercial use: NextConsensus offers commercial licenses for proprietary integration without AGPL obligations. See nextconsensus.com.
- Contributing — how to get started
- Good first tasks — ready-to-pick-up work items
- Discussions — questions, ideas
- Code of Conduct
- Security
- Changelog
- Cite this software
These repos extend the core engine:
| Repo | Purpose |
|---|---|
| refract-docs | Public documentation site (quickstart, CLI, SDK, tutorials) |
| refract-labs | Experimental probes applying the engine to adjacent verticals |
| refract-ui | Standalone visualization — load JSONL, render timelines, diffs, citations |
| refract-demo-data | Safe, fictional datasets for the eval harness (no real PII or medical data) |
| refract-py | Python SDK — typed dataclasses and pandas integration for ML workflows |