wiki2okf

Convert a Wikipedia article into a compact, local Open Knowledge Format (OKF) v0.2 bundle.

wiki2okf turns Wikipedia's readable content, categories, and internal links into inspectable Markdown files with structured YAML metadata. It works deterministically by default and can optionally use an LLM to produce a more concise, structured knowledge concept.

uvx wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing

alan-turing/
├── index.md
├── log.md
└── concepts/
    └── alan-turing.md

Highlights

• Small by default — concise mode keeps the introduction and main encyclopedic sections while removing references, navigation, media lists, and other noise.
• Deterministic — the standard pipeline requires no LLM, account, API key, or optional dependency.
• Agent-friendly — output is plain Markdown and YAML that remains easy to inspect, version, edit, and process.
• Traceable — generated concepts preserve the Wikipedia URL, revision, retrieval time, and generator version.
• Optionally enriched — provider-agnostic LLM enrichment adds grounded facts, typed relations, aliases, and useful questions.
• Safe rendering — models return validated structured data; wiki2okf, not the model, owns the final YAML and Markdown rendering.

Installation

wiki2okf requires Python 3.11 or newer.

Install it as a command-line tool:

uv tool install wiki2okf

Add it to a project:

uv add wiki2okf

Or use pip:

pip install wiki2okf

LLM enrichment is an optional extra:

uv tool install "wiki2okf[llm]"
# or
pip install "wiki2okf[llm]"

The base package does not install LiteLLM or provider SDKs.

Quick start

Convert one English or French Wikipedia article:

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing

The default concise mode:

• retains the introduction and main encyclopedic sections;
• excludes references, bibliographies, external links, portal lists, and navigation;
• caps the article body at 12,000 characters;
• adds an explicit notice when content is truncated.

Use full mode to retain more meaningful source sections:

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --content-mode full \
  --output ./alan-turing-full

Both modes still remove empty sections and navigation noise.

How it works

Wikipedia URL
    │
    ▼
MediaWiki API
    │
    ▼
cleaned WikipediaArticle
    │
    ├── deterministic rendering
    │
    └── optional validated LLM enrichment
             │
             ▼
       deterministic rendering
    │
    ▼
local OKF bundle

The output always passes through the same internal article model and deterministic renderer. LLM responses are validated with Pydantic and never write frontmatter or final Markdown directly.

Example output

The generated concept is ordinary Markdown with YAML frontmatter:

---
type: Wikipedia Article
title: Alan Turing
description: English mathematician and computer scientist (1912–1954).
resource: https://en.wikipedia.org/wiki/Alan_Turing
tags:
  - Alan Turing
  - British computer scientists
generated:
  by: wiki2okf/0.2.1
  at: 2026-07-25T10:30:00Z
status: stable
sources:
  - id: wikipedia-page
    resource: https://en.wikipedia.org/wiki/Alan_Turing
    title: Alan Turing — Wikipedia
    revision_id: 1300000000
---

# Alan Turing

## Early life and education

Turing was born in Maida Vale, London, and was educated at Sherborne School.

## Related concepts

- [Turing machine](https://en.wikipedia.org/wiki/Turing_machine)
- [Cryptanalysis](https://en.wikipedia.org/wiki/Cryptanalysis)

[^wikipedia-page]: Wikipedia article retrieved automatically.

index.md provides bundle navigation, while log.md records the import.

Optional LLM enrichment

Enrichment converts the cleaned source into a shorter, structured concept while keeping the original provenance:

export GEMINI_API_KEY="your-key"

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --output ./alan-turing

The pipeline uses two passes:

1. Each section is assessed independently for relevance and summarized into grounded key points.
2. Retained points are combined into a concept type, concise description, key facts, typed relations, aliases, related concepts, and useful questions.

Every fact and relation must reference its supporting Wikipedia section. Unsupported section references, malformed responses, and schema violations are rejected.

Supported providers

Model routing is handled by LiteLLM. Use any compatible provider/model identifier and its standard environment variable.

Provider	Model identifier example	Authentication
Google AI Studio	gemini/gemini-3.1-flash-lite	GEMINI_API_KEY
OpenAI	openai/<model>	OPENAI_API_KEY
Anthropic	anthropic/<model>	ANTHROPIC_API_KEY
Ollama	ollama_chat/qwen3.5:4b	None

Local Ollama example:

ollama pull qwen3.5:4b

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "ollama_chat/qwen3.5:4b" \
  --fallback fail \
  --output ./alan-turing

Small local models may struggle with strict schemas and source grounding. --fallback fail is useful when evaluating a model because it makes those failures visible.

Caching and fallback

Validated enrichment responses are cached using the article revision, model, prompt version, schema version, pipeline stage, temperature, and content hash:

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --cache-dir ~/.cache/wiki2okf

The cache never contains raw prompts or raw provider responses.

If enrichment fails, the default behavior is to print a warning and produce the deterministic bundle. Use --fallback fail to stop instead:

wiki2okf convert URL \
  --enrich \
  --model PROVIDER/MODEL \
  --fallback fail

Use --keep-raw to append the cleaned Wikipedia sections after the enriched concept.

CLI

wiki2okf convert URL [OPTIONS]

Options:
  -o, --output PATH                 Output directory
      --content-mode concise|full   Content selection (default: concise)
      --enrich                      Enable structured LLM enrichment
      --model PROVIDER/MODEL        LiteLLM model identifier
      --temperature FLOAT           Sampling temperature (default: 0)
      --fallback deterministic|fail Enrichment failure behavior
      --keep-raw                    Keep cleaned source after enrichment
      --cache-dir PATH              Validated response cache directory

Run wiki2okf convert --help for the authoritative command reference.

Scope

wiki2okf currently supports:

• one French or English Wikipedia article at a time;
• MediaWiki action=parse responses;
• introductions and readable h2/h3 sections;
• categories and up to 20 internal article links;
• concise and full content modes;
• deterministic OKF Markdown and YAML;
• optional two-pass LLM enrichment with validation, caching, and fallback.

It intentionally does not include recursive crawling, Wikidata, structured infoboxes, embeddings, a vector database, graph visualization, a web interface, or bulk Wikipedia dump processing.

Development

git clone https://github.com/LucasMarchnd/wiki2okf.git
cd wiki2okf

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,llm]"

ruff check .
pytest
python -m build

Tests use recorded MediaWiki fixtures and fake LLM implementations. They never call model provider APIs.

Roadmap

v0.3 — Multi-article knowledge graphs

• accept multiple explicit Wikipedia URLs or a text file of seed URLs;
• generate one local OKF concept per resolved Wikipedia page;
• resolve relation targets deterministically against MediaWiki links;
• replace external Wikipedia links with bundle-local concept links;
• preserve typed, source-attributed relations as a wiki2okf extension;
• support bounded depth-one expansion with --depth and --max-pages;
• deduplicate redirects and repeated pages using stable MediaWiki identities;
• generate a bundle-wide index and update log.

Planned usage:

wiki2okf build \
  --input people.txt \
  --depth 1 \
  --max-pages 30 \
  --output ./computing-pioneers

v0.4 — Topic-driven graph discovery

Build a complete, bounded knowledge graph from a topic instead of requiring the user to provide every seed URL:

wiki2okf graph \
  --topic "Pioneers of computer science" \
  --language en \
  --depth 2 \
  --links-per-page 5 \
  --max-pages 50 \
  --output ./computing-pioneers

The discovery pipeline will:

• find relevant seed articles through MediaWiki search;
• rank candidate links against the requested topic;
• limit both branching (--links-per-page) and total graph size (--max-pages);
• stay inside the selected Wikipedia language;
• avoid navigation, list, category, portal, and unrelated hub pages;
• support deterministic discovery without an LLM and optional enriched ranking with --enrich;
• cache page revisions and discovery decisions for reproducible reruns;
• produce the same local Markdown links and grounded relations as explicit multi-URL bundles.

Later

• Wikidata identifiers and structured infoboxes;
• incremental updates based on Wikipedia revision IDs;
• official OKF conformance validation;
• graph visualization and bulk dump processing;
• stable Python API and source plugins.

Attribution

Generated bundles preserve their Wikipedia source URL and revision. Wikipedia content remains subject to the licensing and attribution requirements stated by Wikimedia and on each source article.

wiki2okf is an independent project. It is not affiliated with or endorsed by Google, Google Cloud, the Wikimedia Foundation, Wikipedia, or any LLM provider.

License

Released under the MIT License.