A two-pass, massively-parallel LLM pipeline that ingests a book series and generates a complete, interactive, cross-linked wiki from it — one API call per page, then one API call per article.
Live example: wiki.xavion.dev — a wiki generated by this pipeline from a public-domain text, including the spoiler scoping. The same output is in examples/ if you would rather read the raw artifacts.
No book data and no generated output ships with this repo. books_*/ and outputs_*/ are gitignored — the input novels are copyrighted, and so is most of the text embedded in a generated wiki (verbatim descriptions, quotes, page-cited facts). If you notice those entries in .gitignore, that is why. What does ship is examples/: a real run of this pipeline against a public-domain Jane Austen excerpt, so you can see actual output without spending a cent or supplying a key.
| 2,833 pages → 813 cross-linked articles | Spy School, 13 books |
| 1,129 pages → 393 cross-linked articles, ~$41 API spend | Arc of a Scythe, 3 books |
| One API call per page (Pass 1) + one API call per article (Pass 2) | fully parallel, resumable at every stage |
A generated article — infobox, cited sections, and [[cross-links]], all synthesized from raw extracted facts (this one from the examples/ public-domain run):
Every fact the pipeline extracts carries a (book, page) citation — not as metadata bolted on afterward, but as the atomic unit Pass 1 produces. That turns out to unlock something most wikis (Fandom included) don't do well: hiding spoilers past the point you've actually read.
The generated wiki.html ships a spoiler selector. Pick "read through book 2," and:
let SPOILER_MAX = Infinity; // 1-based index of the last "safe" book
function citVisible(cits) {
if (SPOILER_MAX === Infinity || !cits || !cits.length) return true;
const i = bookIdx(cits[0].book);
return i === 0 || i <= SPOILER_MAX;
}
function spoilerHidesSection(heading) {
if (SPOILER_MAX === Infinity || !heading || heading.indexOf('In ') !== 0) return false;
const i = bookIdx(heading.slice(3));
return i > 0 && i > SPOILER_MAX;
}citVisible gates individual facts; spoilerHidesSection hides an entire per-book section (headings are generated per-book, like "In The Toll") once it's past the cutoff, not just the facts inside it. Both, plus infobox fields, relationships, and quotes, filter live against your chosen cutoff — client-side, no regeneration needed. A character's article just quietly reflects less of their arc if you haven't gotten there yet. This isn't a pipeline afterthought; it's the reason the citation discipline in Pass 1 exists in the first place. It's also a real, standalone product idea — spoiler-safe companion wikis are a gap in how fan wikis work today.
books/*.pdf|epub|txt
│
▼
1. LOAD & PAGINATE PDFs keep their real page numbers; EPUB/TXT are split into
│ ~450-word pseudo-pages along paragraph boundaries.
▼
2. PASS 1 — EXTRACT One AI call PER PAGE, all pages in parallel. Each call also
│ sees the previous + next page, but strictly as CONTEXT — the
│ prompt explains why (pronoun resolution, sentences cut across
│ a page boundary) and states plainly that extracting a fact
│ from a neighbor page duplicates it, since that page gets its
│ own identical call. Pulls entities, atomic facts, verbatim
│ descriptions, dialogue, and relationships.
▼
3. AGGREGATE & MERGE Facts are grouped per entity; rule-based alias merging plus
│ an AI resolution pass folds nicknames, titles, and spelling
│ variants ("Harry" / "Harry Potter") into one entity.
▼
4. PASS 2 — SYNTHESIZE One AI call PER ARTICLE, all articles in parallel. Turns each
│ entity's pile of cited facts into a structured article:
│ infobox, sections, relationships, quotes, appearances.
│ Oversized prompts auto-escalate to a pro-tier model; entities
│ whose facts exceed 120K characters (long-running protagonists
│ in big series) get per-book map-reduce instead of one giant call.
▼
5. BUILD THE WIKI wiki.html — self-contained, searchable, spoiler-scoped
+ wiki_data.json — the clean data contract (see below)
A few things that came out of actually running this at book-series scale rather than on a toy example:
- Prompt design that explains itself. The Pass 1 prompt doesn't just say "don't extract from context pages" — it tells the model why: every page in the book gets an identical extraction call, so a fact pulled from a neighboring page will be pulled twice and corrupt the knowledge base with duplicates. Naming the failure mode measurably reduces it.
- Automatic model escalation + map-reduce. Synthesis prompts over 60K characters escalate from the default flash-tier model to a pro-tier model. Entities whose raw fact block exceeds 120K characters and span multiple books skip single-call synthesis entirely: one call per book distills a digest, then a reduce call combines them — so adding book 4 to a series only costs one new digest call, not a full re-synthesis of the protagonist's article.
- A rate limiter that reads its own rate limits. On a 429, it parses the API's own suggested
retryDelayout of the error body and waits exactly that long (plus a small buffer), rather than guessing. No parseable delay falls back to capped exponential backoff, with a longer base for quota/rate-limit errors than for generic transient failures. - JSON repair, not just retries. LLM JSON output gets cleaned for trailing commas and stray control characters before parsing, and if the model appends trailing prose after a valid object, a raw decoder pulls out just the JSON and discards the rest — before ever burning a retry.
- Failed calls are captured to disk, not just logged. Every call that exhausts its retry budget writes its full prompt and error to
outputs/failed_calls/, so a failure can be inspected or manually replayed without re-triggering the whole pipeline stage. - Resumable at every stage. Book text, per-page extractions, per-article syntheses, entity resolution, and even per-book map-reduce digests are all cached to disk independently. Interrupt a 2,000-page run at page 1,500 and re-running only does the missing 500 pages.
- A pre-flight cost estimator that makes zero API calls. It builds every real Pass 1 prompt locally and token-counts it; for Pass 2, if most of Pass 1 is already cached, it runs the real aggregation and builds real synthesis prompts (including detecting which entities will escalate or map-reduce) for an exact estimate instead of a heuristic. This is what prints the
~$41.38estimate before a real run ever starts. - A documented fallback for Gemini's content filter. Gemini's
PROHIBITED_CONTENTfilter is non-configurable and false-positives on ordinary middle-grade fiction (a kid, a gun, a "killed" — spy novels for 12-year-olds trip it constantly). Blocked pages retry without the context window, then optionally fall back to aclaude-*model if one is configured, before giving up and reporting the page as blocked rather than silently losing it. - A 449-line test suite that runs the entire pipeline — every cell, in the same shared namespace the notebook uses — end to end against mocked LLM calls, with 100+ assertions covering prompt construction, cost math, JSON repair, backoff timing and retry-delay parsing, resume-from-cache correctness, map-reduce routing, and content-block fallback behavior.
BookWiki.ipynb isn't written cell-by-cell in Jupyter — it's the build output. The real source lives in notebook_src/cells/ as 23 separate, individually reviewable .py/.md files (config, data structures, the LLM client, each pipeline stage, the HTML builder, tests), and notebook_src/build_nb.py assembles them into the .ipynb in cell order — syntax-checking every code cell before it's written. Editing means editing a source file and rebuilding, not scrolling through a quarter-megabyte JSON blob:
python notebook_src/build_nb.pywiki_data.json is the real output — wiki.html is just one renderer over it:
git clone https://github.com/XavionM/AI-Wikipedia.git
cd AI-Wikipedia
pip install -r requirements.txt
cp .env.example .env # then paste in your GOOGLE_API_KEYDrop your book files into books/ (prefix filenames with numbers — 01 - ..., 02 - ... — to control series order), then open BookWiki.ipynb in Jupyter and run all cells. The last cells give you:
estimate_pipeline_cost() # see the cost BEFORE spending anything - makes zero API calls
pipeline = BookWikiPipeline(wiki_title="My Book Wiki")
articles = pipeline.run()Open outputs/wiki.html in your browser when it finishes. Interrupted mid-run? Just run the cell again — everything already done is cached.
AI-Wikipedia/
├── notebook_src/ # real source — 23 reviewed files, compiled into BookWiki.ipynb
│ ├── cells/ # one file per pipeline stage
│ ├── build_nb.py # compiles cells/ -> BookWiki.ipynb
│ └── test_pipeline.py # 449-line end-to-end mocked test suite
├── BookWiki.ipynb # compiled notebook (run this)
├── examples/ # a real generated wiki (public-domain input) — see above
├── books/ # ← your book files go here (gitignored)
└── outputs/ # generated wiki + caches (gitignored)
books_*/(input novels) andoutputs_*/(generated wikis, which embed large amounts of verbatim book text) are gitignored and have never been committed to this repo —git ls-filesreturns no path under either, and no extraction or text cache, at any point in history.- The only book content in this repo is the public-domain Pride and Prejudice excerpt under
examples/, used because Project Gutenberg text carries zero copyright risk. run.txtis a real console log of the Arc of a Scythe run. It records entity names, counts, timings, and cost — no book prose.- This is a pipeline, not a distribution channel — feed it whatever you have the rights to use.
- No API key has ever been committed;
.envis gitignored and.env.exampledocuments the two variables the pipeline reads.
MIT © 2026 Xavion Mirchandani
Built by Xavion Mirchandani · GitHub
{ "wiki_title": "...", "books": ["Book One", "Book Two"], "articles": { "harry-potter": { "slug": "harry-potter", "entity_type": "character", // character|place|event|organization|item|creature|concept|work|other "short_description": "...", "summary": "paragraphs, may contain [[Links]]", "infobox": { "Label": "value" }, "sections": [{ "heading": "...", "content": "...", "sources": ["Book, p.12"] }], "relationships": [{ "target": "...", "relation": "...", "description": "..." }], "quotes": [{ "quote": "...", "context": "...", "source": "..." }], "appearances": [{ "book": "...", "pages": "3-7, 12" }], "aliases": ["Harry", "The Boy Who Lived"] } } }