A Python tool that discovers poems through category-tree traversal on Wikisource, classifies each page by type, extracts verse structure from rendered HTML, and writes validated JSONL output. It handles collections (with section titles and reading order), multi-version hubs, and author normalization automatically.
Two modes cover different needs. Online mode queries the MediaWiki API and fetches live HTML — ideal for incremental or targeted runs. Offline mode processes local Wikimedia dump files (SQL, XML, NDJSON) with zero network I/O — ideal for building the full corpus in one batch. Both share the same classification engine and produce identical output schemas; only the provenance field differs ("api" vs "dump").
Data license: Source code is MIT-licensed. Extracted data is subject to Wikisource's own licenses (typically Public Domain or CC-BY-SA).
- 🔄 Dual-mode pipeline — online (async API) or offline (local dumps), same output schema
- 📚 Exhaustive collection extraction — natively detects Parsoid HTML markup (
tableItem), ProofreadPage tables, and resolves entire collection structures with clean section titles and reading order - 🏷️ Bibliographic imprint parsing & metadata inheritance — extracts publication year and publisher from edition mentions (
parse_imprint) and automatically inherits missing metadata from parent collections to child poems - 🔁 Multi-pass discovery & SQLite subpage lookup — recursive bounded passes (up to 3 passes) combined with SQLite
pagessubpage queries ensure zero missed poems - 🔀 Hub support — groups multiple editions of the same poem under a single hub
- 👤 Author normalization — cleans names from HTML microdata and wikitext templates
- 📊 Structured output — stanzas, verses, metadata, and normalized text in Pydantic-validated JSONL
- 🔁 Resumable — SQLite index tracks progress; interrupted runs pick up where they left off
- 🤖 Bot authentication — higher API rate limits via Wikisource bot account
- 🧹 Post-processing — built-in deduplication, enrichment, and corpus analysis commands
- 🏆 Golden Record merging — reconcile online and offline outputs into the ultimate corpus, keeping the richest version of every poem automatically
git clone https://github.com/sharle4/scriptorium.git
cd scriptorium
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e .Online mode:
scriptorium scrape --lang fr --category "Poèmes par Auteur" --workers 15Offline mode:
scriptorium scrape --lang fr --category "Poèmes" --mode offline --dumps-dir ./dumps --workers 8Resume an interrupted run:
scriptorium scrape --lang fr --category "Poèmes" --resumeOnline mode — Bot account (recommended)
Anonymous API requests are rate-limited. A bot account gives higher limits and signals legitimate automated use to Wikimedia.
- Log in to your Wikisource account (e.g.,
fr.wikisource.org). Create one if needed. - Go to Special:BotPasswords (
https://fr.wikisource.org/wiki/Special:BotPasswords). - Enter a bot name (e.g.,
Scraper). Your login will beYourUsername@Scraper. - Grant Basic rights and High-volume editing. Click Create.
- Save the generated password (shown once).
Create a .env file at the project root:
WIKISOURCE_BOT_USERNAME=YourUsername@Scraper
WIKISOURCE_BOT_PASSWORD=the_generated_passwordOr pass --bot-user and --bot-pass on the command line.
Offline mode — Dump files (~35 GB)
Offline mode needs three types of dump files in a single directory.
SQL + XML dumps — freely available from https://dumps.wikimedia.org/frwikisource/latest/:
| File | Size | Content |
|---|---|---|
frwikisource-latest-page.sql |
~726 MB | Page metadata |
frwikisource-latest-categorylinks.sql |
~743 MB | Category membership |
frwikisource-latest-linktarget.sql |
~887 MB | Category name resolution |
frwikisource-latest-redirect.sql |
~4.5 MB | Redirects |
frwikisource-latest-pages-articles1.xml-* (4 files) |
~11.4 GB | Raw wikitext |
HTML NDJSON dumps — require a free Wikimedia Enterprise account:
# Edit credentials in the script first
.\dumps\download_enterprise_dumps.ps1Downloads frwikisource_namespace_0_*.ndjson (11 files, ~21.5 GB) with fully rendered HTML for every namespace-0 page.
Total disk space: ~35 GB for dumps + 2–4 GB for the generated index.
📦 Output Data Structure
Each run produces three files in --output_dir:
| File | Description |
|---|---|
poems.jsonl.gz |
One JSON per line per poem (may contain duplicates in online mode) |
poems.cleaned.jsonl.gz |
Deduplicated, title-normalized. One unique poem per page_id. |
poems_index.sqlite |
SQLite index for fast lookups and resume support |
{
"page_id": 1234567,
"revision_id": 9876543,
"title": "Les Fleurs du mal/Spleen",
"language": "fr",
"wikisource_url": "https://fr.wikisource.org/wiki/Les_Fleurs_du_mal/Spleen",
"collection_page_id": 1234000,
"collection_title": "Les Fleurs du mal",
"section_title": "Spleen et Idéal",
"poem_order": 14,
"hub_title": null,
"hub_page_id": 1234567,
"metadata": {
"author": "Charles Baudelaire",
"publication_date": "1857",
"source_collection": "Les Fleurs du mal",
"publisher": "Poulet-Malassis et de Broise",
"translator": null
},
"structure": {
"stanzas": [
["Quand le ciel bas et lourd pèse comme un couvercle", "..."],
["Quand la terre est changée en un cachot humide,", "..."]
],
"raw_markers": ["<div class=\"poem\">"]
},
"collection_structure": null,
"normalized_text": "Quand le ciel bas et lourd...\n\nQuand la terre est changée...",
"raw_wikitext": "<poem>\nQuand le ciel bas et lourd...\n</poem>",
"checksum_sha256": "a1b2c3d4...",
"extraction_timestamp": "2025-01-15T14:30:00+00:00",
"provenance": "api"
}| Field | Description |
|---|---|
normalized_text |
Flat text. Verses joined by \n, stanzas by \n\n. Main field for NLP. |
raw_wikitext |
Original MediaWiki markup. Useful for editorial analysis. |
structure.stanzas |
Nested array: stanzas → verses. |
metadata.author |
From HTML microdata or wikitext templates, normalized. |
collection_page_id |
Links poem to parent collection page. null if unknown. |
section_title |
Section within the collection (e.g., "Spleen et Idéal"). |
poem_order |
Zero-based position in collection. Reconstructs reading order. |
hub_page_id |
Groups multiple editions. Equals own page_id if standalone. |
collection_structure |
Full structure object (first poem of each collection only). |
provenance |
"api" (online) or "dump" (offline). |
checksum_sha256 |
SHA-256 of raw_wikitext. For deduplication and integrity. |
🏗️ Architecture
Both modes use the same modules for classification and extraction:
classifier.py— classifies pages asPOEM,POETIC_COLLECTION,MULTI_VERSION_HUB,AUTHOR,DISAMBIGUATION, orOTHERusing category membership and HTML structure signals.parsing.py— extracts poem structure from rendered HTML (<div class="poem">/<poem>blocks), splitting into stanzas and verses.processors.py— merges HTML metadata (itemprop) with wikitext template metadata (mwparserfromhell), normalizes author names, produces validatedPoemSchema.
Why HTML, not just wikitext? Most Wikisource poems use ProofreadPage transclusion (<pages index="Book.djvu" from=42 to=43 />). The raw wikitext contains only this directive, not actual verses. Only the rendered HTML has fully resolved poem content.
Async Producer-Consumer model:
- Producer crawls the category tree via MediaWiki API.
- N consumer tasks (
--workers) concurrently fetch wikitext + HTML, classify, and route pages. - Writer thread handles all sync disk I/O (JSONL + SQLite).
Rate limiting: semaphore, 10-RPS sliding window, exponential backoff, Retry-After compliance.
Sequential 5-phase batch pipeline with no network I/O:
Phase 1 Phase 2 Phase 3 Phase 4 Phase 5
SQL Dumps ──────> NDJSON Files ───> XML Dumps ─────> In-Memory ──────> Write Output
→ SQLite → Classify → Wikitext Enrichment → JSONL + DB
Index (multiprocess (poems+colls) & Inheritance
& multi-pass)
Phase 1 — Build index: Parses SQL dumps into SQLite. Custom state-machine parser handles MySQL INSERT syntax and varbinary UTF-8 decoding. Dual-root category-tree traversal simultaneously traverses Poèmes and Poèmes par Auteur (in French) to capture all 27,900+ poem categories.
Phase 2 — Stream NDJSON: Streams ~21 GB of Enterprise HTML dumps. Parallel classification with native support for Parsoid HTML (tableItem, ProofreadPage tables, and subpages). Clean section extraction filters out TOC artifacts (headers, leader dots, page numbers). Multi-pass discovery (up to 3 bounded passes) coupled with SQLite subpage resolution ensures all child poems and sub-collections are enqueued.
Phase 3 — Extract wikitext: Streams XML dumps with iterparse at constant memory for both pending poems and collections, retrieving raw_wikitext and metadata templates ({{titre}}, {{infoédit}}, {{header}}).
Phase 4 — Enrich collections & inherit metadata: Parses bibliographic imprints (parse_imprint) to extract publication dates and publishers for each collection. Full Collection objects (with sections, ordered poems, date, publisher) are constructed. Missing metadata in child poems automatically inherits from the parent collection.
Why both NDJSON and XML? NDJSON has rendered HTML (resolved transclusions) but no wikitext. XML has raw wikitext but no rendered HTML. Both are needed.
Phase 5 — Write: Processes each poem through PoemProcessor.process() (same code as online mode). Writes validated poems.jsonl.gz, cleaned poems.cleaned.jsonl.gz, and SQLite index sorted by page_id for strict reproducibility.
The online and offline pipelines have complementary strengths and weaknesses. Neither dataset alone is the ground truth.
The merger.py module reconciles two JSONL outputs into a single "Golden Record" corpus. It streams both files, deduplicates by page_id, and resolves conflicts using a configurable strategy. For the keep_richest strategy, it applies a deterministic richness scoring algorithm to each duplicate pair:
| Signal | Points | Rationale |
|---|---|---|
collection_page_id is present |
+50 | Being linked to a parent collection is the highest-value contextual data. |
hub_page_id ≠ page_id |
+30 | Proper grouping under a multi-version hub adds scholarly context. |
section_title is present |
+20 | Section placement within a collection aids reading-order reconstruction. |
| Each non-empty metadata field | +10 | One bonus per filled field among author, publication_date, source_collection, publisher, translator. |
| Text length | +1 per 100 chars | Tie-breaker: longer normalized_text suggests a more complete extraction. |
When scores are tied, the version from file A is kept. The merger logs a full summary — total records per file, duplicates found, conflict outcomes, and elapsed time — so you can audit every merge.
📋 Full CLI Reference
After installation, the tool is available as scriptorium or python -m scriptorium.
# Online (default)
scriptorium scrape --lang fr --category "Poèmes par Auteur" --workers 15 --tree-log
# Offline
scriptorium scrape --lang fr --category "Poèmes" --mode offline --dumps-dir ./dumps --workers 8
# Resume an interrupted run (either mode)
scriptorium scrape --lang fr --category "Poèmes" --resume| Option | Default | Description |
|---|---|---|
--lang |
(required) | Wikisource language code (fr, en, etc.) |
--category |
(required) | Root category to start from |
--output_dir |
./data/ |
Output directory |
--workers |
3 |
Online: concurrent requests. Offline: CPU cores. |
--limit |
None |
Process at most N pages (testing) |
--resume |
false |
Skip already-processed pages |
--tree-log |
false |
Write per-author exploration tree logs |
--mode |
online |
online or offline |
--dumps-dir |
None |
Path to dump files (required for offline) |
--bot-user |
$WIKISOURCE_BOT_USERNAME |
Bot username (online only) |
--bot-pass |
$WIKISOURCE_BOT_PASSWORD |
Bot password (online only) |
# Clean: deduplicate by page_id, normalize titles
scriptorium clean -i data/poems.jsonl.gz -o data/poems.cleaned.jsonl.gz
# Enrich: fill missing collection_page_id via API
scriptorium enrich -i data/poems.cleaned.jsonl.gz -o data/poems.enriched.jsonl.gz --lang fr
# Analyze: print corpus statistics
scriptorium analyze data/poems.cleaned.jsonl.gz
# Debug: extract poems with unidentified collections
scriptorium debug -i data/poems.enriched.jsonl.gz -o data/debug.unidentified.jsonl.gzCombine an offline and an online corpus (or any two JSONL files) into a single deduplicated Golden Record:
scriptorium merge \
--file-a data/offline_corpus.jsonl.gz \
--file-b data/online_corpus.jsonl.gz \
--output data/golden_record.jsonl.gz \
--strategy keep_richest| Option | Default | Description |
|---|---|---|
--file-a |
(required) | First input file (.jsonl or .jsonl.gz) |
--file-b |
(required) | Second input file (.jsonl or .jsonl.gz) |
--output, -o |
(required) | Merged output file |
--strategy |
keep_richest |
Conflict resolution when a page_id exists in both files |
Strategies:
keep_richest(default) — scores both versions of a duplicate poem and keeps the semantically richer one. Ties go to file A. Best for building the most complete corpus possible.keep_a— always keeps file A's version on conflict. Useful when A is your trusted reference (e.g., a carefully curated offline dump).keep_b— always keeps file B's version on conflict.
MIT. See LICENSE.