read-along

Turn any audiobook you own into a read-along: the audio plays in your browser while the text scrolls in sync, and any word can be clicked for a one-line definition. Everything runs locally, everything is free, and nothing ever leaves your machine.

There is no ebook required — the text is transcribed from the audio itself with whisper.cpp, word-timed from the recognizer's own token timestamps.

Bring your own audiobook. This repository ships no audio and no book text; it is a tool you point at files you already own.

docs/GUIDE.md is the full walkthrough — every step from audiobook file to reading, with time estimates and what each stage actually does. Below is the condensed version.

Quickstart

Requirements: Python 3.11+, ffmpeg, whisper-cpp — macOS: brew install ffmpeg whisper-cpp; Linux: apt install ffmpeg plus whisper.cpp (Homebrew on Linux works too); Windows is untested, WSL should behave like Linux. Then one whisper model file:

curl -L -o models/ggml-large-v3-turbo-q5_0.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo-q5_0.bin

Then three commands:

# 1. Bring the audio in (one .m4b/.m4a, or a folder of per-chapter .mp3s)
python3 pipeline/ingest.py my-book ~/audiobooks/MyBook.m4b

# 2. Transcribe it (roughly 4 minutes per audio hour on an Apple-silicon Mac)
python3 pipeline/transcribe.py --book my-book all

# 3. Build the reader data and serve
python3 pipeline/build_reader.py --book my-book all
python3 reader/serve.py            →  http://127.0.0.1:8899/reader/

ingest.py writes a starter books/my-book/book.toml — edit it to set the title, author and language (see examples/book.toml for every option). The library holds as many books as you like: each one is a folder under books/, the reader opens with a picker, and every book keeps its own reading position and bookmarks.

No audiobook handy? Try it in 10 minutes

LibriVox recordings are public domain, so this demo is fully legal — and any other LibriVox title works the same way:

curl -L -o "/tmp/The Velveteen Rabbit.mp3" \
  https://archive.org/download/velveteen_rabbit_librivox/the_velveteen_rabbit_williams.mp3
python3 pipeline/ingest.py velveteen-rabbit "/tmp/The Velveteen Rabbit.mp3"
python3 pipeline/transcribe.py --book velveteen-rabbit all    # ~29 min of audio, ~2 min of work
python3 pipeline/build_reader.py --book velveteen-rabbit all
python3 reader/serve.py

Open http://127.0.0.1:8899/reader/ and you're reading along.

What's on screen

• Follow-along: the sentence being read is highlighted and kept centred; scroll freely and one tap on “Back to current line” returns you.
• Click a word for a one-line definition in the side rail (when the dictionary stage has been run — see below). Nothing is underlined; a defined word turns blue under your cursor.
• Click the left gutter of any sentence to play from there.
• Contents / Search / Bookmarks in a drawer; search covers the whole book.
• Sleep timer that only counts playing time, and per-book resume.
• Keyboard: Space play/pause, ←/→ ±10 s, ↑/↓ sentence, [/] chapter, / search, T contents, B bookmark, D define, S sleep, J back to the current line.

How accurate is the text?

It is automatic speech recognition: expect roughly ~2% word error rate — excellent for reading along, short of perfect. Two things it gets wrong most: invented proper nouns (fantasy names come out phonetically mangled) and homophones ("plumes" written as "plums"). The optional AI stages below exist to claw those back; on the book this tool was developed with, they took the text to 99.97% sentence-level accuracy.

Planned: if people like this tool, an update will add the option to use the book's PDF to improve text accuracy — the timing would still come from the transcription, but the words would come from the real book.

Optional quality stages (AI-assisted)

The core pipeline above needs no AI at all. If you use Claude Code, the repo ships four agent definitions and a skill (transcribe-book) that add three quality layers:

1. Proper-noun correction — a researcher agent builds books/<slug>/lexicon.json from fan wikis; pipeline/correct.py fixes mangled names transcript-wide and learns each book's recurring mishearings.
2. Real-word proofreading — pipeline/proof.py finds valid-but-wrong words ("aspen steaks") using confusable sets; two agents with opposing stances judge each one, and a fix lands only where both agree.
3. Whole-book dictionary — pipeline/glossary.py collects every word in the book and agents write one-line, in-context definitions, so clicking any word in the reader answers. Needs pip install wordfreq in a venv.

Order matters: proofread (3) before glossing, and run glossary.py --book X once before proof.py --scan (the scan reads word rarity from the glossary's candidate file). What each stage does and how long it takes: docs/GUIDE.md. The runbook Claude Code follows is .claude/skills/transcribe-book/SKILL.md; design docs in docs/.

How it works

audio files ── ingest.py ─→ books/S/audio.m4b  (one file, faststart, chaptered)
                 ↓ transcribe.py --book S      (whisper.cpp, token timestamps)
             books/S/derived/chNN/{transcript.json, meta.json}
                 ↓ correct.py                  (optional: proper-noun fixes)
             books/S/derived/chNN/transcript.corrected.json
                 ↓ proof.py                    (optional: real-word error fixes)
                 ↓ glossary.py                 (optional: whole-book dictionary)
                 ↓ build_reader.py --book S
             books/S/data/{book.json, chNN.json, dictionary.json}  →  the reader

The design hinge: whisper.cpp emits a timestamp for every token, and the reader's sync comes from those. Text corrections never touch the raw transcript — the corrected text is matched back onto the raw timing per segment with difflib, so fixing a word can never break the highlighting. Every stage writes plain JSON to disk and re-runs independently.

What is where

books/<slug>/            your book (never committed — see .gitignore)
  book.toml              title, author, language + optional knobs
  audio.m4b              single streamable file, produced by ingest.py
  lexicon.json           proper nouns (optional; see examples/lexicon.json)
  derived/               transcripts and intermediate stage outputs
  data/                  built reader JSON: book.json, chNN.json, dictionary.json
pipeline/                the scripts, stdlib-only except glossary.py (wordfreq)
reader/                  the web app: one HTML file, one CSS, one JS, no build
lexicon/confusables.json 1,200+ sound-alike sets used by proof.py
models/                  whisper model weights (you download these)
docs/                    design docs for the four pipeline stages

Technical notes worth knowing:

• One audio file per book. The browser streams it whole; chapters are time offsets. ingest.py remuxes with -movflags +faststart so playback starts instantly, and serve.py handles Range requests so seeking works.
• Timing always comes from the raw transcript's token timestamps; text corrections are matched back onto them, so fixing a word never breaks sync.
• The reader is intentionally frameworkless — ES5, one IIFE, no build step.
• exFAT drives: if anything acts strangely, delete the ._* litter (find . -name '._*' -delete) and keep your Python venv on an internal disk.

Legal

You supply your own audiobook files and this tool never uploads, converts for redistribution, or shares them; all output stays in your local books/ folder, which git ignores. Transcribing a book you own, for your own reading, is personal use. Do not publish generated transcripts of copyrighted books.

Credits

• whisper.cpp — transcription
• LanguageTool confusion sets (LGPL) and pimentel/homophones — the generic parts of lexicon/confusables.json
• wordfreq — word rarity for the dictionary stage

MIT licensed. Built for one book first, generalized because it worked.