epythet v2 — decision record and fleet migration plan (2026-09)

[thorwhalen]

epythet v2 — decision record and fleet migration plan (draft, 2026-09-10)

Status: draft for maintainer review. Written by the crowsnest session after five parallel research/audit tracks and a baseline render of real fleet packages. Every claim marked [measured] was reproduced locally today; claims marked [research] come from the linked reports and were spot-checked where cheap.

Source reports (local, not committed): ~/.local/share/epythet/docs-modernization-2026-09/ — research_doc_systems.md (64 refs), research_themes.md (96 refs), research_validation.md (91 refs), fleet_impact.md, recall_briefing.md, plus the baseline artifact scan (scan_results.txt, warning_classes.txt). Sanitized copies land in misc/docs/ with the v2 branch.

0. What epythet wants to be

A way to get beautiful, correct documentation from a Python package with no boilerplate in the package: point it at a repo, and conventions (README, docstrings, package layout, pyproject metadata) produce the site. Every rendering concern lives in epythet's layer, not in docstrings. Docstrings carry the minimum markup that both a human reader of source and an agent need, and epythet absorbs the rest at build time. The same source produces a human site and agent-readable twins.

1. Where we are (facts that constrain the design)

Fact	Value	Why it matters
Fleet repos referencing the Pages action directly	102, all @master, none pinned	One epythet release changes every site on its next push
Repos via the wads reusable CI stub (also calls the action @master)	118	Same exposure, no per-repo with: control
The action installs	pip install epythet (unpinned) then epythet quickstart . --ignore …	The CLI contract to keep: quickstart, --ignore, output at ./docsrc/_build/html/
Repos with a committed docsrc/	107 (all stale template copies; zero customizations; 1 repo with hand-written pages)	CI overwrites conf.py anyway [measured: live dol footer has no copyright line], so committed copies are dead weight
[tool.epythet] users	0	The config section is free to redefine
Docstring markup census	~18,000 RST constructs in >1,000 files vs ~5,200 Markdown-isms in ~600 files; 23,627 >>> lines	RST must keep rendering; Markdown habits must stop breaking
Packages mixing RST field lists and Google sections	66 (incl. dol, epythet)	Rules out griffe/mkdocstrings (one style per package)
Modules per package	median 8; 117 repos >10; 45 repos >20; 11 repos >50	Nested navigation is needed for roughly half the fleet
Baseline render of dol / i2 with current epythet [measured]	127 / 119 docutils warnings; 54 / 52 problematic spans; 77 / 24 "Returns:" one-liners rendered as prose; 31 / 21 accidental definition lists; landing page is a flat 48-entry module list	The artifacts are real, frequent, and mostly silent (no warning)
Sphinx activity [measured via GitHub API]	506 PRs merged in 2025, 14 in 2026; last release 9.1.0 (2025-12-31)	Sphinx is stalled, not dead
MkDocs ecosystem [research]	MkDocs 1.6.1 unmaintained since 2024-08; Material for MkDocs feature-frozen since 2025-11 with maintenance promised for about a year; mkdocstrings in maintenance mode; Zensical alpha (0.0.x) without automated API-nav plugins	Not a foundation for a 200-repo fleet this year

2. Decisions I would make

D1. Engine: stay on Sphinx 9, pinned >=9,<10

Not MkDocs, not Zensical, not pdoc. The MkDocs side is a fork war; Sphinx is a stall, and a stalled Sphinx still builds. Revisit in about 18 months when Zensical ships native API docs. Pin Sphinx, MyST, and the API generator with upper bounds; today's Sphinx>=3.3.1 unbounded already splits the fleet across Sphinx 8.1 and 9.0 depending on the action's Python version [research].

D2. Markup: RST-compatible docstrings, Markdown pages, and a build-time normalizer

Pages and docstrings are parsed by different machinery: autodoc emits RST regardless of page format [research, verified]. So:

• Pages are MyST Markdown. index.md is a {include} of ../README.md with relative-image fixups, GitHub alerts, badges, mermaid fences, and heading anchors all rendering [research, verified]. This is the landing page fix.
• Docstrings are left alone. napoleon keeps RST field lists and Google sections both working, simultaneously.
• A docstring normalizer runs at build time (autodoc-process-docstring) and fixes the artifacts agents and humans actually produce: blank line before >>>/bullets after prose; ``` fences to .. code-block::; ## Heading to a bold rubric; [text](url) to an RST link; Returns: text one-liners to a proper section; *args/**kwargs escaped in prose. default_role = "code" makes single backticks render as code, which matches the Markdown habit. This is retroactive across 200 repos with zero docstring edits, and it also restores doctests that Sphinx currently silently drops (367 at-risk blocks across 83 packages [research, measured on the fleet]).
• The convention for new docstrings ("epythet dialect") is the intersection of Markdown and RST that renders correctly under the normalizer: plain paragraphs; Google sections (Args:, Returns:, Raises:, Examples:); doctests as the examples; single or double backticks for code; **bold**; bullet lists preceded by a blank line; types in annotations, never in the docstring. No RST roles or directives needed. This is what the shipped skill teaches agents.

Not doing: a blanket Markdown-to-RST conversion of docstrings (the dead commonmark hook in the template). It would destroy the fleet's RST field lists.

D3. API pages: an off-the-shelf recursive generator replaces autogen.py

Two candidates, both verified to give a nested tree with zero per-project templates [research]: sphinx-autoapi (static parsing, no import, one option) and recursive autosummary (stdlib Sphinx, imports the package, stock template recurses since Sphinx 3.1). The seam is one keyword: api_generator = "autosummary" | "autoapi". My leaning for the default is autosummary, because it keeps autodoc in the loop so the normalizer hook fires and today's import-based behaviour is preserved; autoapi is the alternative when a package cannot be imported in CI. The v1 spike decides after checking whether the normalizer can be hooked under autoapi. Either way autogen.py and the RST half of templates.py are deleted, and nesting appears automatically for the 117 repos with more than 10 modules. Facade __init__.py re-exports must not double-document (drop imported-members).

D4. Themes: furo by default, a curated seven, deterministic variety

Default furo: 400 KB static payload versus 9.94 MB for sphinx_rtd_theme [research, measured], real light/dark, the de-facto choice for pip, pytest, attrs, black, mypy. Curated set exposed to the theme-choice skill:

Theme	Use when
furo	default; small-to-medium API library, no brand
shibuya	package has a brand or landing page to sell; most actively developed
pydata-sphinx-theme	data/ML/scientific, deep hierarchies, version switcher; healthiest project
sphinxawesome-theme	the shadcn/ui-flavoured option (Tailwind + Alpine); best search
sphinx-book-theme	notebook-heavy docs
alabaster	genuinely single-page utilities only
sphinx_rtd_theme	legacy/familiarity only

Excluded with evidence: sphinx-immaterial (broken on Sphinx 9), sphinx-material, sphinx-press-theme.

Config: [tool.epythet] theme = "furo" | "auto" | <name>, accent = "#hex" (default: derived from the package name in OKLCH with fixed lightness, so contrast is WCAG AA by construction across all 274 fleet names [research, measured]), mode = "auto" | "light" | "dark", and [tool.epythet.theme_options] as a verbatim passthrough that always wins. theme = "auto" hashes the package name into the curated list, so the fleet is varied but every package is stable across rebuilds (never rotate on date or build number: it breaks snapshots). Drop sphinx-toggleprompt (its pixel offset is tuned to RTD geometry; copybutton already handles prompts).

D5. Configuration becomes a single source of truth; docsrc/ becomes optional

docsrc/conf.py is generated at build time from [tool.epythet] plus conventions, and a committed docsrc/conf.py becomes a two-line shim (from epythet.sphinx_conf import * then optional overrides). This closes i2mint/epythet#13 structurally. parse_config keeps its 5-tuple (wads and 106 frozen conf.py copies unpack it); theme and layout live in a new dataclass accessor. Fix the precedence bug where setup.cfg silently wins over pyproject.toml in the 10 dual-config repos. Config keys: display_name, copyright, theme, accent, mode, theme_options, ignore, api_generator, agent_outputs, docs_dir.

D6. Agent-facing outputs by default

Every build also emits: llms.txt (index) plus rendered .md twins of every page via sphinx-llm with the full dump disabled, <link rel="alternate" type="text/markdown"> relations injected (the plugin does not do this itself [research, verified]), and the objects.inv that already ships. Cost: a second build pass (roughly doubles build time; builds are seconds today). The rendered site is justified by humans; the twins and index are the cheap agent surface, since GitHub Pages has no content negotiation. Later, and only if wanted: a generated fleet-wide intersphinx_mapping so cross-package references become live links (nobody else has 200 interlinked packages).

D7. An "AI artifacts" section in every site

epythet discovers a repo's agent artifacts by convention ({pkg}/data/skills/*/SKILL.md, .claude/skills, .claude/agents, CLAUDE.md, AGENTS.md, .codex/, .cursor/rules) and renders a "For AI agents" section from a default template: what skills exist, how to install them (gh skill install owner/repo …), what subagents exist, and the llms.txt pointer. A shipped skill tells an agent where to find these artifacts in any repo.

D8. epythet validate: five tiers, doctree-based detection, a growing ledger

The decisive research result [research, measured on a 30-case specimen]: a strict sphinx-build -W -n flags 9 of 21 artifact classes; the other 12 render silently and wrongly, and the errors it does raise mean content was silently deleted. Detection must therefore read the docutils doctree of each docstring, obtainable without a build at about 1,000 docstrings per second, catching 20 of 21 classes with one inherently ambiguous false positive.

Level	Reads	Cost	Gates CI?
0 lint	docstring text (ruff D, pydoclint, coverage, backslash-in-non-raw-string)	~2 s	yes
0.5 parse	per-docstring doctree with stub Sphinx roles registered	~0.5 s	yes, the main gate
1 build	Sphinx warning stream with [type] suffixes	10–40 s	yes
2 render	-b text snapshots, empty pages, unresolved xrefs, dangling anchors, images	5–20 s	yes, snapshot diff
3 review	LLM over rendered text/screenshots of changed pages	minutes	never; it proposes ledger rules

Exit codes distinguish the failing level; --format table|json|jsonl|sarif. One seam: backend= (Levels 0 and 0.5 are backend-independent; only 1 and 2 touch Sphinx).

Ledger: one YAML file per rule (id, title, detector: regex|doctree|build-warning|render|llm, pattern, example_bad, example_good, fix, severity, precision, autofixable, first_seen) with a sibling annotated .py fixture that is also the regression test. Rules ship inside epythet and are public. Observations (occurrences with file paths and snippets from real repos) are append-only JSONL kept outside the repo under ~/.local/share/epythet/ledger/, because they are derived data from repos that are not all public. Level 3 and human reviewers add rules in proposed status; a maintainer promotes them. Seed: the 21 specimen classes plus the baseline findings above (e.g. the Returns: text one-liner rule, 77 hits in dol alone).

D9. Repair and migration tooling for the fleet sweep

• epythet repair: applies the normalizer's safe rewrites to source (blank lines, fences, one-liner sections, heading rubrics) with a diff for review; unsafe cases (prose *args, unmatched backticks) stay diagnostics. Keeps diagnose_doctest_code_blocks and repair_package import paths stable because wads' wads-docstring-render skill imports them (pins >=0.1.14).
• epythet migrate-style: optional RST-field-list to Google conversion via docstring_parser.compose() plus LibCST for formatting-safe rewrites [research]. Opt-in per module; never a fleet-wide mass conversion by default.
• epythet sweep: runs validate at level 0.5 across the fleet (about two minutes total) to produce the real artifact frequency distribution that decides severities.
• wads' skills stay the fleet-level orchestrators; epythet exposes library functions they call, not a competing CLI.

D10. AI artifacts for epythet itself

Consumer skills shipped in epythet/data/skills/ (layout per the skill-package-setup policy; gh skill install i2mint/epythet in the README): epythet-setup (convention-over-config quickstart), epythet-docstring-style (the dialect), epythet-validate (levels, ledger workflow), epythet-repair-migrate (the sweep procedure), epythet-theme (choose and parametrize, with links to each theme's docs and sphinx-themes.org), epythet-ai-artifacts (finding and documenting agent artifacts), epythet-pages (the existing Pages diagnosis). Subagents: docs-reviewer (Level 3 packet review), docs-migrator (per-repo sweep). These are documented in the README and rendered in epythet's own "For AI agents" section.

3. Back-compatibility and the fleet migration plan

The fleet's only coupling to epythet is the composite action (which installs epythet from PyPI unpinned) and the quickstart CLI. That gives a one-line kill switch and a one-line rollout.

1. Safety pin, before any v2 release. Change the action's install line to pip install "epythet<0.2". One commit in epythet, zero fleet PRs; 220 repos are frozen on today's behaviour. Rollback of anything later is reverting this file.
2. Develop v2 as 0.2.x on a branch, breaking changes allowed. Keep the contract: epythet quickstart <dir> --ignore … still works and still writes HTML to ./docsrc/_build/html/. make_docsrc, make_autodocs, make remain as thin compatibility entry points that route to the new pipeline. parse_config keeps its signature. Python floor for epythet moves to 3.11 (Sphinx 9 requires it); the action's default Python moves to 3.12. The floor only affects the CI docs job, not the packages' own requires-python.
3. Validate v2 against the fleet before anyone sees it. Build a sample of 20 to 30 repos of varied size and style locally with v2; require: every build succeeds; artifact counts at Level 0.5 do not regress; -b text snapshots of v1 versus v2 are reviewed for content loss. This is the first real use of epythet validate.
4. Pilot. Tag publish-github-pages@v2 in the action directory (installing epythet>=0.2,<0.3); 5 to 10 direct-CI repos switch their uses: ref (small PRs). Watch their published sites for a week.
5. Flip the default. Point @master's install line at >=0.2,<0.3. Rollout is naturally gradual: each repo's site rebuilds on its next push to its default branch. Also pin the wads stub and templates to a tagged action ref so future flips are deliberate.
6. The fleet documentation sweep (agent-driven, one session per repo, using the shipped skills): validate → repair → review → fix correctness and completeness → set theme if the default is wrong for the package → delete the committed docsrc/ (CI regenerates it; keep only where hand-written pages exist) → land. Ledger observations accumulate; new rules get proposed.

Out of scope for the flip: the 24 repos on the legacy epythet make . github path, which keep working via the compatibility entry points and are migrated during the sweep.

4. Decisions that are yours

1. Do the safety pin now? I recommend committing epythet<0.2 in the action immediately (reversible, no fleet PRs). Say no if you prefer every fleet site to track PyPI latest.
2. All 200 sites change look at once when the default flips (step 5)? I assume yes; the alternative is per-repo opt-in via [tool.epythet], which costs 200 PRs.
3. Committed docsrc/ in 107 repos: delete during the sweep (recommended: CI regenerates it; add to .gitignore), or refresh to the thin shim and keep committing it for local builds?
4. Agent twins on by default (llms.txt plus .md twins, second build pass), or opt-in?
5. Theme policy: theme = "auto" fleet-wide (deterministic variety), or have the sweep agent pick explicitly per package using the theme skill? Both are supported; this is about what the default does when nobody chooses.
6. Docstring style migration: normalizer-only (recommended: existing RST field lists keep rendering; new code follows the Google-style dialect), or a mass RST-to-Google conversion during the sweep?
7. Level 3 review runner: the in-session agent (skill drives it, no API key, my recommendation) or an API-backed epythet review command?
8. Version: 0.2.0 (recommended) or 1.0.0 for the new epythet?

5. Work packages (one session each, spawned after your answers)

WP	Scope	Depends on
WP0	Safety pin in the action; pin the wads stub to a tagged action ref	Q1
WP1	Core v2: config SSOT, generated conf.py, MyST index with README include, recursive API generator, theme registry and OKLCH accent, normalizer, agent outputs, compatibility entry points, tests, smoke test on 3 fleet repos	D1–D7
WP2	epythet validate levels 0, 0.5, 1 and the ledger with seed rules and fixtures	finding model from WP1's package layout
WP3	validate levels 2 and 3, repair, migrate-style, sweep	WP2
WP4	Shipped skills, subagents, "For AI agents" template, README rewrite	WP1, WP2
WP5	Action v2, fleet validation run, pilot, flip	WP1–WP4
WP6	Fleet sweep (many sessions, one per repo)	WP5

Related open issues folded in by design: #13 (conf.py SSOT), #9, #10, #8 (Markdown vs RST), #2 (doctest prompt toggle), #11 (doctest detection), #7 (version and date on the page). The wiki's "Documentation problems ledger" page becomes the first ledger entries and the wiki page is retired.

[thorwhalen]

Maintainer decisions (2026-09-10)

All eight recommendations accepted as written: (1) safety pin now; (2) fleet-wide look change on the default flip; (3) delete committed docsrc/ during the sweep; (4) agent twins on by default; (5) theme = "auto" as the default; (6) normalizer-only, no mass style conversion; (7) Level 3 review driven by the in-session agent through a skill; (8) version 0.2.0.

Two additions

A. Coverage and correctness sweep, coupled with the artifact repair. When the fleet sweep edits a docstring to fix rendering, it also improves it: coverage (every public callable documented, with an example), correctness (docstring matches signature and behaviour), completeness. A research track on what makes documentation good for agents and humans feeds the docstring-style skill and the sweep procedure.

B. Single-document aggregates as part of the default rendering. Every site also publishes one flat document containing the whole documentation, at a stable direct URL: <site>/<package>.md (for agents) and <site>/<package>.pdf (for humans). Nested hyperlinked HTML is great for humans and poor for agents; a flat aggregate is trivial to point either at. epythet offers the tools to produce them (epythet aggregate), the landing page links to them, and the "For AI agents" section names the Markdown URL. Tracked as WP1b in #16.