architecture

That's sufficient context. Writing the page now.

Architecture

Relevant source files

• src/repodocs/cli.py
• src/repodocs/scan.py
• src/repodocs/plan.py
• src/repodocs/generate.py
• src/repodocs/backend.py
• src/repodocs/citations.py
• src/repodocs/gitlinks.py
• src/repodocs/translate.py
• src/repodocs/render.py
• src/repodocs/publish.py
• src/repodocs/_util.py

Overview

repodocs is organized as a linear pipeline of stdlib-only modules under src/repodocs/, each owning one stage: scan inventories a repository, plan turns that inventory into a page list (via an LLM or a deterministic heuristic), generate writes one Markdown page per plan entry, translate produces a localized copy of the generated pages, render bundles pages into a self-contained HTML viewer, and publish pushes the built wiki to GitHub Pages or the GitHub Wiki. backend.py is the shared LLM dispatch layer used by plan, generate, and translate; citations.py and gitlinks.py are cross-cutting utilities for validating and rewriting the Sources: citation lines that every generated page must carry. cli.py is the entry point that wires subcommands to these modules, and _util.py holds small stdlib-only helpers (path safety, slugging, logging) shared throughout.

Sources: src/repodocs/cli.py:L1-L61

Module responsibilities

Module	Responsibility
cli.py	Argument parsing, subcommand dispatch (scan, plan, generate, translate, html, publish, publish-wiki, all, setup), console-script entry points cli()/cli_all().
scan.py	Deterministic filesystem walk producing src_files, line_counts, top_dirs, manifest/README/CI/test facts.
plan.py	Builds the planner prompt from scan facts (optionally sharpened by a graphify digest), calls the LLM backend, validates/caches the resulting page list as plan.json, and provides a heuristic fallback (plan_pages).
generate.py	Reads plan.json, builds one writer prompt per page, decides via SHA-256 hashes which pages need (re)generation, dispatches writer calls in parallel, repairs/lints citations, and writes index.md.
translate.py	Translates generated pages and plan.json titles/purposes to another language, preserving structure and citations.
render.py	Builds wiki.html, a client-side viewer over the generated Markdown, with optional CDN vendoring and citation-link rewriting.
publish.py	Stages the built wiki and pushes it to a GitHub Pages branch or exports pages to the GitHub Wiki, guarded by citation enforcement and secret scanning.
backend.py	Backend selection (omp/claude/codex), prompt/contract construction, subprocess dispatch (run_llm), and a bounded thread pool (parallel_llm) for concurrent LLM calls.
citations.py	Regex-based parsing/validation/repair of [path:La-Lb](path#La-Lb) citations, and the publish-time enforcement gate.
gitlinks.py	Git-remote parsing (owner/repo, wiki URL), working-tree cleanliness checks, and rewriting citation links to absolute github.com/.../blob/<sha> URLs.
_util.py	Stdlib-only shared constants and helpers: source-file detection, path-escape-safe file resolution, slugify, dedup, logging.

Sources: src/repodocs/cli.py:L45-L61, src/repodocs/scan.py:L11-L68, src/repodocs/plan.py:L240-L282, src/repodocs/generate.py:L118-L185, src/repodocs/backend.py:L120-L205, src/repodocs/citations.py:L15-L99, src/repodocs/gitlinks.py:L38-L109

Pipeline flow

flowchart TD
    CLI["cli.py: main()"] -->|scan| SCAN["scan.py: scan / scan_inventory"]
    CLI -->|plan| PLAN["plan.py: llm_plan"]
    CLI -->|generate| GEN["generate.py: cmd_generate"]
    CLI -->|translate| TRANS["translate.py: cmd_translate"]
    CLI -->|html| RENDER["render.py: build_html"]
    CLI -->|publish| PUB["publish.py: cmd_publish"]
    CLI -->|publish-wiki| PUBW["publish.py: cmd_publish_wiki"]
    CLI -->|all| ALL["cli.py: cmd_all"]

    ALL --> SCAN
    ALL --> PLAN
    ALL --> GEN
    ALL --> RENDER

    PLAN --> SCAN
    PLAN -->|prompt| BACKEND["backend.py: run_llm"]
    PLAN -->|plan.json| GEN

    GEN -->|prompt per page| BACKEND
    GEN --> CITE["citations.py: repair_citations / lint_citations"]
    GEN -->|*.md, index.md| RENDER
    GEN -->|*.md| TRANS

    TRANS -->|prompt per page| BACKEND
    TRANS --> CITE
    TRANS --> RENDER

    RENDER --> GITLINKS["gitlinks.py: github_base / rewrite_citation_links"]
    RENDER -->|wiki.html| PUB
    GEN -->|*.md| PUBW

    PUB --> CITE
    PUB --> GITLINKS
    PUBW --> CITE
    PUBW --> GITLINKS

Loading

Sources: src/repodocs/cli.py:L397-L426, src/repodocs/cli.py:L234-L260, src/repodocs/generate.py:L118-L185

scan → plan

scan() walks the repository once (skipping .git, node_modules, and its own output directories), collecting src_files, per-file line_counts, top_dirs (files grouped by top-level directory), and presence facts for manifests, README, CONTRIBUTING, CHANGELOG, SECURITY, CI workflows, and test files; symlinks that escape the repo root are dropped. scan_inventory() wraps this with README heading extraction (readme_headings) for the planner prompt.

Sources: src/repodocs/scan.py:L11-L82

plan.py builds the planner prompt (planner_prompt) from that inventory, appending an optional graph_digest() computed from an OPTIONAL graphify-out/graph.json node-link graph (most-connected "god nodes" and most-imported files, used to bias the LLM toward core modules). It computes mandatory_slugs() (e.g. overview, installation, architecture, changelog, security, development) that the planner must always emit for a given repo's facts. The planner's JSON output is parsed (parse_pages) and validated (validate_pages) — bad slugs, duplicate slugs, and files that don't exist under the repo are dropped with a stderr warning; safe_repo_file from _util.py guards against absolute paths and .. traversal. llm_plan() caches the result via a SHA-256 fingerprint of the prompt (plan_fingerprint) in <out>/.plan.hash, so a rerun with unchanged scan facts reuses plan.json without calling the LLM. On any planner failure (missing CLI, timeout, invalid/incomplete JSON, missing mandatory slugs), plan_pages() — a deterministic heuristic based on directory grouping and file sizes — is used instead, and the stale fingerprint is deleted so a heuristic result is never cached as if it were LLM-settled.

Sources: src/repodocs/plan.py:L92-L282

plan → generate

generate.py's cmd_generate() loads the plan via load_plan() (auto-running plan if plan.json is missing), drops any entry whose slug isn't a safe filename component (_safe_page, since plan.json may have been hand-edited), and optionally filters to --pages. For each page it computes SHA-256 hashes of the page's candidate source files (compute_file_hash, memoized per run in hcache) and compares them against <out>/.hashes.json (decide_page/generate_decision) to decide generate vs skip: a page is regenerated when its .md is missing, any of its files' hashes changed, or files were added/removed from its file list. Pages needing generation are dispatched through backend.parallel_llm (bounded by REPODOCS_JOBS, default 4), and each writer prompt is built by page_prompt(), which lists the page's candidate files and — when graphify-out/graph.json exists — points the writer at it before falling back to reading files directly. Completed output is passed through citations.repair_citations() (clamping EOF-overshoot line ranges) before being written to <out>/<slug>.md; after all pages, citations.lint_citations() runs as a non-blocking warning pass and write_index() writes index.md from the full plan.

Sources: src/repodocs/generate.py:L16-L37, src/repodocs/generate.py:L63-L211

backend dispatch

backend.py centralizes how any prompt reaches an LLM. backend_name() reads REPODOCS_BACKEND (omp, claude, or codex; default claude), and effective_model() resolves REPODOCS_MODEL or a backend default (claude-sonnet-5 for claude). backend_contract() selects the vendored AGENTS.md + wiki-planner.md/wiki-writer.md contract based on whether the prompt begins with "Plan the wiki pages" or "Write the wiki page", caching the loaded text per mode. run_llm() builds and runs the actual subprocess per backend: omp invokes the vendored profile with an isolation config and restricted tools; claude runs claude -p --safe-mode with --tools Read,Grep,Glob and the contract injected via --append-system-prompt; codex runs codex exec --sandbox read-only against a symlinked, ephemeral view of the repo, with a one-time warning that read-only sandboxing does not stop the model from reading files outside the repo. parallel_llm() runs a bounded ThreadPoolExecutor (jobs_count(), clamped 1–16) over (key, prompt) items, used by both generate and translate.

Sources: src/repodocs/backend.py:L26-L205

citations and gitlinks

Every generated page is expected to carry [path:La-Lb](path#La-Lb) citation lines. citations.py defines the regexes (CITATION_RE, FULL_CITATION_RE) and three layers of checking: lint_citations() is a warning-only pass run after generate/translate; citation_error()/citation_problems() are the strict validators (label/href must match, range must be in-bounds, path must resolve inside the repo) used as the blocking gate before publish/publish-wiki via enforce_citations(); and repair_citations() auto-clamps a common LLM mistake (citing past a file's actual EOF) before problems reach the lint/enforce stages. requires_evidence() decides whether a page's prose content obligates at least one valid citation, exempting only the "Relevant source files" bullet list itself.

Sources: src/repodocs/citations.py:L12-L169

gitlinks.py supplies the git-remote-aware pieces shared by render.py and publish.py: _github_slug()/wiki_remote_url() parse a GitHub origin remote (ssh or https) into an owner/repo slug and the corresponding .wiki.git URL; github_base() combines that slug with the current HEAD sha into a blob/<sha> base URL; citations_safe() decides whether it's honest to rewrite citations to that absolute URL (requires a clean tracked tree and a pushed HEAD, tolerating only untracked files under the repodocs output directory); and rewrite_citation_links() performs the actual substitution, HTML-escaping link text and URL-quoting paths so page prose can't inject markup into the rendered <a> tag.

Sources: src/repodocs/gitlinks.py:L24-L109

generate → translate / render / publish

translate.py's cmd_translate() (not fully read in this pass beyond its prompt-construction helpers) translates each generated page's Markdown via one backend call per page (translate_prompt(), which instructs the model to preserve structure, code, URLs, mermaid, and Sources: lines), and translates plan.json titles/purposes in a single call (translate_plan_prompt()/translate_plan_file()), falling back to an untranslated copy on failure. It reuses generate.write_index(), citations.lint_citations(), backend.parallel_llm()/run_llm(), and publish._pages_for().

Sources: src/repodocs/translate.py:L1-L64

render.py's build_html() (not directly excerpted here beyond its constants) assembles <out>/wiki.html, pulling in gitlinks.github_base()/citations_safe()/rewrite_citation_links() to turn relative citations into absolute GitHub blob links when the tree is clean and pushed, and _git_out() for git plumbing. It defines CDN_ASSETS/SRI/VENDOR_FILES for the default CDN-backed or --vendor offline asset loading, and LANG_LABELS/LANG_NAMES/lang_labels() for localized UI strings when rendering a translated output directory.

Sources: src/repodocs/render.py:L1-L70

publish.py stages the built wiki tree (stage_publish(), which rejects any symlinked file or directory) and pushes it either as an orphan commit to a GitHub Pages branch (publish_push(), via a throwaway detached worktree, never touching the caller's working tree) or exports pages into the repository's GitHub Wiki (cmd_publish_wiki()). Both paths call citations.enforce_citations() before allowing a real push, and use gitlinks._github_slug()/wiki_remote_url()/citations_safe()/rewrite_citation_links() to resolve the wiki remote and rewrite citation links to absolute blob URLs when safe.

Sources: src/repodocs/publish.py:L1-L100, src/repodocs/publish.py:L160-L160, src/repodocs/publish.py:L348-L382

Shared safety primitives

_util.py provides the primitives relied on throughout: safe_repo_file() resolves a repo-relative path and rejects absolute paths, .. traversal, NUL bytes, and symlink escapes — used by scan, plan.validate_pages, generate.page_prompt/decide_page, and citations.citation_error. SLUG_RE (^[a-z0-9-]+$) is the single slug-validity rule shared by plan.validate_pages and generate._safe_page. is_source()/is_test() classify files during scanning, and slugify()/dedup() support heuristic plan construction.

Sources: src/repodocs/_util.py:L36-L103