generation

Page Generation

Relevant source files

• src/repodocs/generate.py
• tests/test_fix_generate.py

Overview

repodocs.generate implements cmd_generate, the command that turns a validated wiki plan into Markdown pages. It re-hashes each candidate source file with SHA-256, skips pages whose inputs are unchanged since the last run, and dispatches the remaining pages to the LLM backend through a bounded parallel pool, persisting hash state after every completed page so an interrupted run can resume.

Sources: src/repodocs/generate.py:L1-L14, src/repodocs/generate.py:L118-L136

cmd_generate pipeline

cmd_generate(repo, out, only, dry_run, force) runs in five stages:

1. Load the plan via load_plan(repo, out, allow_omp=not dry_run) and drop any page whose slug is unsafe with _safe_page, since plan.json may be hand-edited and is treated as untrusted input.
2. If only is given, filter to the requested slugs, calling die(...) if none match.
3. Load the on-disk hash store (load_hashes) and, for dry_run, print the [action] slug (reason) -> path decision plus the full generation prompt for every page without invoking the backend.
4. Otherwise, create out and run Phase 1 (main thread): call decide_page for every page against a shared hcache, printing [skip] slug (unchanged) immediately for pages that don't need work and collecting the rest into todo.
5. Phase 2: build (slug, prompt) items for todo and hand them to parallel_llm; only the main thread writes files and persists hashes as results arrive.

Sources: src/repodocs/generate.py:L118-L149

Sources: src/repodocs/generate.py:L118-L185

SHA-256 change detection

Each candidate file is hashed with compute_file_hash, which reads the file in 64KB chunks through hashlib.sha256(). Hashes are cached per run in hcache so files shared by multiple pages are hashed only once.

Sources: src/repodocs/generate.py:L40-L46, src/repodocs/generate.py:L89-L91

decide_page(repo, out, page, hashes, force, hcache) resolves each page's action:

• For every file in page["files"], it resolves the path with safe_repo_file(repo, f), skipping candidates that escape the repository (e.g. symlinks pointing outside, or ../-relative paths) before hashing them.
• If force is set, the action is always ("generate", "forced", current).
• Otherwise it looks up the stored record for the slug in hashes, defensively coercing a non-dict record or a non-dict files value to {}, and calls the pure decision function generate_decision.

Sources: src/repodocs/generate.py:L79-L99

generate_decision(md_exists, stored_files, current_hashes) is a pure function with no I/O:

Condition	Result
.md file does not exist	("generate", "new")
a file was added to the page's set	("generate", "changed: <path> (added)")
a file was removed from the page's set	("generate", "changed: <path> (removed)")
a shared file's hash differs from the stored hash	("generate", "changed: <path>")
none of the above	("skip", "unchanged")

The comparison is set-based (cur - sto, sto - cur) over the current and stored file dictionaries, then a per-file hash diff for files present in both.

Sources: src/repodocs/generate.py:L63-L76

The .hashes.json store maps slug -> {"files": {relpath: sha256hex}}. load_hashes treats a missing file, an OSError, or a JSONDecodeError as an empty store, forcing regeneration of everything rather than crashing; save_hashes writes it back with json.dumps(..., indent=2).

Sources: src/repodocs/generate.py:L49-L60

test_decide_page_corrupt_hash_record_falls_back_to_regenerate exercises this defensiveness directly: passing {"p": "not-a-dict"} or {"p": {"files": "not-a-dict"}} as the stored hashes both yield action == "generate" instead of raising.

Sources: tests/test_fix_generate.py:L101-L117

test_decide_page_skips_escaping_candidate_before_hashing verifies that a symlink pointing outside the repo (escape.py) and a literal ../outside/secret.py path are both excluded from current, leaving only real.py, while the page still generates because it's new.

Sources: tests/test_fix_generate.py:L51-L69

Slug safety

_safe_page(page, out) guards against a hand-edited plan.json supplying a malicious slug. It requires slug to match SLUG_RE and requires the resolved destination out / f"{slug}.md" to remain under out (via Path.relative_to), printing skip: unsafe slug {slug!r} in plan.json to stderr and returning False otherwise. cmd_generate applies this filter to all_pages immediately after loading the plan, before any --pages filtering or hashing.

Sources: src/repodocs/generate.py:L102-L120

test_safe_page_rejects_traversal_slug confirms ../evil is rejected with an "unsafe slug" stderr message while good-page passes. test_reject_unsafe_slug_writes_nothing_outside_out runs cmd_generate end-to-end with a plan containing both an unsafe and a safe page (bypassing plan.py's own validation by monkeypatching load_plan), and confirms no evil.md is written anywhere under the repo while good-page.md is written normally.

Sources: tests/test_fix_generate.py:L17-L48

Prompt construction and the graph hint

page_prompt(page, repo) builds the exact prompt sent to the backend for a page: the slug, title, purpose, a bullet list of candidate files (or (none detected)), a "context economy" instruction to read in slices, and a reference to the AGENTS.md contract. If repo is given and safe_repo_file(repo, "graphify-out/graph.json") resolves to a real in-repo file, an extra paragraph is inserted directing the writer to consult the tree-sitter knowledge graph first. safe_repo_file again ensures a symlinked graph.json that escapes the repo is not surfaced.

Sources: src/repodocs/generate.py:L16-L37

test_page_prompt_omits_hint_for_escaping_graph_symlink and test_page_prompt_includes_hint_for_real_in_repo_graph confirm the hint text graphify-out/graph.json appears in the prompt only when the file is a genuine in-repo file, not an escaping symlink.

Sources: tests/test_fix_generate.py:L74-L98

Parallel generation and crash-resume persistence

items = [(s, page_prompt(pagemap[s], repo)) for s in todo] is passed to parallel_llm(repo, items, on_submit), which yields (slug, result) pairs as jobs complete; on_submit logs [start] slug (reason) when a job is dispatched. For each result:

• FileNotFoundError triggers die(missing_resource_message(res), 1) (backend executable missing).
• subprocess.TimeoutExpired prints a timeout message referencing REPODOCS_TIMEOUT and increments failed.
• A generic Exception prints its message and increments failed.
• A nonzero returncode prints {backend_name()} exit {code}: {failure_detail(res)} and increments failed.
• Otherwise the page is written: (out / f"{slug}.md").write_text(repair_citations(repo, res.stdout)), the hash record is updated (hashes[slug] = {"files": curmap[slug]}), a [done n/total] log line is emitted, and save_hashes(out, hashes) is called immediately — persisting progress after each individual page so an interrupted run does not lose completed work.

save_hashes is called once more after the loop as a final flush.

Sources: src/repodocs/generate.py:L150-L177

test_index_includes_full_plan_when_pages_filtered exercises this pipeline through _fake_parallel_llm, a stand-in that never spawns a real subprocess and instead yields a CompletedProcess with returncode=0 directly.

Sources: tests/test_fix_generate.py:L10-L14, tests/test_fix_generate.py:L121-L140

Citations linting and the index

After the generation loop, cmd_generate computes written (pages whose .md exists) and calls lint_citations(repo, out, written). It then computes all_written from all_pages (not the --pages-filtered subset) and calls write_index(repo, out, all_written), so index.md always reflects the full plan regardless of a --pages restriction. _gitignore_notice(repo, out) runs last, and if no pages were written at all, a diagnostic is printed pointing at the backend's errors.

Sources: src/repodocs/generate.py:L177-L185

write_index(repo, out, pages) writes # <repo name> Wiki, a ## Pages heading, and one Markdown link per page (- [title](slug.md)).

Sources: src/repodocs/generate.py:L207-L211

test_index_includes_full_plan_when_pages_filtered generates only beta via only={"beta"} while alpha.md already exists from an earlier run, and confirms index.md lists both Alpha/alpha.md and Beta/beta.md.

Sources: tests/test_fix_generate.py:L119-L140

.gitignore notice

_gitignore_notice(repo, out) computes out's path relative to repo; if out is outside repo (ValueError) it returns silently. If out == repo, rel.parts is empty and the function returns without indexing into it. Otherwise, if the repo has a .git directory and out's top-level directory entry ("<dir>/") is not already present in .gitignore, it runs git -C repo ls-files <rel>; if the output dir is already tracked by git, no nag is printed (the docs are deliberately committed), otherwise it prints a note: suggesting .gitignore or a deliberate commit.

Sources: src/repodocs/generate.py:L188-L204

test_gitignore_notice_out_equals_repo_does_not_raise confirms calling _gitignore_notice(tmp_path, tmp_path) returns rather than raising an IndexError on the empty rel.parts.

Sources: tests/test_fix_generate.py:L143-L146

Plan table rendering

render_plan_table(pages) formats a fixed-width text table with columns SLUG, TITLE, FILES (count of page.get("files", [])), and PURPOSE, padding each column to the widest value and inserting a ----style separator row after the header.

Sources: src/repodocs/generate.py:L214-L224