docs: modernization pipeline + C API tree (Phase 1 of N) - #119
Merged
Conversation
Reverse the lost-source DocBook HTML into maintainable Markdown, rendered to
HTML by a small stdlib+pandoc generator. Version/copyright/boilerplate now live
in ONE place instead of duplicated across 2703 generated pages.
Scaffold:
- docs-src/_data/site.toml project/copyright/base-url (version read LIVE
from dist/RELEASE, not hard-coded)
- docs-src/_templates/page.html.tmpl the ONE HTML shell (header/nav/footer/
version/copyright) replacing per-page boilerplate
Generator (docs-src/build.py):
- loads site.toml + live version (5.3.33 from dist/RELEASE)
- walks docs-src/**/*.md, renders each via pandoc + the template -> docs-build/html
- PDF and man outputs stubbed as clear phase-3/4 seams (build_pdf/build_man)
Extractor (docs-src/_migrate/extract.py):
- strips DocBook boilerplate (navheader/navfooter/libver/generator/Prev-Next) by
stable classes, converts the semantic body to GFM, restores the API schema
(title front-matter + Description/Parameters/Errors/Class/See-Also sections),
rewrites foo.html->foo.md cross-links, keeps programlisting code verbatim
No-loss gate (docs-src/_migrate/verify.py):
- per-page OLD-prose vs NEW-prose word-multiset retention + code-block and
parameter/error sub-section drop detection; exits non-zero on hard drop
Proven end-to-end on the C API tree (470 pages): extract -> build ->
verify reports mean word retention 99.98%, 0 outliers, 0 hard drops. Generated
pages carry version 5.3.33 (not the old 11.2.5.3) via the shared template.
flake.nix: add pandoc to the devShell so the pipeline is reproducible.
.gitignore: ignore docs-build/ (generated) and __pycache__.
Does NOT touch src/ or the old docs/ (kept until full sign-off). C++/STL and
the guide trees, man pages, PDF, and CI are follow-up phases.
Coccinelle convention checksNo new violations. ✅ Resolved since baseline (2) -- update dist/cocci/baseline.txt to lock these in. |
ABI diff vs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of N — the machinery + one proven tree
This is the foundation of the docs modernization in
docs-src/PLAN.md:a Markdown-source + generator pipeline that reverses the lost-source DocBook
HTML into maintainable Markdown and renders it to HTML. It is proven
end-to-end on one doc tree — the C API reference (470 pages) — with a
real no-loss content gate. The other 16 trees, man pages, PDF, and CI are
deliberately left for follow-up phases (clean seams left in place).
What's in this PR
Scaffold
docs-src/_data/site.toml— project name, copyright, base URL in ONE place.The version is read live from
dist/RELEASEat build time (→5.3.33),never hard-coded.
docs-src/_templates/page.html.tmpl— the single HTML shell (header, nav,footer, version, copyright) that replaces the boilerplate previously
duplicated across 2703 generated pages.
Generator —
docs-src/build.py(stdlib + pandoc, no framework)site.toml+ live version, walksdocs-src/**/*.md, renders each viapandoc + the template into
docs-build/html/(boilerplate injected once).build_pdf()/build_man()are explicit phase-3/4 stubs so the shapeis fixed and a follow-up wires pandoc without reshaping the file.
Extractor —
docs-src/_migrate/extract.py(one-time reverse-DocBook)Library Version 11.2.5.3,generator meta, stylesheet link, Prev/Next) by stable DocBook classes;
API schema: front-matter (title/api-name/source) then Description /
Parameters / Errors / Class / See-Also as headings;
foo.html→foo.mdcross-links; keepsprogramlistingas verbatimfenced
ccode.No-loss gate —
docs-src/_migrate/verify.pymultiset, and separately asserts no code block or parameter/error sub-section
was dropped; exits non-zero on any hard structural drop (CI-ready).
Completeness-verifier results (C API tree, 470 pages)
Before / after evidence
Library Version 11.2.5.3(hard-coded on every page)Version 5.3.33(fromdist/RELEASE, via template)navheader/navfooter/libver/generator meta per pagepage.html.tmpldbgetsections##/###headingsdbtstruct code<pre class="programlisting">```cblockxrefrows.htmlTooling
flake.nix: added pandoc to the devShell (python3 already present) so thepipeline is reproducible under
nix develop. Verifiednix developstillevaluates.
Scope / safety
docs-src/,flake.nix(devShell input),.gitignore.src/ortest/(a concurrent security review owns those).docs/(kept until full sign-off).docs-build/is git-ignored (generated).Honest status
retention with zero drops on the full C API tree; version/copyright/header
now single-sourced.
extract.py, tune per-tree cleanup), man pages (build_manstub), PDF(
build_pdfstub), thedocs.ymlCI (spelling/links/mandoc/completeness gateagainst the full
db.hAPI surface), and publishing.