Make pinned Rustkyll source builds byte-reproducible

[alexeygrigorev]

Parent: #115

Outcome

Pinned legacy source builds are byte-for-byte reproducible across separate clean workspaces and processes. This unblocks the generated-source freshness gate for Studio → Courses without weakening provenance or accepting a wall-clock/random-order snapshot.

Incident evidence

Independent #115 QA found scripts/build_pinned_legacy_sources.py --check stale after the current-main integration. Regenerating aligned the two checked artifacts only temporarily: two consecutive full builds changed them again.

The repeat-build delta is exactly 209 generated paths:

• 203 main-site podcast pages whose season JSON-LD endDate inherits the current build time from an undated collection item;
• docs and podwiki feed.xml timestamps;
• main-site sitemap.xml ordering; and
• the three .rustkyll-manifest.json map orderings.

The affected builds use pinned Rustkyll 0.4.6 and 0.5.0 code paths. Local source inspection found no existing SOURCE_DATE_EPOCH or reproducible-build flag. Rustkyll uses Utc::now/Local::now in collection fallback dates, site.time, feed fallback, and future-post filtering; sitemap input and manifest maps originate from Rust HashMap iteration/serialization. The manifest also records source mtimes.

Required implementation

Rustkyll compatibility lanes

• Add standard SOURCE_DATE_EPOCH support to compatible 0.4.x and 0.5.x releases/backports; do not force an unrelated renderer upgrade.
• Parse the value once and fail closed on invalid, negative, overflow, or ambiguous input.
• Thread one frozen UTC instant through collection default dates, site.time, empty-feed fallback, and future-post cutoff.
• In reproducible mode, use UTC when the source has no explicit timezone; preserve an explicit configured timezone.
• Sort final sitemap entries deterministically, root first and then lexicographically by final loc, without changing membership.
• Canonically serialize manifest map keys. Preserve the manifest rather than excluding it from checked output.

Website wrapper

• Derive SOURCE_DATE_EPOCH from each already-verified pinned commit's committer timestamp with an exact revision-bound git query.
• Pass only the minimal deterministic environment to Rustkyll and record source-date-epoch=<value> in checked deterministic-override provenance.
• Normalize copied tracked source-file mtimes to that same epoch before rendering so manifest values are stable across clean clones.
• Reject missing, malformed, mismatched, or out-of-range revision/epoch/mtime data with bounded diagnostics.
• Regenerate generated-path-baseline.jsonl and source-build-provenance.json only after deterministic rendering is proven.

Current pinned epoch reference values, to be independently derived rather than hard-coded as authority:

• main: 1785872368
• docs: 1786017922
• podwiki: 1785736104

Acceptance criteria

• Two full builds of every pinned source in separate fresh workspaces/processes are byte-identical for the complete generated trees and all checked artifacts.
• A third full build followed by scripts/build_pinned_legacy_sources.py --check remains green without changing tracked bytes.
• The exact generated path membership and route contract remain unchanged; main sitemap retains the same 811-URL multiset in documented deterministic order.
• All 203 affected podcast pages retain valid JSON-LD shape with stable source-derived/frozen dates; docs/podwiki feeds have stable timestamps.
• All three manifest files have stable key order and stable source-mtime values across clean clones.
• Tests cover one frozen instant shared by every Rustkyll time consumer, invalid epoch inputs, explicit timezone behavior, deterministic sitemap membership/order, and canonical manifest serialization in both compatibility lanes.
• Website tests cover strict revision-bound epoch derivation, minimal environment inclusion, invalid data failure, safe tracked-file mtime normalization, and deterministic override provenance.
• --artifacts-only or an immediate --check is not accepted as reproducibility proof because neither independently proves a clean full rebuild.
• Make Studio → Courses the canonical course-operations surface #115's 26 canonical Studio Courses targets, exact /cadmin source mappings, access matrix, current-main Import historical registration totals safely and show aggregate event counts #112/Recover fail-closed ECR manifest verification for active release capture #117/Ship real public content on the final editorial route map #105 behavior, CMP adoption, and public route inventory remain unchanged.
• Normal uv-backed compatibility/Django/quality/Playwright gates pass, with no secret, credential, private source row, or production mutation.

Process

This is a technical blocker for #115. Rustkyll changes require their own review/release evidence, after which website pins must be updated and the #115 candidate refreshed to current main before a new independent QA pass. Do not bypass the generator, post-process volatile output as an unexplained mask, or weaken the freshness check.

[alexeygrigorev]

Engineering handoff — reproducible Rustkyll legacy builds

Implemented in three isolated, uncommitted worktrees:

• website: branch issue-118-rustkyll-repro, base 5b6a8418f105f7cbdf7d3821d5a3c4f0a4f4c613
• Rustkyll 0.5 lane: branch issue-118-source-date-epoch-0.5, base v0.5.0 (c37e90bca2910dc63f27de6368f7b6334bf98ac5)
• Rustkyll 0.4 lane: branch issue-118-source-date-epoch-0.4, base v0.4.6 (fc83da6fa4586fd8f64c02252ce80f4e704d22c9)

Implementation

Rustkyll now:

• parses SOURCE_DATE_EPOCH once per build as strict, non-negative ASCII decimal Unix seconds;
• rejects empty, signed, fractional, whitespace-padded, non-numeric, overflowing, and unrepresentable values with a bounded value-free diagnostic;
• threads one frozen UTC instant through collection fallback dates, site.time, empty-feed fallback time, and the future-post cutoff;
• uses UTC in reproducible mode when the site has no explicit timezone, while preserving an explicit site timezone;
• emits sitemap root first, then all remaining entries lexicographically by final loc, without changing membership;
• serializes manifest maps canonically while preserving the existing manifest schema and loader compatibility.

The website wrapper now:

• obtains the epoch with the exact pinned revision-bound query git show -s --format=%H%n%ct --no-patch <full-sha> and verifies the returned SHA;
• bounds and validates the revision, epoch, tracked-file inventory, file types, paths, total count/bytes, and nanosecond mtime range;
• normalizes every tracked regular source file mtime to the pinned commit epoch and verifies inode/type/mtime after the write;
• adds only SOURCE_DATE_EPOCH to the existing credential-free minimal child environment for Rustkyll release and PyPI builds;
• records source-date-epoch=<value> in generated build provenance for Rustkyll sources.

TDD and verification evidence

Before implementation, the new six-test Rustkyll suite failed 6/6 in both compatibility lanes for the intended reasons: live wall clocks/timezones, future-post leakage, unstable sitemap order, unstable manifest map order, accepted invalid epochs, and different complete output trees.

After implementation:

• Rustkyll 0.4 focused debug suite: 6 passed
• Rustkyll 0.5 focused debug suite: 6 passed
• Rustkyll 0.4 focused release suite: 6 passed
• Rustkyll 0.5 focused release suite: 6 passed
• separate source workspaces + separate processes + different ambient TZ values: complete generated trees are byte-identical
• Rustkyll 0.4 full suite: 4,059 library tests plus all integration/doc tests passed
• Rustkyll 0.5 full suite: 4,148 library tests plus all integration/doc tests passed
• both lanes: project-standard Clippy with warnings denied passed
• both lanes: formatting checks passed
• both lanes: optimized release builds passed
• website focused wrapper suite: 27 passed
• website full compatibility suite: 302 passed
• website full Django suite: 1,511 passed
• website Ruff/format checks: passed
• website mypy: 279 source files passed
• git diff --check: passed in all worktrees

The common 0.4/0.5 implementation files and the issue test are byte-identical where their bases permit it; main.rs, generator.rs, and lib.rs were adapted to each compatible base without importing unrelated 0.5-only code into 0.4.

Deliberately unresolved lifecycle work

No commit, push, tag, release, pin update, artifact regeneration, merge, deploy, issue acceptance, or issue closure was performed.

The next owning steps are:

1. Review and commit the two Rustkyll compatibility-lane changes through the Rustkyll lifecycle.
2. Publish compatible 0.4.x and 0.5.x releases (release binary and PyPI lane).
3. Update the website's Rustkyll versions, download URL, and verified digests only after those releases exist.
4. Rebuild the real pinned DTC legacy sources in independent fresh workspaces with the released tools; compare complete trees byte-for-byte.
5. Regenerate the checked compatibility artifacts so their provenance contains the derived source-date-epoch=<value> entries, then run the offline source-artifact check.
6. Separate tester verification and product acceptance before any website commit/merge.

[alexeygrigorev]

QA Review — phase 1 patch validation

PASS for the exact uncommitted phase-1 candidates. This validates patches on the website-pinned Rustkyll v0.4.6 and v0.5.0 bases only. It is not a release-candidate, version, pin, artifact, or publication claim; newer existing Rustkyll tags are outside this candidate.

Candidate identity and preservation

• Rustkyll 0.4: /home/alexey/git/rustkyll/.tmp/issue-118-source-date-epoch-0.4, HEAD fc83da6fa4586fd8f64c02252ce80f4e704d22c9, branch issue-118-source-date-epoch-0.4.
  • pre/post binary-diff SHA-256: 02a2ae905f338a7117a499cf31a294a1e5bcaace786f41ccaeaeb3dcbc83ea0
  • pre/post porcelain-status SHA-256: d5f678be11aacd22456d2850d2d8deaa5ee9101ec20160200f3d1f5a3b208a0f
• Rustkyll 0.5: /home/alexey/git/rustkyll/.tmp/issue-118-source-date-epoch-0.5, HEAD c37e90bca2910dc63f27de6368f7b6334bf98ac5, branch issue-118-source-date-epoch-0.5.
  • pre/post binary-diff SHA-256: dbc401a4b43c896d14f1d0f7096336033cc28fb81af985422666ea047926c813
  • pre/post porcelain-status SHA-256: d5f678be11aacd22456d2850d2d8deaa5ee9101ec20160200f3d1f5a3b208a0f
• Website: /home/alexey/git/dtc-website/.tmp/website-118-rustkyll-repro, HEAD 5b6a8418f105f7cbdf7d3821d5a3c4f0a4f4c613, branch issue-118-rustkyll-repro.
  • pre/post binary-diff SHA-256: 8312d35282dbf1b5380742981167a89cda74b228b4f7eed9492a71fa2e343706
  • pre/post porcelain-status SHA-256: a6d55e5e8dff08a66e9bf9a4bb2404c5ba30c92a490747959b7b52423e1a8914
  • no untracked files
• The two Rust lanes retain the same two untracked candidate files with identical Git blob IDs in both lanes: src/build_time.rs = 5e360e710c90b852ad7b76f1dafee1088f6cda9a; tests/issue_118_reproducible_builds.rs = dcd0581176dd3c937fa1404c4d0926d6761cb709.
• The normalized issue-change stream, including the two new-file blob identities, has SHA-256 2b4404f468794c99e323ccbadcdbd48f0f59b56241f9dd985e2c1e938522567d in both Rust lanes. The seven common tracked files and two new files are byte-identical; the changed-line stream for base-adapted generator.rs, main.rs, and lib.rs is also identical (e7a1dc98d96c08281f54374cb267b5ecd7707f97910a5fc468642182b5db55c8, 116 changed lines).

Independent review

• SOURCE_DATE_EPOCH is read once at build entry, accepts only non-empty unsigned ASCII decimal seconds representable by Chrono, and rejects signs, whitespace, fractions, non-ASCII/non-Unicode, overflow, and out-of-range instants with the bounded value-free invalid SOURCE_DATE_EPOCH diagnostic.
• The same frozen UTC instant is threaded through collection fallback dates, future-post filtering, site.time, populated/empty feeds, and both full-site context paths. Reproducible builds without a configured timezone use UTC; a valid explicit configured timezone remains authoritative.
• Sitemap membership is unchanged and serialized root-first, then lexicographically by final <loc>. Incremental manifest maps use canonical BTreeMap serialization while remaining load-compatible with the existing schema.
• The website derives the epoch with exact git show -s --format=%H%n%ct --no-patch <full-sha> output and verifies the returned full SHA, strict epoch shape/range, and two-line result without echoing hostile values.
• Only SOURCE_DATE_EPOCH is added to the existing credential-free child environment. Provenance is added only for Rustkyll sources.
• Tracked-file normalization uses a bounded NUL-delimited Git inventory, rejects invalid/escaping/missing/non-regular entries and count/size/range violations, preserves access time, avoids symlink following, and verifies type, inode, and exact nanosecond mtime after each write.
• Website pins/download URL/digests remain at 0.4.6; compatibility/source_config.py, checked artifacts, versions, and lockfile are untouched. No release, artifact regeneration, pin, commit, push, tag, merge, deploy, acceptance, or closure occurred.

Commands and results

Rustkyll, independently in both lanes:

• cargo test --test issue_118_reproducible_builds — 6 passed in 0.4 and 6 passed in 0.5.
• cargo test --all-targets --quiet — 0.4: 4,059 total library tests (4,057 passed, 2 ignored) plus every integration target passed; 0.5: 4,148 total library tests (4,146 passed, 2 ignored) plus every integration target passed.
• cargo test --doc --quiet — passed in both lanes (0 doctests).
• cargo fmt --all -- --check — passed in both lanes.
• Project-standard CI Clippy, cargo clippy -p rustkyll --no-deps -- -D warnings — passed in both lanes.
• Optimized tests were not repeated because debug full/focused suites plus direct binary whole-tree evidence were sufficient; the task permitted optimized execution only if evidence required it.

Website:

• uv run pytest compatibility/tests/test_baseline.py -q — 27 passed.
• make test — compatibility 302 passed; Django 1,511 passed.
• make test-playwright-core — 53 passed, 3 deployment-only skipped.
• uv lock --check, make lint, make format-check, make typecheck, git diff --check — passed; mypy checked 279 source files, Ruff format checked 357 files.
• make migrations-check, make django-check, make database-portability-check — passed, including OpenAPI and management-capability parity checks.

Adversarial complete-tree proof

I created separate fresh fixture workspaces for each lane, assigned the two copies deliberately different source mtimes (946684800 versus 1893456000), indexed them independently, ran the website candidate's real _normalize_tracked_source_mtimes(..., 1723158000), then invoked separate Rustkyll processes with TZ=Pacific/Honolulu versus TZ=Europe/Berlin and the same SOURCE_DATE_EPOCH=1723158000.

• 0.4: 31 generated files in each tree; diff -qr clean; normalized complete-tree SHA-256 a9f42da035b82eb54f913155ff4985bacc9e46854c34b51cea12697d98cfccfd in both workspaces.
• 0.5: 34 generated files in each tree; diff -qr clean; normalized complete-tree SHA-256 bdee64d904e45e87519cefe8c5289d1cbab30df6d93822410ce7ea2a4085a6bb in both workspaces.

As an adversarial control, running before mtime normalization changed only .rustkyll-manifest.json, whose source/global timestamp values reflected the deliberately different mtimes. That confirms the website normalization is necessary and that the specified combined pipeline removes the last observed source-metadata variance.

Non-blocking toolchain observation

An additional nonstandard cargo clippy --all-targets --all-features probe under installed Clippy 1.95 exits 101 in both lanes solely at unchanged base test code src/yaml.rs:601 (Some(3.14), deny-by-default clippy::approx_constant; 0.4/0.5 source at HEAD is identical there). Allowing that one base-only lint makes the broader probe complete successfully. The repository's project-standard warnings-denied Clippy command above passes without exceptions, so this is not a candidate regression or phase-1 blocker.

No screenshots were required for this non-UI phase.

[alexeygrigorev]

Product Acceptance — phase 1 patch validation

ACCEPT for the exact uncommitted phase-1 patches on their pinned bases. This is a patch-validation gate only. It is not acceptance of a release candidate, published Rustkyll version, website pin/artifact integration, the full #118 checklist, or #115.

Accepted candidate identity

• Website issue-118-rustkyll-repro: HEAD 5b6a8418f105f7cbdf7d3821d5a3c4f0a4f4c613; binary-diff SHA-256 8312d35282dbf1b5380742981167a89cda74b228b4f7eed9492a71fa2e343706; NUL-delimited porcelain-status SHA-256 a6d55e5e8dff08a66e9bf9a4bb2404c5ba30c92a490747959b7b52423e1a8914; no untracked files.
• Rustkyll 0.4 lane issue-118-source-date-epoch-0.4: HEAD/tag fc83da6fa4586fd8f64c02252ce80f4e704d22c9 / v0.4.6; binary-diff SHA-256 02a2ae905f338a7117a499cf31a294a1e5bcaace786f41ccaeaeb3dcbc83ea0e; NUL-delimited porcelain-status SHA-256 d5f678be11aacd22456d2850d2d8deaa5ee9101ec20160200f3d1f5a3b208a0f.
• Rustkyll 0.5 lane issue-118-source-date-epoch-0.5: HEAD/tag c37e90bca2910dc63f27de6368f7b6334bf98ac5 / v0.5.0; binary-diff SHA-256 dbc401a4b43c896d14f1d0f7096336033cc28fb81af985422666ea047926c813; NUL-delimited porcelain-status SHA-256 d5f678be11aacd22456d2850d2d8deaa5ee9101ec20160200f3d1f5a3b208a0f.
• Both Rust lanes have the same new-file object IDs: src/build_time.rs = 5e360e710c90b852ad7b76f1dafee1088f6cda9a; tests/issue_118_reproducible_builds.rs = dcd0581176dd3c937fa1404c4d0926d6761cb709.

Product and safety acceptance

• One frozen instant governs the affected renderer behaviors; reproducible builds use UTC only when no timezone is explicitly configured, while an explicit valid site timezone remains authoritative.
• Sitemap order and manifest serialization become deterministic without changing membership or manifest schema/load compatibility.
• Invalid or hostile epoch/revision metadata fails closed through short, value-free diagnostics. Inputs are shape/range bounded and never reflected into logs.
• The website derives the epoch from an exact full-SHA-bound Git query, verifies the returned SHA, and normalizes only a bounded tracked regular-file inventory with path, type, inode, size/count, and exact-mtime checks.
• Only SOURCE_DATE_EPOCH is added to the existing minimal credential-free Rustkyll child environment; provenance is explicit and Rustkyll-only.
• Independent QA passed both full Rust suites, project-standard warnings-denied Clippy and formatting, the complete website/Django/Playwright/quality gates, and adversarial separate-workspace/process/tree comparisons. No UI changed, so screenshots are not applicable for this phase.

The independent QA evidence is accepted as sufficient for these exact fingerprints: #118 (comment)

Required next lifecycle gates

Existing v0.4.8 and v0.5.1 releases mean these pinned-base branches must not be published as-is or reuse those versions.

1. Port the accepted semantics, uncommitted, onto the latest 0.4.x and 0.5.x maintenance tips intended for v0.4.9 and v0.5.2.
2. Independently test those exact maintenance-tip candidates and obtain their release acceptance before commit/tag/publication.
3. After both releases exist, update website versions, download URL, and verified digests.
4. Rebuild every real pinned DTC source in separate fresh workspaces/processes and prove complete-tree byte equality, stable 811-URL sitemap membership/order, all 203 affected podcast JSON-LD pages, feeds, manifests, and source mtimes.
5. Regenerate the checked artifacts/provenance, pass a third rebuild plus --check, and repeat independent website QA and product acceptance.
6. Only then refresh Make Studio → Courses the canonical course-operations surface #115 onto current main and repeat its complete independent gate.

No #118 checkbox is checked and the issue remains open because its release and real-source integration criteria are deliberately outside this phase-one acceptance.

[alexeygrigorev]

Release-candidate engineering handoff — GREEN (uncommitted)

Phase-1 QA and PM acceptance were treated as the immutable source of truth. Before porting, I re-verified all accepted candidates unchanged:

• accepted 0.4 lane: HEAD fc83da6fa4586fd8f64c02252ce80f4e704d22c9; tracked diff SHA-256 02a2ae905f338a7117a499cf31a294a1e5bcaace786f41ccaeaeb3dcbc83ea0e; status SHA-256 d5f678be11aacd22456d2850d2d8deaa5ee9101ec20160200f3d1f5a3b208a0f
• accepted 0.5 lane: HEAD c37e90bca2910dc63f27de6368f7b6334bf98ac5; tracked diff SHA-256 dbc401a4b43c896d14f1d0f7096336033cc28fb81af985422666ea047926c813; same status SHA-256
• accepted website lane: HEAD 5b6a8418f105f7cbdf7d3821d5a3c4f0a4f4c613; tracked diff SHA-256 8312d35282dbf1b5380742981167a89cda74b228b4f7eed9492a71fa2e343706; status SHA-256 a6d55e5e8dff08a66e9bf9a4bb2404c5ba30c92a490747959b7b52423e1a8914

Release-candidate lanes

• 0.4.9: /home/alexey/git/rustkyll/.tmp/issue-118-portcheck-04, branch issue-118-release-0.4.9, base 98a1c26e5fae9cc717733b81583c09a8d6bc94eb (v0.4.8)
• 0.5.2: /home/alexey/git/rustkyll/.tmp/issue-118-portcheck-05, branch issue-118-release-0.5.2, base ed6faff4e3fded124d67b8ac3f7d86d0417ac097 (origin/main, with v0.5.1 ancestor)

Both lanes are intentionally uncommitted.

Port and version evidence

• Ported the accepted strict SOURCE_DATE_EPOCH parser, one frozen UTC build instant, deterministic site/collection/feed/future-cutoff behavior, root-first final-location sitemap ordering, and canonical manifest serialization.
• src/build_time.rs is byte-identical to the accepted blob in both lanes: 5e360e710c90b852ad7b76f1dafee1088f6cda9a.
• The 0.4 regression test is byte-identical to the accepted blob: dcd0581176dd3c937fa1404c4d0926d6761cb709.
• The 0.5 regression test has only the required semantic API adaptation for the current collect_entries(..., config) signature; blob 689e7de7297645ca51cbc87b94472b13457516e1.
• The 0.5 sitemap merge retains the newer output: false collection and sitemap: false page/document exclusions, then sorts only retained entries by final absolute loc, keeping the root first. Existing exclusion tests and the accepted ordering regression pass together.
• Release versions are aligned in Cargo.toml, the root Rustkyll entry in Cargo.lock, python/pyproject.toml, python/uv.lock, and python/rustkyll/__init__.py.
• Optimized binaries report exactly rustkyll 0.4.9 and rustkyll 0.5.2.

Gates run in each lane

• ./scripts/cargo-safe test --test issue_118_reproducible_builds -- --nocapture — PASS, 6/6
• ./scripts/cargo-safe test — PASS, full suite
• cargo fmt --all -- --check — PASS
• ./scripts/cargo-safe clippy -p rustkyll --no-deps -- -D warnings — PASS
• ./scripts/cargo-safe build --release — PASS
• ./scripts/cargo-safe test --release --test issue_118_reproducible_builds — PASS, 6/6
• uv run --frozen --project python python -m unittest discover -s python/tests -p test_main.py — PASS, 18/18
• git diff --check — PASS

The regression suite explicitly builds in separate workspaces/processes and compares complete normalized output trees byte-for-byte; it passes in debug and release profiles for both candidates.

Final candidate fingerprints

• 0.4.9 tracked binary diff SHA-256: 52dc4c036c8e77c634615875c618c1cac7e575b566dfa9bd65481ca0d9660fcc
• 0.5.2 tracked binary diff SHA-256: d3ce17073a6723ea0480d648c7d99b8846291f66f36947994d7a63acc57e3aac
• both status-path SHA-256: 1d39e13dff51723caddde451ca225ecf3868eaea79d7f705ea8da04eb4be79bc
• expected shape in each lane: 12 modified tracked files plus new src/build_time.rs and tests/issue_118_reproducible_builds.rs

Remaining release lifecycle

Independent QA should verify these exact release-candidate fingerprints, followed by PM release acceptance. Only the release owner should then commit/push, tag, publish 0.4.9 and 0.5.2, and record artifact checksums. Website pin changes remain a separate accepted lifecycle step. I did not commit, push, tag, publish, or edit the website candidate.