Normalize the pinned Podwiki source into typed route-neutral records

[alexeygrigorev]

Parent: #43
Downstream projection/search owner: #44

Normative authority:

• _docs/specs/03-github-content-and-people.md#source-ownership and #podwiki-adapter
• _docs/specs/01-platform-architecture.md, for GitHub-owned Podwiki pages, typed links, citations, graph source, and search source
• _docs/specs/02-url-link-seo-compatibility.md#canonical-wiki-route, noting that route selection belongs downstream and does not rename the source identity
• closed owner decision Decision: Use PostgreSQL search while preserving public search contracts #24, which assigns ranking/indexing and compatibility projection behavior to Build Django-owned PostgreSQL search and cross-source graph activation #44

Product outcome

Add one deterministic, network-free adapter for the exact accepted DataTalksClub/podwiki checkout. It turns source Markdown/frontmatter and source-owned graph/search inputs into typed, route-neutral records that #43 can map onto the canonical public Wiki and #44 can use to build search/graph projections.

This child is the source-normalization boundary. It must preserve what the source says without assigning a website mount, resolving cross-repository public URLs, ranking search results, publishing graph JSON, or activating database/public state.

Exact accepted input and current evidence

The implementation binds:

• repository: DataTalksClub/podwiki;
• immutable commit: 988b79d0d655bf4755945c3118544cb9e0dbead6;
• immutable tree: 60e8821fa16af4688e782759a4a65eb1e36f9f85;
• exactly 282 _wiki/*.md public source pages;
• generated source inputs graph/graph.json with 1,072 nodes and 13,006 links, and search/search-corpus.json with 2,998 documents;
• the allowlisted public source asset assets/og-default.png; and
• caller-supplied immutable git-derived date evidence, rather than a network or moving-branch lookup inside the adapter.

At current website origin/main face8e4808d65afbf0374d1ced7a88079950d663, the checked artifacts are characterization evidence: wiki.json SHA-256 b2525b45a61aed28d035cd996852ffda17ae35f20325b33b66222dbcd102120b, wiki_graph.json f6ec4fb29e71dd179cdaa79526bfb4174ad25ee4991dfcba7f2fe3c5b0e99a13, and wiki_search.json 74d6373df63386aead5108e5addd5ecf2a83a885e891338ca72022e6bdcd3fef. They are not the new route-neutral output format and must not be copied or hand-edited as implementation.

The exact source contains 18,434 typed chip occurrences in source order: 9,008 wiki, 8,571 citation, 622 person, 153 podcast, and 80 book. Of the citation occurrences, 2,622 carry a timestamp. Frontmatter uses the observed allowlisted fields and includes the five accepted editorial type tags (guide, comparison, roadmap, transition, and how-to). These source-occurrence counts are the normalization census; the current public projection's 10,643 deduplicated relation rows are a downstream rendering choice, not source authority.

Scope

• Parse bounded YAML frontmatter and Markdown from the exact caller-supplied checkout without running repository code, plugins, WASM, Jekyll, JavaScript, Make targets, or network requests.
• Emit versioned typed page records with stable source key/path, title, summary, SEO fields, tags/type, ordered safe content blocks/headings, source checksum, immutable commit/tree provenance, and supplied git dates.
• Emit every typed chip occurrence in source order with an occurrence identity and bounded source pointer. Preserve target kind/key, visible alias label, citation episode key, citation timestamp/label, and relation context; do not silently discard timestamp data or collapse repeated occurrences.
• Normalize source aliases/labels, related/related_wiki, tags, headings, fragments, citations, and mirrored person/podcast/book keys as typed source values. Cross-repository keys remain opaque until their owning canonical resolvers consume them.
• Validate the source graph/search input schemas and preserve their route-neutral node/document/link identities and typed fields as source seeds. Build Django-owned PostgreSQL search and cross-source graph activation #44 owns website URL localization, search-document construction, ranking/indexing, deterministic public graph projection, query/filter behavior, activation/fallback, and Lambda retirement evidence.
• Produce deterministic canonical serialization and bounded allowlisted diagnostics. Two clean supported runs over the same checkout and date evidence must be byte-identical.

Explicit non-goals

No /wiki or /podwiki route, redirect, canonical, sitemap, template, browser behavior, public asset response, checked public projection rewrite, database model/migration/write, direct-sync runner/webhook/reconciliation, candidate/activation/rollback, search ranking/query service/index, public graph JSON, cross-repository URL resolution, Person inference, Studio/admin API, GitHub write, provider/production access, Lambda change/retirement, commit, push, merge, or deployment.

The source repository name may remain Podwiki for provenance. The adapter must not encode /podwiki, /wiki, or another public mount into route-neutral identities.

Acceptance criteria

• The adapter accepts only the exact repository/commit/tree and bounded allowlisted paths above; moving branch input, changed pin/tree, symlink/submodule/traversal, unexpected executable/plugin input, unsupported file, and resource-limit excess fail before output.
• All 282 pages normalize once with unique safe source keys/paths, complete checksums/provenance, deterministic ordering, supplied git dates, allowlisted metadata, and safe typed content blocks.
• All 18,434 chip occurrences normalize in source order with the exact kind census above; all 2,622 timestamped citations retain their episode key and timestamp/label; visible aliases remain presentation labels and never become inferred identity.
• related/related_wiki, the five accepted type tags, headings/fragments, citations, and opaque mirrored entity keys are represented explicitly. Unknown kinds/fields, malformed tokens/timestamps/frontmatter, duplicate identities, ambiguous headings, and unsafe Markdown/HTML/URLs fail with bounded deterministic diagnostics.
• Graph/search source inputs validate at 1,072 nodes / 13,006 links / 2,998 documents and retain route-neutral typed seeds without website URL localization, ranking, filtering, public JSON generation, or activation.
• Exact repeated generation is byte-identical and binds schema version, adapter version, source commit/tree, input digests/counts, date-evidence digest, output counts, and one canonical output digest.
• Failure emits no partial normalized corpus or database/public mutation; diagnostics exclude source bodies, arbitrary exception text, checkout paths, credentials, tokens, private/account data, and production data.
• Tests prove there is no network/provider call, repository-code execution, GitHub write, database write, public-route change, checked public-artifact drift, or dependency on Adopt secure direct-sync content ingestion and management #38/Establish canonical Person short source and relationship resolution #40/Build Django-owned PostgreSQL search and cross-source graph activation #44 runtime implementation.
• A current change-selective plan, uv-backed format/lint/type/migration checks, focused parser/schema/security/determinism tests, independent tester PASS, and PM acceptance exist before commit.

Required test scenarios

1. Normalize the exact pin twice with independently ordered filesystem enumeration and identical date evidence; compare canonical bytes/digest, 282 identities, metadata/type census, content blocks, 18,434 ordered occurrences, 2,622 timestamped citations, and graph/search seed counts.
2. Cover every observed frontmatter field, all five type tags, wiki/person/podcast/book/citation chips, visible aliases, timestamp variants, related lists, headings/fragments, duplicate visible occurrences, and Unicode.
3. Reject changed SHA/tree, missing/extra/duplicate page, unsafe key/path, symlink/submodule/traversal, YAML alias/entity/depth/node/size overrun, unknown field/kind/tag, malformed citation/timestamp/chip, ambiguous identity/heading, unsafe URL/HTML/script, invalid graph/search schema/link, and missing/inconsistent git-date evidence.
4. Prove mirrored keys remain opaque and that names, labels, case folding, URLs, email/account data, or source prose never resolve a Person/Podcast/Book identity.
5. Prove the adapter performs no network, provider, database, public-route/template, checked-projection, Studio/API, job, GitHub-write, or production action.

Verification and screenshots

This is a source-only parser/normalizer. Screenshots are not_applicable only when the change-selective graph and artifact comparison prove no public artifact, route, template, CSS, JavaScript, or rendered behavior changed. The independent tester still verifies the frozen diff, exact source/census, focused Django/unit gates, required repository gates, and documented smoke disposition.

Dependencies and sequencing

Satisfied authority: #24 and the immutable pinned source above. This child does not depend on #38, #40, #43, or #44 and may be implemented independently in an isolated worktree.

After acceptance, #43 consumes the typed pages/relations and owning canonical resolvers to produce the /wiki public page projection. #44 consumes the typed source/public identity contracts for downstream search, graph, cross-source link projection, query behavior, and parity. Neither downstream issue may move source parsing or token semantics back into its projection layer.

[alexeygrigorev]

Grooming Complete — source-only Podwiki normalization

Split from #43 after auditing current origin/main face8e4808d65afbf0374d1ced7a88079950d663, the exact local immutable Podwiki source pin/tree, specs 01/02/03, and closed decision #24.

This child is engineer-ready as an independent network-free lane. It owns exact source parsing and typed route-neutral records only: 282 pages, 18,434 ordered chip occurrences (including 2,622 timestamped citations), and validated route-neutral graph/search seeds from DataTalksClub/podwiki@988b79d tree 60e8821. It creates no route, public projection, search ranking/index, graph projection, activation, database write, GitHub write, provider call, or browser change.

#43 consumes the accepted typed pages and canonical resolvers for /wiki; #44 remains downstream owner of search/graph/query/parity. Screenshots are not_applicable only if the independent change graph proves public artifacts and rendering byte/semantically unchanged.

[alexeygrigorev]

Software Engineer Report

Verdict

Implementation is complete and frozen uncommitted, but it is not tester-ready yet because the graph-required Playwright smoke could not take an exclusive browser lease and current origin/main has three independently reproduced non-#294 gate failures. I did not overlap or terminate foreign browser processes.

Frozen candidate

• Issue: Normalize the pinned Podwiki source into typed route-neutral records #294
• Worktree: /home/alexey/git/dtc-website/.tmp/issue-294-engineer
• Branch: issue-294-engineer
• Base / observed HEAD / origin/main: face8e4808d65afbf0374d1ced7a88079950d663
• Status: four intent-to-add paths, all content unstaged and uncommitted
• Patch SHA-256: b07153fdd9e4f0fa55831959013e2f6d2a36df43ffed21f964fa741d6aac1014
• Changed files (1,831 insertions):
  • content_sync/podwiki/__init__.py
  • content_sync/podwiki/adapter.py
  • content_sync/podwiki/contract.py
  • content_sync/tests/test_podwiki_adapter.py

Implemented behavior

• Adds one pure, network-free adapter bound to DataTalksClub/podwiki@988b79d0d655bf4755945c3118544cb9e0dbead6, tree 60e8821fa16af4688e782759a4a65eb1e36f9f85, and selected-input SHA-256 e7645850d748e369a21e8fdf809c9527a2fd4856c9aa61aff355fa19a6d1c39c.
• Selects only 282 _wiki/*.md pages, graph/graph.json, search/search-corpus.json, and assets/og-default.png; source scripts, Jekyll layouts, JavaScript, WASM/plugins, and other assets are never read or executed.
• Emits versioned, route-neutral typed pages; bounded Markdown AST blocks; unique headings/fragments; typed related values; every ordered chip occurrence with stable occurrence identity, source pointer, relation context, opaque target key, visible alias, and citation episode/timestamp.
• Preserves typed graph/search source seeds, their original source URL field as source evidence (never as identity), graph generator/count fields, git-date evidence digest, input digests/counts, and one canonical output digest.
• Exact accepted result: 282 pages; 18,434 occurrences (wiki 9,008, citation 8,571, person 622, podcast 153, book 80); 2,622 timestamped citations; 1,072 graph nodes; 13,006 graph links; 2,998 search documents. Canonical payload digest: 0dbfb4794bdbe2f25f26a8f292c7830b2e9e8356f4c7e2349a1dd7c247d5524f.
• Rejects identity/pin/tree drift, selected-input drift, unsupported selected files/plugins, symlinks/traversal/file kinds, count/byte/time excess, unsafe/ambiguous YAML, malformed metadata/chips/timestamps/headings, unsafe Markdown/HTML/URLs, invalid graph/search schemas or references, and missing/inconsistent typed git-date evidence with bounded content-free diagnostics.
• Adds no route, redirect, template, CSS, JavaScript, checked projection, model/migration/write, job, provider, GitHub client/write, public JSON, search ranking/index, graph projection, or activation behavior.

Change-selective plan

• Plan: .tmp/verification-294-final/verification-plan.json
• Plan file SHA-256: 62e69b4ae28750c7972960befd960238fd2b5c34ce61a09cdf8ec08c54834d76
• Graph SHA-256: 05ffeee99244c9becd5e98e40be08188f1c5e2a61204daece984c3beb1d1bce6
• Source-tree manifest SHA-256: bbb64743009d144dd04b6554e7065cb9f6b88ab2ac2e5e0fd6a62cb03037d457
• Worktree tree OID: c8496fb2a37529eb7c63e1eeb9e35e29a2aadada
• Direct node: shared.content_sync; no downstream or unknown nodes.
• Risk flags: shared_runtime, test_infrastructure.
• Render decision: impact=false, no routes/captures; render digest 708da6bda70a0814bdfb3d18b83c292fb15ee295ffd62b047803704c03c71f9b.

Four disposition buckets:

• rerun: selector, quality, Django full, Playwright smoke, container, evidence validation — fresh full regression selected because a new test path carries test_infrastructure and the adapter is under shared runtime.
• reused: none.
• skipped: none.
• not_applicable: compatibility (outside_impact_closure), content invariants (outside_large_content_closure), screenshots (render_inputs_unchanged_without_reusable_visual).

No reuse provenance or expiry applies.

Passing evidence on the final source

• PODWIKI_CHECKOUT=... uv run pytest content_sync/tests/test_podwiki_adapter.py -q — 9 passed, 18 subtests passed. This includes two exact full-corpus runs with reversed date-evidence ordering and a separately copied selected checkout; canonical bytes/digests match.
• PODWIKI_CHECKOUT=... uv run --frozen python manage.py test content_sync.tests.test_podwiki_adapter --noinput — 9 passed.
• PODWIKI_CHECKOUT=... uv run --frozen python manage.py test content_sync --noinput — 68 passed, 8 skipped.
• uv run ruff format --check content_sync/podwiki content_sync/tests/test_podwiki_adapter.py — pass.
• uv run ruff check content_sync/podwiki content_sync/tests/test_podwiki_adapter.py — pass.
• uv run mypy content_sync/podwiki content_sync/tests/test_podwiki_adapter.py — pass.
• make database-portability-check — pass.
• make security-check — pass, including bounded artifact/canary scan.
• make lint — pass.
• make migrations-check — pass, no changes detected.
• make django-check — pass; OpenAPI and management parity current.
• make deployment-check — pass.
• make test-ci — 607 passed.
• make verification-container VERIFY_CONTAINER_OUTPUT=.tmp/verification-294-final/evidence/container-check.json — pass, 13 assertions. Artifact SHA-256 b3954adbc0731524e6d993d4c23d25d208c97a1d256118e79be796b7bb1f8647; runtime result digest f5eed5e66fba2a82191e477b252c8731b9c5980c4e275fa0e0905f12324bf516.
• git diff --check — pass.

The security/no-side-effect test fails the adapter if it opens a socket, launches a process/source repository tool, or executes a database query, and verifies byte-identical website/urls.py plus all three checked Wiki public projections before/after adaptation.

Pending and inherited failures

• make test-playwright-smoke is pending, not skipped. The exact process census found active foreign browser ownership, including website .tmp/user-changes-20260829 make test-playwright PID 979256 and AI Shipping Labs issue-1501 make test-playwright-core PID 1560100, with live driver/Chromium descendants. Per the exclusive lease rule I neither ran nor killed/overlapped anything and did not poll those runs.
• make test-django-full discovers 2,562 tests, then aborts on inherited content.tests.test_collection_hub.CollectionHubPaginationTests.test_controls_are_one_labelled_landmark_with_one_current_page: the base assertion expects the old class token sequence without interactive-lift; Django's parallel runner then hits its existing unpicklable-traceback error. No Normalize the pinned Podwiki source into typed route-neutral records #294 file participates.
• make verification-quality stops at inherited frozen-projection digest drift for content/public_projection/{podcasts,wiki,wiki_search}.json.
• Full make format-check reports six inherited files outside the Normalize the pinned Podwiki source into typed route-neutral records #294 diff; full make typecheck reports four inherited errors in content/event_speaker_bio_normalization.py. Candidate-scoped format/lint/type checks are green.
• Because required Playwright/Django/quality components are unresolved, make verification-run was not started and no aggregate verification report was fabricated. Independent tester verification and PM acceptance remain pending.

Screenshot disposition and limitations

Screenshots are not_applicable by the final graph: the four changed paths cannot reach a route, public artifact, template, CSS, JavaScript, or rendered state. This is an engineering disposition only; the independent tester must recompute the frozen graph.

The exact accepted-corpus test requires a tester-supplied immutable PODWIKI_CHECKOUT and passes _data/wiki_dates.yml values into the adapter as typed caller evidence; the adapter itself does not select/read that file or consult Git/network. The candidate remains frozen without commit, push, merge, issue acceptance, database/public mutation, or provider/production action.

[alexeygrigorev]

PM lifecycle review — HOLD before tester

Verdict

HOLD at the engineering gate. Do not advance #294 to independent testing, product acceptance, commit, merge, or push. This is not a rejection of the proposed product behavior; the required verification envelope is incomplete, and inherited failures do not waive a required gate under _docs/PROCESS.md.

Frozen-candidate audit

I independently re-read the groomed issue, engineer report, process rules, PM role contract, frozen worktree metadata, change-selective plan, and available container artifact without editing or testing the candidate.

The current filesystem evidence still matches the engineer handoff:

• worktree /home/alexey/git/dtc-website/.tmp/issue-294-engineer, branch issue-294-engineer;
• base and observed HEAD face8e4808d65afbf0374d1ced7a88079950d663;
• exactly four intent-to-add, uncommitted paths under content_sync/podwiki/ and content_sync/tests/test_podwiki_adapter.py;
• 1,831 insertions and no additional changed path;
• git diff --check passes;
• binary patch SHA-256 remains b07153fdd9e4f0fa55831959013e2f6d2a36df43ffed21f964fa741d6aac1014;
• plan SHA-256 remains 62e69b4ae28750c7972960befd960238fd2b5c34ce61a09cdf8ec08c54834d76;
• plan source-tree OID c8496fb2a37529eb7c63e1eeb9e35e29a2aadada, source-tree manifest bbb64743009d144dd04b6554e7065cb9f6b88ab2ac2e5e0fd6a62cb03037d457, and graph digest 05ffeee99244c9becd5e98e40be08188f1c5e2a61204daece984c3beb1d1bce6 match the report;
• the available container artifact SHA-256 remains b3954adbc0731524e6d993d4c23d25d208c97a1d256118e79be796b7bb1f8647 and records 13 passing assertions.

The plan classifies only shared.content_sync, with no downstream or unknown nodes, and carries shared_runtime plus test_infrastructure risk. That supports the engineer's source-only scope statement, but it does not replace the required fresh gates.

Why the gate cannot advance

The plan requires fresh selector, quality, full Django, Playwright smoke, container, and evidence-validation components. The handoff still has all of the following unresolved:

• Playwright smoke is pending, not passed. Deferring because another owner held the exclusive browser lease was correct; it is still missing evidence.
• Full Django is red on the independently identified collection-pagination/base failure.
• Quality is red on frozen public-projection drift.
• Full repository format/type gates remain red on the reported out-of-scope base files.
• No aggregate verification-run report exists because required inputs are unresolved.
• No independent tester has recomputed the plan, validated the exact Podwiki pin/census and security boundaries, or issued a tester-final PASS.

The focused adapter, content-sync, CI, security, portability, migration, deployment, and container results are valuable engineering evidence. They cannot establish the broader required verdict while selected components are failed or pending.

Screenshot disposition

The engineer plan records render impact false, no routes/captures, and render digest 708da6bda70a0814bdfb3d18b83c292fb15ee295ffd62b047803704c03c71f9b. Therefore screenshots are provisionally not_applicable for this source-only candidate; there is no justification for unrelated product screenshots. That disposition becomes final only if the independent tester recomputes the frozen graph and confirms no public artifact, route, template, CSS, JavaScript, or rendered behavior entered the closure.

Recovery and handoff order

1. Preserve this candidate frozen and uncommitted; do not overlap or terminate the foreign browser owner.
2. Let the release-recovery lane repair the main-base causes behind the quality/Django/format/type failures in its established dependency order. Browser availability resumes only on its owner's completion event, not polling.
3. Reconstruct or rebase Normalize the pinned Podwiki source into typed route-neutral records #294 onto the resulting current green base, then regenerate the source tree, graph, plan, render decision, and all digests. The current plan/evidence must not be carried across a base change as if still current.
4. The engineer completes every graph-required fresh rerun, including exclusive Playwright smoke, full Django, quality, container, evidence validation, and the aggregate verification report. Any remaining red or pending component keeps the issue at engineering.
5. Only then may a separate tester recompute the plan from the new frozen base/head, verify every acceptance criterion and exact source/census/security contract, confirm the screenshot N/A classification, and post tester-final PASS.
6. PM product acceptance follows that complete tester report. Until then, keep every acceptance checkbox and the issue open; no commit, merge, push, or deployment is authorized.