Build the legacy URL, link, fragment, asset, and SEO manifest crawler

[alexeygrigorev]

Parent epic: #3

Normative spec: 02 — URL/link/SEO compatibility

Scope

Create a deterministic, versioned inventory tool that combines committed generated sites and a production crawl for main/docs/FAQ/Podwiki/current course platform. Record source path, public/status/redirect/canonical, metadata, headings, language/robots, structured data, fragments, links/forms/assets, main-content fingerprint, and sitemap/lastmod state. Classify each row preserve/redirect/retire with reviewable exceptions.

Non-goals

Do not implement Django replacement pages, modify production, follow unsafe arbitrary networks, or auto-approve differences.

Acceptance criteria

• Inventory covers all generated/public paths and machine contracts in the spec, including query/filter/fragment samples and course HTML/API routes.
• Crawl is bounded, repeatable, respectful, records provenance/timestamp/tool version, and redacts credentials/PII.
• Source-vs-production differences are surfaced rather than overwritten.
• Classification rejects homepage catch-alls and requires owner/reason/test for redirect or 410.
• Manifest format is stable, diffable, schema-validated, and consumable by later tests/Lambda generation.

Test scenarios

1. Fixture site covers redirect chain/loop, soft 404, Unicode/percent case, trailing slash, relative links, fragments, JSON, forms, assets, canonicals, JSON-LD, robots/sitemap, and external query strings.
2. Interrupted/repeated crawl is deterministic and cannot escape allowlisted hosts.
3. Removing a known route/fragment/asset or changing a canonical produces an actionable diff.

No product Playwright UI is added; use browser/network crawl tooling only where server-rendered output requires it.

Dependencies

Depends on #1. #16 is not a prerequisite for baseline capture; it supplies consumer owner/auth/volume/migration inputs required before approving legacy-course-host redirects or Lambda activation. Blocks all parity adapter exit gates.

[alexeygrigorev]

PM Readiness Clarification

Issue #34 may proceed after closed dependency #1. Issue #16 does not block baseline capture: it supplies consumer owner/auth/volume/migration evidence required later, before approving legacy-course-host redirects or Lambda activation. No scope or acceptance criterion changed.

[alexeygrigorev]

Independent tester gate: PASS.

Evidence:

• make lint, make format-check, and make typecheck: pass (178 formatted files; 120 typed source files).
• Migration drift, Django system, and production deployment checks: pass.
• make test: 204 compatibility tests pass; 1,041 Django tests pass with 7 skips.
• make test-playwright-core: 3 pass; 3 deployment-only scenarios skip as expected.
• make compatibility-artifacts-check: pass; checked manifest validates and the 4,918-row comparison regenerates byte-for-byte.
• make compatibility-source-artifacts-check: pass for all 6 source-derived artifacts.

Artifact audit:

• Schema v2, crawler tool v3, current crawl-policy fingerprint, five exact pinned source revisions, and explicit 2026-08-08T05:55:00Z capture timestamp all match current code/config.
• 2,965 unique manifest URLs: exact 2,937-source set, exact 2,965-production seed set, 28 production-only, no source-only/missing/duplicate rows.
• All 2,965 classifications remain preserve / proposed_preserve; no parity or exception was auto-approved.
• Production accounting agrees across manifest, ledger, tests, and README: 2,951 200; 4 401; 2 403; 7 404; 1 405; 2,976 responses; 258,215,736 bytes; 11 HTTP redirect chains; 2 client redirects; 9 soft 404s; 0 capture errors.
• Manifest and comparison regenerate byte-exact to their checked digests; the 4,918 differences and all kind counts match the unresolved ledger.
• Raw credential/PII pattern scan and strict model redaction validation pass.

All five acceptance criteria and all three groomed test scenarios are covered by focused tests. Screenshots are not applicable because this issue changes no product UI and explicitly adds no product Playwright UI.

[alexeygrigorev]

Product Acceptance

ACCEPT — the delivered baseline satisfies all five groomed acceptance criteria and all three test scenarios. Engineering may commit with Closes #34.

Acceptance evidence:

• Inventory coverage is exact and reviewable: 2,937 generated source paths, 50 machine-contract samples, 1,401 FAQ fragment contracts, 1,072 Podwiki graph fragment contracts, 89 adopted course route patterns, and 5,507 reconciled public contracts. The checked manifest contains the exact 2,937-source set and exact 2,965 anonymous-production seed set.
• The production crawl is bounded and respectful: two exact HTTPS hosts, public-address validation and connection pinning, robots checks, static anonymous GET-only requests, pacing/retry/response/aggregate/deadline limits, strict resumable checkpoints, provenance for five pinned revisions, explicit timestamp, tool version, and crawl-policy fingerprint. Sensitive URL values and query-shaped fragments are rejected or deterministically redacted.
• Source and production observations remain separate. The canonical comparison deterministically reports 4,918 actionable differences and regenerates byte-for-byte; none was overwritten or silently accepted.
• All 2,965 manifest classifications are preserve with review_state=proposed_preserve. No parity, redirect, or retirement was auto-approved. Model and manifest validation require explicit owner/reason/test evidence for exceptions, reject homepage catch-alls, require exactly one permanent redirect hop ending in a successful response, and require a real 410 for retirement.
• Manifest schema v2, public-contract schema, stable difference IDs/actions, canonical serialization, exact checked digests, and validate, compare, and compare-versions commands make the result suitable for later parity tests and explicit Lambda-map generation.

Independent PM checks passed:

• make compatibility-source-artifacts-check
• make compatibility-artifacts-check (2,965 valid rows; 4,918 byte-identical differences)
• uv run python scripts/build_legacy_manifest.py public-contracts --check (5,507 rows)
• make test-compatibility (204 passed)

The fixture/unit coverage exercises redirect chains and loops, soft 404s, Unicode and exact percent encoding, trailing/relative paths, fragments, forms, assets, canonical and structured metadata, JSON, robots/sitemap, external queries, deterministic interruption/resume, allowlist confinement, and actionable route/fragment/asset/canonical diffs.

Deliberate residual work is honestly documented and does not block this inventory-tool issue:

• All 4,918 source/production differences remain unresolved review inputs, including four removed routes, three canonical changes, nine soft-404 observations, and the recorded status/asset/fragment/content-field differences.
• The 68 parameterized/authenticated course examples were not production-probed; no credentials or learner/operator data were used. Issue Decision: Inventory course API consumers before legacy-host redirects #16 supplies the consumer/auth/volume inputs needed before course-host redirect approval or activation.
• The ledger retains the pinned docs tool-version choice, frozen FAQ generation time, and one malformed legacy meta browser-fingerprint exception.

This acceptance approves the crawler, inventories, and evidence only. It does not approve SEO cutover, authenticated probes, any redirect/retirement, the future course Lambda, or production changes. No product UI changed, so screenshots and product Playwright pages are not applicable; the independent tester separately passed the repository core Playwright gate.