Decision: Use PostgreSQL search while preserving public search contracts

[alexeygrigorev]

Normative decision: Open decision 13

Decision needed

Confirm PostgreSQL search for main/docs/FAQ/Podwiki MVP while preserving exact FAQ JSON and Podwiki query/filter/graph contracts; retire the current search Lambda only after relevance parity.

Non-goals

Do not redesign search UX or remove compatibility endpoints during initial migration.

Acceptance criteria

• Search index unit, ranking fixtures, language/tokenization, and fallback behavior are accepted.
• Existing query/filter/response contracts and zero-result behavior are inventoried.
• Legacy search retirement has measured parity and rollback gates.

Validation scenarios

Compare representative exact, partial, heading, FAQ, Podwiki type-filter, Unicode, empty, and malicious queries against the accepted baseline.

Dependencies

Blocks content search implementation under #4.

[alexeygrigorev]

PM decision brief — search contract

Approve a backend-portable PostgreSQL search projection for main site, docs, FAQ, and Podwiki while preserving exact FAQ JSON and Podwiki query/filter/graph contracts. Keep the current search Lambda until measured relevance parity and rollback evidence pass; do not redesign the public search UX during migration.

The owner response should confirm the accepted ranking/index fixtures, language/tokenization behavior, fallback/zero-result behavior, malicious-query handling, compatibility response inventory, parity threshold, and rollback trigger.

Reply "ACCEPT: contract-preserving search migration", "ACCEPT WITH CHANGE: ...", or "REJECT: ...".

[alexeygrigorev]

Read-only grooming/decision audit — 2026-08-24

Verdict: OPEN / not implementation-ready. Audited clean origin/main at 1db4882. The shared checkout is dirty on unrelated #216 work; it was neither used as evidence nor changed. #24 has one PM brief, no owner ACCEPT/ACCEPT WITH CHANGE/REJECT, and all acceptance boxes remain unchecked.

Exact owner decision still required: ACCEPT WITH CHANGE: use a PostgreSQL-backed search projection for deployed Django, while keeping document/index semantics backend-portable and SQLite-testable; make /wiki the sole canonical public route, keep /podwiki* as an unredirected 404 per spec 02, preserve the FAQ JSON and the existing Podwiki q/level/document_type/graph contracts under the canonical Wiki surface, and retain the current Lambda as fallback/rollback until an owner-approved relevance threshold passes. No PostgreSQL-only application contract, UX redesign, or Lambda retirement in this phase. The owner must additionally freeze the ranking metric/threshold, tokenization/language rules, malicious-query policy, fallback mode, and rollback trigger. This reconciles spec 01, spec 02, spec 03, and open decision 13.

Compatibility matrix

Surface	Contract to freeze	Current evidence / gap
Unified index	Main, docs, FAQ, and Podwiki; headings/sections where source behavior depends on them; deterministic release-bound projection	Django has only a baked 2,998-document corpus. ContentDocument.normalized_text and nullable search_build_id are storage seams, not a search index.
FAQ	/faq/json/courses.json and /faq/json/<course>.json; exact fields, stable 10-character IDs, raw Markdown answers, separate from search	Checked target is 6 courses / 70 sections / 1,401 questions; tests prove JSON shape, but search-inclusion parity is unproven.
Wiki query/route	/wiki canonical; trimmed/capped q, empty/zero-result, Unicode, escaping, and result limit; /podwiki* 404	Current Django case-folds/splits and scans substring terms, caps at 100, has no ranking/tokenization, and ignores the companion filter. It restricts results to /wiki/, dropping 742 podcast/person/book registry documents from the 2,998-document corpus. Reader
Filters / Lambda	Preserve levels wiki, guide, comparison, roadmap, transition, how_to, podcast_summary, person, book, section, `document_type=page	section, missing-q` 400, response shape, and filtered results
Graph	Stable /wiki/graph/graph.json schema, IDs, links, counts, and hash/deep-link behavior	Static checks exist (1,072 nodes / 13,006 links); candidate search/graph activation and rollback evidence do not.
Safety / lifecycle	Query surfaces private, no-store; candidate builds precede activation; failure leaves source pages and the prior projection available	Normative in specs, with no interrupted/concurrent rebuild or rollback report.

Downstream blockers: #40 is OPEN with the canonical Person boundary unresolved and a #39↔#40 dependency cycle. #43 is OPEN but still requires the retired /podwiki/ family, contradicting spec 02 and current unredirected-404 tests. #44 is OPEN, depends on #24 and #37–#43, while #43 depends on #44’s shared projection: a #43↔#44 cycle. Re-groom/split these issues and reconcile the route decision before assigning engineering.

Acceptance evidence required before dispatch / retirement planning

1. Pinned-source golden request/response fixtures for exact, partial, title/heading/section, FAQ, every filter, Unicode, empty, zero-result, overlong, encoded, and malicious queries: exact status, body/schema, URLs, escaping, and cache headers.
2. Search-document/index unit tests, deterministic ranking/tie-break fixtures, tokenization/language tests, and replay against both the current Lambda and browser fallback. Publish the owner-approved metric and threshold; unexplained contract differences fail.
3. Candidate build/activation tests proving deterministic digests, no partial publication, prior-projection fallback, interrupted/concurrent rebuild safety, content-page availability during search failure, and explicit rollback.
4. Redaction/no-store evidence plus focused Django tests and make test-playwright-core with desktop/mobile screenshots if the UI changes. Independent tester PASS and PM acceptance are required by _docs/PROCESS.md before any commit or Lambda retirement.