Creative Ingredients Catalog: scraps, ingredients, pgvector sync, bible backfill#542
Merged
Merged
Conversation
…e backfill
Postgres-backed catalog of canon "ingredients" (characters/places/objects)
with pgvector embeddings, LLM-driven extraction from raw scraps, and a
per-table sync_sequence federation gate (separate from the main pipeline
sync) so a peer ahead on the catalog schema cannot corrupt an older peer.
- DB: catalog_ingredients/_scraps/_ingredient_sources/_ingredient_refs
with HNSW vector indexes and per-row sync_sequence triggers; schema
gate added to PORTOS_SCHEMA_VERSIONS as `catalog`.
- Bible backfill (server/scripts/migrateBibleToCatalog.js) promotes
embedded universe canon into the catalog using SHA-256-derived
deterministic ids over (universeId, kind, entry.id) so peers running
the migration independently converge on the same row instead of
minting divergent UUIDs and orphaning one side under LWW. Soft-deleted
rows revive in place. updateUniverse gains { silent: true } so the
boot walk doesn't fan out a recordUpdated per universe.
- Sync envelope is size-capped to match create-side limits (rawText
2MB, payload 200KB, name/tag/ref_id caps, array lengths, portosMeta
4KB). /sync/apply rejects ahead-of-version peers with 412 and reports
per-row {inserted/updated/skipped/failed} + errors[] instead of
aborting the batch on one malformed row.
- /sync accepts scalar OR per-kind cursors and rejects arrays; per-kind
is needed because the four sync_sequence columns advance independently.
- Extraction neutralizes triple-backtick fences in user-pasted scraps
(zero-width joiner between the ticks) so a scrap containing a code
fence can't close the LLM prompt's own fence.
- listIngredients ORDER BY now references the user-query parameter by
captured index instead of $1; type/tag filters no longer rank the
filter literal. New staleEmbeddingModel filter + /embeddings/backfill
{ includeStale: true } so a provider/model switch can re-embed rows.
- /ingredients/:id strips the 768-float embedding unless
?includeEmbedding=true. Detail/list pages use inline Delete? Yes/No
rows instead of armed-delete.
- CatalogIngest commit no longer drops every extracted field: payload is
built from the flat bible-shaped draft minus control keys, with the
description rubber-band fixed.
- Storage barrels + READMEs updated for catalogDB, catalogSync,
catalogValidation, catalogExtraction, catalogEvents, embeddings, and
apiCatalog so future agents can find them via the catalog rule.
… field per kind, validate cursor-object values - Drop stray tryReadFile mock that polluted the wardrobe-cap fixture in storyBible.test.js (copy-paste artifact). - CatalogIngest review textarea now binds to physicalDescription for characters (not description) so the LLM-extracted prose renders and user edits land on the canonical field. - normalizeCursors() now regex-validates per-kind cursor values; a non-numeric since[scraps]=abc would have surfaced as a Postgres 500. - updateScrap / updateIngredient now WHERE id = AND deleted = false so PATCH on a soft-deleted row 404s instead of silently mutating. - Catalog delete-failure path restores the row at its original index, not the top of the list. - Rename shadowed q binding in /api/catalog/ingredients to params. - Fix misleading dot-notation example in catalogSync docstring; routes/catalog.js already had the correct bracket form.
… simple parser, always run idempotent catalog DDL - Express 5 defaults query parser to 'simple' (Node's querystring), which leaves '?since[scraps]=10' as a flat 'since[scraps]' key instead of a nested object. The /api/catalog/sync route now reconstructs the per-kind object from the flat keys itself, so the documented bracket protocol survives regardless of parser config — without this, peers would keep pulling the first page and loop on hasMore. - server/lib/db.js ensureSchema no longer early-returns when all four catalog tables exist. The block is fully idempotent (CREATE IF NOT EXISTS / CREATE OR REPLACE FUNCTION / DROP TRIGGER IF EXISTS + CREATE TRIGGER), and the previous probe could skip indexes/triggers if a prior boot crashed between table-create and artifact-create — leaving the schema marked ready with the update triggers and HNSW indexes never installed. - PLAN.md: tracked [catalog-ref-deletion-tombstones] — catalog_ingredient_refs uses hard DELETE so peer deletions never propagate. Architectural follow-up (needs soft-delete schema migration); deferred from this PR.
…atalog list snippet falls back to it, cursor cope with repeated params - CatalogIngredient: PAYLOAD_FIELDS.character now lists physicalDescription + background (the canon shape from sanitizeCharacter). The bare 'description' was rendering empty for every bible-backfilled character and edits there were creating a sibling key the canon doesn't read. - Catalog list snippet now checks payload.physicalDescription before payload.description so character rows show their narrative blurb. - /api/catalog/sync cursor reconstruction takes the last value when Node's simple parser collapses repeated keys into an array, instead of dropping the cursor and falling back to '0'. - PLAN.md: tracked three deferred items — search_tsv missing character depth fields (needs schema migration + version bump), scrap commit non-transactional (sizable refactor — catalogDB doesn't accept a client), and scrap embeddings written but never read (decide: remove the call or ship the search endpoint).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Postgres + pgvector catalog of canon "ingredients" (characters / places / objects), an LLM-driven extraction flow from raw scraps, and a dedicated per-table-cursor federation sync (separate from the main pipeline sync) so a peer ahead on the catalog schema cannot corrupt an older peer.
Highlights
catalog_ingredients(768-dim HNSW vector),catalog_scraps,catalog_ingredient_sources,catalog_ingredient_refs. Per-rowsync_sequencetriggers on every write path (including the new source-span UPDATE trigger).server/scripts/migrateBibleToCatalog.js) promotes embedded universe canon into the catalog. IDs are SHA-256-derived over(universeId, kind, entry.id)so two peers running the migration independently converge on the same row instead of minting divergent UUIDs and orphaning one side under LWW. Soft-deleted rows revive in place.updateUniversegains{ silent: true }so the boot walk doesn't fan out arecordUpdatedper universe.PORTOS_SCHEMA_VERSIONS.catalog./api/catalog/sync/applyrejects peers ahead oncatalogwith HTTP 412./synccursors are per-kind (?since[scraps]=A&since[ingredients]=B&…) because the foursync_sequencecolumns advance independently. Scalar form (?since=N) still works for the first-pull case; array form is rejected.rawText2MB,payload200KB,name/tag/ref_idcaps, array lengths,portosMeta4KB) — mirrors the create-side limits because a peer running an older PortOS could skip its own validation.applyRemoteChangeswraps each upsert in try/catch and returns per-kind{inserted, updated, skipped, failed}+errors[]. One malformed row no longer poisons the batch.listIngredientsORDER BY now references the user-query parameter by captured index instead of$1; type/tag filters no longer rank against the filter literal. NewstaleEmbeddingModelfilter +/embeddings/backfill { includeStale: true }so a provider/model switch can re-embed model-mismatched rows./ingredients/:idstrips the 768-float embedding unless?includeEmbedding=true. Detail/list pages use inline "Delete? Yes/No" rows.CatalogIngestcommit no longer drops every extracted field: payload is built from the flat bible-shaped draft minus control keys, with the description rubber-band fixed.catalogDB,catalogSync,catalogValidation,catalogExtraction,catalogEvents,embeddings,apiCatalog).Test plan
cd server && npm test— 8204 / 8204 pass (370 files, 7 skipped, 0 failed)cd client && npm test— 634 / 634 pass (64 files)ensureSchema()end-to-end, creates all four catalog tables + HNSW indexes, then runs the bible backfill exactly once (marker file gates subsequent boots).ingredientId.?includeEmbedding=truereturns the 768-float vector; default response strips it.catalog=1, peer B fresh install — verify/sync/applyrejects with 412 when peer A is ahead.Plan reference
Design doc:
docs/plans/2026-05-29-creative-ingredients-catalog.md(archived at approval).