Skip to content

Lorebase 0.3.0

Choose a tag to compare

@github-actions github-actions released this 30 Oct 18:03
· 17 commits to main since this release

Knowledge that stays correct: a reconciliation queue for contradictions between Sources, corrections that survive Source syncs, document versions, ingestion modes and stewards, Traits and lenses in search and context packs, the merge of text values, and the resolution of words to schema and values with consolidation from use and coverage signals; the web app gains a Reconciliation screen and a passage-by-passage Conflict resolver. Pre-1.0: this minor version changes interfaces; read the breaking changes before upgrading.

Ten migrations (0038 to 0047). New configuration sections keeper.reconciliation, keeper.stewards, scribe.reconciliation, scribe.consolidation, store.values, store.resolve_misses, store.coverage, search.resolve and read.coverage, and new keys store.resolve_misses.caller_key, store.storage.hnsw_m, store.storage.hnsw_ef_construction, search.semantic.iterative_scan and search.semantic.max_scan_tuples. New endpoints POST /domains/{d}/resolve and /domains/{d}/reconciliation; responses add coverage to POST /query and POST /resolve, semantic_truncated to POST /search, can.settings to GET /domains/{d}, and source_key, version, replaces and status to documents. A Source sync no longer overwrites a value someone corrected on the target, and automatic merges into a protected branch wait for steward and role approvals (below, Breaking changes). The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Breaking changes

  • Source syncs no longer put back a value someone corrected on the target: an auto_when_clean Source that merged silently stops at an open Proposal with a DV1 or DV2 Conflict when upstream changes such a value, and on its first run after migration 0040. Set local_edits: source_wins on the Source to keep the old behavior (below, Changed and Upgrade notes).
  • Source syncs with auto_when_clean or auto_with_authority and extraction in direct mode no longer merge into a protected branch whose rules include required_approvals_from_stewards or required_roles; the Proposal stays open (merge_blocked: approvals, direct_blocked: approvals). Installations without those rules see no change (below, Changed).
  • A text Attribute edited in different sentences on two Branches merges automatically (resolver policy:text_merge) instead of raising a Conflict. Set text_merge: false in the Domain's merge_policy setting to keep the Conflict (below, Changed).
  • Semantic search reads the HNSW index at any size: unfiltered and broad-filter lists are approximate where they were exact (recall 0.92 to 0.99 at 300,000 vectors); a one-Kind filter stays exact (below, Changed).

Added

  • Resolution of words to schema and values (design 8.9, AC-39). POST /domains/{d}/resolve {text, kinds?, levels?, query_vector?, limit?, branch?, at?} returns ranked candidates the words of a text may stand for: Kinds, Attributes (by name or alias, with their type) and stored values with a ready-to-use where predicate and the count of Records holding them, each with its level (exact, synonym, trigram, embedding), score and the words matched, plus the words nothing matched. It never interprets the question, runs a query or calls a model; the embedding level only uses a caller's query_vector when no other level matched. Candidates never name a Kind or Attribute the caller may not read. Herald lorebase_resolve, CLI lorebase resolve TEXT, Python client records.resolve (sync and async). Settings search.resolve (agents).
  • Value index: mat_values holds per branch the distinct short values of selected Attributes, kept by triggers on mat_records in the writing transaction (migration 0045, additive). Enums are selected always, plain strings while their distinct values stay few (store.values); the Scribe value_index pass (scribe.scheduler.value_index_interval_s, default 300 s) and lorebase-scribe value-index DOMAIN [--rebuild] decide and backfill the selection. Sensitive and vault values are never indexed. A pass that changes the selection fences only its own Domain's writers (a transaction advisory lock taken shared by the triggers), for at most store.values.pass_timeout_ms; a schema change that makes an Attribute ineligible removes its values in the same commit. At 100,000 Records resolution answers in 11.2 ms at p95 (levels 1-2); the triggers cost about 90 µs per statement where nothing is held and 0.3 ms per updated Record whose held values change (performance, operations).
  • Value index (design 8.9): an Attribute's distinct-value limit now holds between value_index passes. The limit is max_values: hard_max_distinct for enums and resolvable: true, max_distinct × hysteresis for plain strings. The write that takes a branch past it marks the Attribute overflowed in the same transaction: resolution stops returning its values at once, later writes stop indexing it, and the next pass drops it. A schema commit or merge on the default branch applies a new limit in the same commit. Reads also skip a branch past its limit that is not yet marked.
  • Migration 0047 adds mat_value_sizes, the per-branch count of held values kept by the value triggers, and max_values and overflowed to mat_value_attributes. Passes read the sizes instead of counting mat_values. On upgrade, existing selections get hard_max_distinct and their sizes are counted; the first pass of each Domain counts its sizes again and sets its own limits. Trigger cost: about 90 µs per statement where nothing is held, as before; 0.3 to 0.4 ms per updated Record whose held values change, against 0.29 ms before (measured with tests/perf/resolve_bench.py on a loaded machine; performance).
  • Schema: aliases on Kinds and Attributes, value_aliases (stored value to other words) and resolvable: true|false on Attributes; absent by default, so existing schemas keep their hashes. Validation refuses resolvable and value_aliases on types other than string and enum (alone or in arrays), on sensitive and on link Attributes, and value aliases of an enum that name no value (schema.attribute_flag_invalid). The web Modeler keeps them on round trips.
  • Herald lorebase_query now documents counting with "aggregate": {"group_by": attr} and points to lorebase_resolve.
  • Consolidation from use (design 8.10, AC-40): with store.resolve_misses.enabled (off by default), Keeper keeps the phrases POST /resolve matched to nothing per Domain, branch and phrase (counts, first and last seen, digests of distinct callers; migration 0046), off the request path and only for head reads of callers holding search.resolve.misses.permission Domain-wide; requests opt out with record_misses: false, and searches narrowed by kinds or levels are not recorded. One caller adds and holds a bounded number of phrases; a full Domain evicts its least missed phrase. Phrases expire after retention_days (30) and go with their Domain, and so do the vocabulary suggestions quoting them (their share removed, a suggestion left without phrases deleted, decided or not). A Scribe vocabulary_consolidation pass (scribe.scheduler.consolidation_interval_s, CLI lorebase-scribe consolidate DOMAIN) turns frequent misses into schema suggestions of the new types value_alias, attribute_alias, kind_alias (near values and names; never for sensitive, vault, derived or non-resolvable Attributes, never a phrase that already names something) and, with scribe.consolidation.llm.enabled, attribute_description (task schema_consolidation; the model sees names and phrases only). They are reviewed like other suggestions; accepting one opens a schema change Proposal. Vocabulary suggestions have their own cap (schema.suggestions.max_vocabulary_tracked) so they cannot crowd out extraction's; they are withheld from reviewers who may not read their Kind's Records or Attribute's values; an alias that would name two things is refused at acceptance. lorebase-scribe misses purge DOMAIN deletes a Domain's misses and vocabulary suggestions.
  • Coverage signals (design 8.11, AC-41): POST /query and POST /resolve answer with a coverage block (coverage: false leaves it out; default read.coverage.enabled): the share of Records missing each Attribute used, Source freshness (last successful sync, stale_after exceeded, Records unseen), the phrases unknown in the Domain, and one note per signal. It reads per-Kind statistics of the default branch head (mat_coverage, migration 0046) counted by a Scribe coverage pass (scribe.scheduler.coverage_interval_s, CLI lorebase-scribe coverage DOMAIN) from one snapshot per Kind, Records unseen by Sources recounted every pass; the last successful run of a Source is one probe of the partial index ix_source_runs_last_succeeded; masked Attributes and Kinds outside the caller's scope are never described, and other branches and Revisions get no statistics. The two lookups take p95 3 ms at 100,000 Records and 600,000 Source runs.
  • Python client Coverage on QueryResult and ResolveResult, coverage on QueryRequest and ResolveRequest, record_misses on ResolveRequest, the vocabulary suggestion types and attribute, value, description on SchemaSuggestion; CLI lorebase query and lorebase resolve print the coverage notes; Herald lorebase_query and lorebase_resolve end with coverage: lines.
  • Web app, Review > Schema suggestions: vocabulary suggestions with the alias, the Attribute, the value they stand for or the proposed description, and the missed phrases as evidence; their Documents cell stays empty, as they come from no document.
  • store.storage.hnsw_m (default null: pgvector's 16) and store.storage.hnsw_ef_construction (default 128; pgvector's own default is 64) set the build parameters of the HNSW index of embeddings, for a new installation (migration 0001) and for indexes storage apply builds later. storage apply does not rebuild an existing index: installations from before this release keep ef_construction 64 until they rebuild it with ALTER INDEX ... SET and REINDEX INDEX CONCURRENTLY (operations). Measured at 300,000 vectors, ef_construction 128 raises mean recall@20 from 0.910 to 0.988 unfiltered and from 0.964 to 0.989 with a 40% filter, same index size, build 66 s instead of 50 s. Documentation now says the semantic list is approximate and gives these figures (search).
  • Incremental Sources and upstream deletions (design 9.1). Each run of an incremental Source records the Records it read, changed or not (source_written.last_seen_run and last_seen_at, migration 0043, additive). The new Source setting stale_after (ISO 8601 duration, default null: off; incremental Sources only) makes the reconciliation sweep raise a stale candidate with signal source_unseen for each Record the Source owns that no run has read for longer; nothing is deleted. Such a candidate takes keep_current or a dismissal (another choice answers 422) and goes obsolete once a run sees the Record again (source_seen). scribe.reconciliation.stale.source_unseen (default true, Domain override allowed) turns the check off (Scribe).
  • Web app, Modeler: the Source editor sets stale_after; the Reconciliation screen explains the source_unseen signal.
  • Document versions and lifecycle (design 9.2, AC-35). An upload may name a source_key (multipart field or JSON member; at most keeper.documents.max_source_key_chars, 1024): the bytes of the key's active version return it (200), other bytes become version n + 1 with replaces the previous one, now superseded, audited document.version. Keyless uploads keep their content addressing. GET .../documents?source_key= and GET .../documents/{id}/versions list versions; documents show source_key, version, replaces and status (active, superseded, retired, purged), and a document's details open_stale_candidates, can_retire and can_purge. Migration 0041, metadata only (operations).
  • The document_version_check job: for a new version, each value whose current provenance is extraction from an earlier version, and that the new version does not propose again, has its old evidence quote looked for verbatim (white space aside) in the new text; a gone or purged quote raises a stale reconciliation candidate with signal evidence_gone ({document, from_version, to_version, quote}), masked like evidence for sensitive Attributes. Nothing is retracted and no model is called. Bounded by scribe.extraction.versions.max_values_per_job (5000) with continuations; a Domain turns it off with reconciliation.triggers.document_versions: false (extraction).
  • Retirement and purge of documents: POST .../documents/{id}/retirement {reason} (new permission document.retire, held by maintainer and admin) runs the check with empty text and stops extraction for good (409 document-retired when an upload asks to extract it; extraction jobs naming it fail). POST .../documents/{id}/purge {reason, confirm} (new permission document.purge, admin only; confirm is the document id) deletes the bytes, parsed text, chunks and cached answers of the document and replaces its evidence quotes in Proposal items, candidates and schema suggestions by [erased], keeping provenance rows; irreversible, audited document.purge in the same transaction (operations).
  • Python client client.documents.versions|iterate_versions|retire|purge and source_key on upload, upload_url and list; CLI lorebase documents upload FILE --key KEY, lorebase documents list --key KEY, lorebase documents versions ID, lorebase documents retire ID --reason TEXT, lorebase documents purge ID --reason TEXT --confirm ID.
  • Web app, Documents: key, version and status in the library and on the document page, the versions of a document, its open stale candidates linked to the Reconciliation queue, a source key on upload, and Retire and Purge (typing the document id to confirm) for those who hold the permissions.
  • Read presets (design 8.8). POST /search and the search and query seeds of POST /context take traits: a Trait stands for the Kinds implementing it that the caller may read, and kinds with traits search the union. Herald's lorebase_search and lorebase_context take traits and lens, the CLI has lorebase search --trait and --lens and lorebase context --trait and --lens.
  • Lenses: lenses.{name} in the schema document is a named read preset (description, kinds, traits, relations, depth, include_schema, include_recent_changes, include_lineage, budget_tokens), a schema element with the id prefix lns_ that merges and is validated like the others (schema.lens_reference_unknown for a name the schema does not hold, schema.lens_invalid for a depth below 0 or a budget below 1). lens on POST /search and POST /context supplies the defaults: members of the request win, a lens over the caps is served at them (budget.capped), and the lens's Kinds and Traits limit neighbors and search and query seeds while refs seeds are always kept. A lens is not a permission: Kinds, Traits and Relations the caller may not read are dropped from it, and an unknown lens, or one whose Kinds are all unreadable, is 422 lens_unknown; with keeper.auth.authz.invisible_kinds, a Trait none of whose implementing Kinds is readable is 422 unknown_trait like an unknown one. GET /schema lists the lenses; Herald adds the resource lorebase://{domain}/lenses/{lens} and a LENSES part to lorebase_schema (agents).
  • Vector search that keeps its limit under filters: with pgvector 0.8 or later and search.semantic.iterative_scan: auto (default) the semantic list runs under hnsw.iterative_scan = relaxed_order bounded by search.semantic.max_scan_tuples (20000) and is ordered again by exact distance; otherwise the index returns search.candidates * search.semantic.overfetch_factor (4) nearest rows before the filters. The response says semantic_truncated when the scan bound left a list short while more Records match. /readyz reports the case in a vector_search check (search).
  • Python client: traits and lens on SearchRequest and ContextRequest, semantic_truncated on SearchResult, and the lenses section of schema.element.
  • Reconciliation queue (design 10.6): values on which Sources disagree no longer pass unnoticed when a one-sided write replaces another Source's value. Scribe raises candidates of type contradiction, superseded, stale and duplicate while extraction items are assembled, before a Source sync merges, and in the reconcile_sweep job (scribe.reconciliation.sweep_interval_s, off by default); nothing is changed, hidden or decided because a candidate exists. Stages run cheapest first: identity, values, normalization (units of a shipped table, numeric tolerance, case and punctuation, negation words) and an optional semantic stage (off by default) that marks paraphrases equivalent and compares stored Record vectors for duplicates. Candidates are fingerprinted by their values, so a dismissal holds until a new value appears (Scribe).
  • Reconciliation decisions in Keeper under /api/v1/domains/{d}/reconciliation: list with filters, get, POST .../{id}/resolution (take a claim, keep the current value, a custom value, or a policy authority, newest_wins, prefer_current, prefer_incoming; dry_run), POST .../{id}/dismissal, POST .../{id}/reopen, POST .../reconciliation:resolve in bulk (at most keeper.reconciliation.bulk_max) and POST .../reconciliation/sweeps (202). Values are written by the Kind's update_{kind} with a reconciliation provenance; when the branch needs review the write goes on a new reconcile/{candidate}-{random} branch and Proposal. Duplicates merge into a surviving Record through a Proposal. A candidate whose decision waits in an open Proposal refuses new decisions (409), and a claim or policy decision is refused when a newer value was written since detection. Decisions need conflict.resolve plus what the write needs, and clearance for sensitive or vault values, and are audited (reconciliation.resolve, reconciliation.dismiss, reconciliation.reopen).
  • Proposal check reconciliation: warn (default) lists the open candidates of a Proposal, block stops the merge while contradictions are open (409 reconciliation-open). Set by keeper.reconciliation.merge_check, scribe.reconciliation.merge_check for the merges Scribe runs, and the Domain setting reconciliation_check; the stricter one applies.
  • Valid time: an Attribute may declare valid_time: true (not on derived Attributes). Its values carry a validity period on their provenance, never in content: from Source mappings (attribute_mapping.{attribute}.valid_from, valid_to), from tier 2 extraction when the dates are in the quote, and from the valid parameter of update_{kind}. Disjoint periods make a superseded candidate instead of a contradiction.
  • Python client client.reconciliation (sync and async) and CLI lorebase reconcile list|show|resolve|dismiss|reopen.
  • Web app: a Reconciliation screen (/{domain}/reconcile) with a tab and count per type, filters by status, Kind, Attribute, source and signal, the claims of a candidate side by side with source, time, period and evidence, decisions with a preview and a required reason, duplicate merges, bulk decisions and sweeps. The Record page marks values with open candidates, the Proposal page links the candidates of its check, and the Source editor names the validity columns of valid_time Attributes. New settings VITE_RECONCILIATION_PAGE_SIZE and VITE_RECONCILIATION_COUNT_LIMIT (tour).
  • Metrics lorebase.scribe.reconciliation.candidates, lorebase.scribe.reconciliation.sweep.duration and lorebase.scribe.reconciliation.open (metrics).
  • Ingestion modes of extraction (design 9.2, AC-36): a Domain's settings.extraction chooses review_all (nothing accepted at once), doubts_only (the default, unchanged behaviour) or direct, with auto_accept_threshold, min_confidence and rules by source_key prefix (longest wins). Tightening always applies; loosening only for the members in scribe.extraction.domain_may_loosen (default [mode]), others are ignored and listed in ignored_settings of GET /domains/{d}. PATCH /domains/{d} takes settings.extraction (domain.settings, audited domain.update with before and after values). New settings scribe.extraction.mode (direct refused at startup), domain_may_loosen, direct_merge_protected (extraction).
  • direct ingestion: Scribe merges an extraction Proposal itself only when every item came from tier 0 and none is pending, the target accepts it and the merge is clean and not blocked by contradictions; any tier 1 or 2 item caps the job at doubts_only (mode_capped: tier). The job result says mode, direct_blocked (tier, pending_items, protected, reconciliation, conflicts), direct_merged and merged_revision; the merge is audited proposal.merge with mode: direct.
  • Stewards (design 10.7, AC-37): stewards: [role, ...] on domain and per Kind in the schema (an undefined role is the warning schema.steward_role_unknown, returned in warnings by schema:validate). Proposals and reconciliation candidates carry steward_roles and, for callers with principal.read, computed assignees (keeper.stewards.max_assignees_listed, 50); GET .../proposals and GET .../reconciliation take assigned=me|any|none. A principal who cannot read a Kind the work touches is never an assignee. Proposals record the Kinds they touch (proposals.kinds, migration 0042).
  • Protection rule required_approvals_from_stewards (keeper.proposals, Domain setting of the same name; default 0): approvals that must come from assignees, the Proposal needing the larger of it and required_approvals; too few assignees answers 409 stewards-unavailable. The approval block shows required_from_stewards, steward_approvals and stewards_unavailable.
  • Protection rule required_roles (keeper.proposals, Domain setting of the same name; default empty) (design 10.3): each listed RBAC role must have approved a Proposal into a protected branch, through a principal holding it who may review the Proposal. A merge missing one answers 409 approvals-required naming the roles; Proposal approvals show required_roles and missing_roles.
  • Python client assigned on proposals.list|iterate and reconciliation.list|iterate, steward fields on Proposals and candidates, settings on domains.update; CLI lorebase proposal list --mine and lorebase reconcile list --mine. Web app: an "Assigned to me" filter and the steward roles and assignees on Review and Reconciliation, the steward approvals on the Proposal page, and the ingestion mode and ignored settings on the Domain settings screen.
  • A purge no longer races a running extraction job of the same document: the purge and every write of the job (text, chunks, items, candidates, suggestions) lock the document row, the job stops (cancelled) when the document was retired or purged meanwhile, the purge cancels the queued and running extraction jobs of the document (jobs_cancelled in its audit entry), and a model answer is cached only while its chunk is stored.
  • GET /domains/{d} and PATCH /domains/{d} report can.settings: whether the caller holds domain.settings, computed by the authorizer (null in the Domain list).
  • Web app, Settings: the ingestion settings of a Domain (mode, auto-accept threshold, minimum confidence, rules by source key prefix) are editable for callers with domain.settings, with Keeper's validation errors at the fields they name and ignored values marked; others see the settings read only.
  • Web app: the Explorer's Kind filter offers the Traits of the schema, and a lens selector (Explorer search, and the Retrieval lab's searches and context pack) appears when the schema declares lenses.

Changed

  • Automatic merges no longer skip the rules that name who must approve (design 9.1). A Source sync with auto_when_clean or auto_with_authority and an extraction job in direct mode merged into a protected branch, with scribe.sources.auto_merge_protected and scribe.extraction.direct_merge_protected on (the default), whatever its approval rules said. Those settings still waive the plain required_approvals count, but when the target's effective rules include steward approvals (required_approvals_from_stewards above 0) or required roles (a non-empty required_roles), from keeper.proposals or the Domain setting of the same name, the merge is not performed and the Proposal stays open: the Source run's stats say merge_blocked: approvals, the extraction job's result direct_blocked: approvals, both with blocking_rules. The rules are evaluated by one implementation that Keeper and Scribe share (lorebase_store.approval_rules); Scribe reads keeper.proposals from the shared configuration. Installations without steward or role rules see no change (operations).
  • Semantic search reads the HNSW index at any size (design 8.8). The nearest vectors of the searched Kinds are taken from embeddings alone and joined to the Records, instead of joining first (which sorted every vector of the model): 25 ms against 329 ms p50 unfiltered at 300,000 vectors, 34.5 ms p95 against 513, inside the 200 ms budget. Unfiltered and broad-filter lists are approximate (HNSW recall 0.92 to 0.99 here) where they were exact; a one-Kind filter is exact. Over-fetch (iterative_scan: off) restricts to the Kinds' vectors before its limit.
  • Web app: a dialog opens with focus on its first control (a confirmation on its cancel button), or on its title when it has none, instead of on its Close button.
  • The include_schema, include_recent_changes and include_lineage members of a context request are unset by default (a lens may set them) and behave as true without one; the Python client's ContextRequest no longer sends them unless given.
  • Object type code 10 (schema.lens) joins the stored object types; no migration is needed.
  • Source syncs: corrections stick (design 9.1, AC-34). A Source now writes only the values that changed upstream since it last wrote them, kept per Source as hashes in source_written (migration 0040, advanced in the transaction that merges the Source's Proposal). A person's correction on the target is no longer put back by the next sync while upstream did not change; when upstream does change that value, the Source's Proposal holds a DV1 Conflict whose base is the Source's previous value, and a snapshot run that would delete a Record holding someone else's values holds a DV2, so an auto_when_clean Source waits for a person in those cases. The new Source setting local_edits chooses: review (default), keep (resolved at once for the person's value, resolver policy:local_edits) or source_wins. To get the old behavior, set local_edits: source_wins on the Source. Under auto_with_authority, Attributes whose authority is the Source are overwritten as before. Links a person removed or added are left until upstream changes them. A Record kept by a DV2 keeps its Links (the merge restores Links the deleting side removed with it), and a Record someone else deleted that comes back from upstream comes back with its Links. Under keep, each run also decides again the Conflicts a person reopened by editing a kept value while the Proposal was open. Each local edit is audited as source.local_edit_conflict, and run stats count local_edits and seeded. The first run of each existing Source seeds the state; a value it did not write that differs from upstream counts as a local edit that once (operations, Scribe).
  • Extraction: a value item over a value a person set last (an Operation, or a resolution or reconciliation decision by a principal) stays pending whatever its confidence, with hold reason local_edit (held_local_edit in the job stats) (extraction).
  • Web app, Modeler: the Source editor sets local_edits.
  • Merge: a text Attribute edited in different places on two Branches now merges automatically instead of being a Conflict. The values are compared by sentence (or paragraph) against the base; edits to different sentences are both kept, the result is recorded as an automatic resolution with resolver policy:text_merge, and the Review screen lists it under "Resolved by policy". Edits to the same sentence, or two insertions at the same place, are still a Conflict (neighbouring sentences rewritten on each side merge), which now carries details.text_hunks (the whole text in order, with the merged clean parts and the conflicting ones); its fingerprint is unchanged. Other types (string, uri, enums, numbers, json, lists) and values added on both sides are not affected. To get the old behavior set text_merge to false in the Domain's merge_policy setting, or only for a Kind, Relation or Attribute with text_merge_elements or text_merge_attributes; text_merge_granularity (sentence or paragraph) and text_merge_max_chars (characters over the three values, default 200000; fixed caps on the number of sentences and on the cost of repetitive text also apply) tune it (design 6.8, 6.10).
  • Web app, Conflict resolver: a text value that changed in the same sentences on both Branches (a DV1 with text_hunks) opens as passages: unchanged text folded away, each conflicting passage with Ours, Theirs and Base and a choice (ours, theirs, both in either order, base, or edit in place), a live preview with its length against the Attribute's limits, and "Use merged text" once every passage has a choice. [ and ] move between passages and 1 to 4 and 0 choose.
  • Web app, Conflict resolver: string values that moved from the base are shown word by word (added and removed words marked, with text for screen readers), and the Ours and Theirs columns carry the names of the target and source Branches. In the custom editor, "Copy base", "Copy ours" and "Copy theirs" load a side into the field. Saving a resolution opens the next open Conflict. Outside text fields o, t and b pick a side, e opens the custom editor, s or Enter saves and n moves to the next open Conflict. Leaving a Conflict with unsaved changes asks first, and "Apply resolutions" asks when Conflicts are still open or the one on screen is unsaved. The bulk choice uses the same names as the resolver ("Take ours", "Restore the Record") instead of the raw choice.
  • The release workflow can be run by hand (workflow_dispatch, input tag) on an existing tag, and publishes the ten packages to PyPI through trusted publishing, each from its own environment (pypi-<package>), when the repository variable PYPI_TRUSTED_PUBLISHING is true (release process).
  • Documentation: the release verification commands name 0.2.3 and the identity of a run started by hand; the README, the quickstart and the MCP guide say how to install lorebase-cli and lorebase-herald from PyPI.

Fixed

  • Web app, Modeler: a save also dropped members the editor does not model inside nested objects: a Kind's search, the Trait implementations of a Kind, Attribute constraints, Operation parameters and preconditions, and a Source's relation_mapping entries and target_branch_policy. They are now kept as they were.
  • Semantic search under iterative scans (pgvector 0.8 or later, the default) left hnsw.ef_search at pgvector's 40, so each pass of the index scan offered 40 candidates and recall@20 was about 0.92 at 300,000 vectors. The read transaction now raises it to search.candidates * search.semantic.overfetch_factor (800 by default, capped at 1000) with and without iterative scans: at 300,000 vectors the 40% filter goes from 0.929 to 0.964, p95 61 ms. The unfiltered list stays at 0.910 with the default index (a query in a region the graph does not reach); see the new HNSW build settings (Added) (performance).
  • docs/reference/configuration.md did not follow the changed description of search.semantic.overfetch_factor: the page is generated from the comments of config/lorebase.example.yaml, which now carries it.
  • Web app, Modeler: in live mode, saving the schema dropped the lenses, the stewards of the Domain and of Kinds, and any member or section of the document the editor does not model. The editor now carries them through untouched and writes them back as they were.
  • Semantic search without iterative scans (pgvector before 0.8, or search.semantic.iterative_scan: off) over-fetched less than configured: an HNSW scan ends after hnsw.ef_search rows (40), so search.candidates * search.semantic.overfetch_factor nearest rows never came back, and a filter keeping 1% of the Records left about 0.2 results of 20. The read transaction now raises hnsw.ef_search to the over-fetch count, up to pgvector's limit of 1000: 7.8 results of 20 on 100,000 vectors, flagged semantic_truncated (performance).
  • Web app, mock mode: accepting an attribute schema suggestion reports an update of its Kind, as Keeper does.

Security

  • Resolution misses: caller digests are HMAC-SHA256 under store.resolve_misses.caller_key, required when store.resolve_misses.enabled is on (at least 32 characters, a secret:// reference), so a reader of resolve_misses can no longer confirm a guessed principal id. Changing the key restarts the count of distinct callers; a caller seen under both keys counts twice until its phrases expire, and lorebase-scribe misses purge DOMAIN starts clean. Migration 0047 clears the digests stored before it.

Not done yet

  • Corrections that stick: the last-written state of a Source is not shown in the web app or the API.
  • Reconciliation: no queries by valid time (periods classify claims, reads stay at Revisions), no key aliases after a duplicate merge (references to the absorbed key are not redirected), and no MCP tool for the queue. In the web app, tab counts stop at VITE_RECONCILIATION_COUNT_LIMIT and the signal filter and ranking apply to the pages loaded.
  • Allowed merge origins per protected branch (roadmap).
  • Resolution misses are not linked to the next query of the same agent session, and context packs and search carry no coverage block (roadmap).
  • The performance targets of the design at 10M Records on the reference hardware, and the 1M-Record neighborhood on branches other than main, just over budget when the working set does not fit in memory (performance).
  • The stability guarantees for /api/v1 bind from 1.0 (API stability).

Upgrade notes

  • Where steward approvals or required roles apply to a protected branch, Source and direct-extraction merges into it now wait for review (above, Changed); give Scribe the same configuration file as Keeper so it sees keeper.proposals. Existing embedding indexes keep ef_construction 64 until rebuilt (operations).
  • stale_after suits incremental feeds that deliver their live rows again within the window; a feed that sends only changed rows flags every unchanged Record once stale_after passes (false positives): leave it null there (Scribe).
  • Migration 0044 builds ix_mat_records_object_hash on mat_records (domain_id, branch, object_hash) per partition with CREATE INDEX CONCURRENTLY (writes continue; no restart; extra space about 40 bytes per row). See operations.
  • Migration 0038 creates reconciliation_candidates and adds two nullable columns to provenance (no backfill); 0039 adds indexes to the candidates. Run the migrations to 0039, then restart Keeper and Scribe. Detection is on by default (scribe.reconciliation.enabled) and only raises candidates; the merge check defaults to warn, which blocks nothing.
  • Migration 0040 creates source_written and source_written_pending, empty. The first run of each Source after it is compare-only: every owned Attribute whose upstream value differs from the target's and was not written by that Source (an import, a hand edit, another Source) is a DV1 Conflict under the default local_edits: review, and a snapshot deletion of a Record holding such values a DV2, so an auto_when_clean Source that merged silently stops at an open Proposal until someone reviews it. Run each Source with dry_run: true first (local_edits, seeded). When the target's values are stale copies of upstream, set local_edits: source_wins for that first run (the old behavior, each overwrite audited) and back to review after; otherwise review in bulk with lorebase proposal conflicts PROPOSAL --status open --json and lorebase proposal resolve PROPOSAL --file resolutions.json (or POST .../proposals/{p}/conflicts:resolve), then merge (operations).
  • Migration 0042 adds proposals.kinds (null for existing Proposals until their next merge preview; meanwhile their Kinds are computed when steward roles, assignees or the steward approval rule are read). Without settings.extraction and without stewards in a schema nothing changes; scribe.extraction.mode: direct is refused at startup (operations).
  • Enabling store.resolve_misses.enabled now requires store.resolve_misses.caller_key (at least 32 characters, a secret:// reference); the settings fail validation without it.
  • From 0.2.3: replace the images (or the packages) with 0.3.0, run the migrations to 0047 (0041, 0043, 0045, 0046 and 0047 are additive), then restart Keeper and Scribe. Read the notes on 0040 and 0042 above before the first Source runs.
  • From 0.2.0, 0.2.1 or 0.2.2: as from 0.2.3; the migrations include 0037 of 0.2.2.
  • From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.3.0.