Lorebase 0.3.1

A patch for the first contact with the project: the published images run on arm64, the PyPI packages carry their license and links, a Scribe worker keeps all its slots when the database fails while it records a job and gets a liveness check, the console scripts answer --help, and the quickstart commands point at the Keeper they started. No migration and no breaking change. Three new settings, all optional: scribe.worker.outcome_retry_delay_s, scribe.worker.liveness_file and scribe.worker.liveness_interval_s. The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Added

• Scribe liveness (Scribe): with scribe.worker.liveness_file set, the job loop rewrites that file every scribe.worker.liveness_interval_s (10 s) with its live slots, the jobs running and the time it last reached the store, and python -m lorebase_scribe.liveness FILE reads it, with the standard library only, and exits 1 with the reason (no file, a stale file, no live slot or, with --max-store-age, a store not reached). The compose scribe service writes /tmp/scribe-liveness.json and runs that command as its healthcheck; the scribe values of the chart show the exec probe
• Metric lorebase.scribe.worker.slots (gauge, state live or lost) and the status unrecorded of lorebase.scribe.jobs (metrics)
• lorebase-herald --help prints the variables an MCP host sets (LOREBASE_URL, LOREBASE_API_KEY, LOREBASE_DOMAIN, LOREBASE_BRANCH, LOREBASE__HERALD__TRANSPORT) and the documentation link, --version the version; at start Herald logs the transport, the Keeper it calls and the default Domain to stderr, so a host pointed at the wrong Keeper sees it before the first call (MCP)
• make up prints the export LOREBASE_URL=... LOREBASE_DOMAIN=supply line for the CLI and the env entry for an MCP host over stdio, with the port set in .env; make shell-env prints the export line again, for eval "$(make shell-env)" (quickstart)
• make up checks every host port the chosen profiles publish before it starts anything and names the variable to change, and refuses to start while a *_BIND value in .env would publish the Keeper of the stack, which runs with authentication off, beyond this machine; KEEPER_PUBLISH_INSECURE=true, in .env or the shell, starts it anyway with a warning (quickstart)
• make env sets HERALD_DOMAIN=supply while it is unset, so the first call of an MCP host against make up PROFILES=mcp needs no Domain of its own
• server.json at the repository root describes Herald for the official MCP registry as io.github.aantenore/lorebase-herald, with the PyPI package lorebase-herald (run with uvx) and the image ghcr.io/aantenore/lorebase, which now carries the io.modelcontextprotocol.server.name label the registry checks; the entry is submitted after a release (releasing)
• MCP server: host configurations that run Herald from PyPI with uvx lorebase-herald, as one claude mcp add command, a Cursor .cursor/mcp.json entry and a VS Code code --add-mcp command
• Comparison: DataHub, the Fabric IQ ontology MCP server, Semantica, Doltgres with the Dolt MCP server, TypeDB, mnem and RDF triple stores, each with a linked source checked on 2026-11-06, and a Standards section: LinkML import and export are supported; RDF, OWL, SHACL and JSON-LD are not native and are reached through LinkML's generators
• Images ghcr.io/aantenore/lorebase, lorebase-ml and lorebase-web for linux/arm64 as well as linux/amd64: each platform is built on a runner of its own architecture, and the release joins them into one multi-arch index, which it signs and attests with the SBOM (releasing)
• PyPI metadata of the ten packages: license Apache-2.0 with the license text in every sdist and wheel, a README, author, keywords, classifiers and links to the repository, documentation, changelog and issues; scripts/release_tool.py check fails when a package lacks them, and also checks server.json (its versions, its name and the image label)

Changed

• Scribe: an outcome the store refuses to take (a dropped connection, a statement timeout) is written once more after scribe.worker.outcome_retry_delay_s (1 s); when that fails too the run counts unrecorded and the job runs again once its lock goes stale, as a lost job does (Scribe)
• README and the documentation home open with who Lorebase is for and one MCP call with the JSON answer and its Revision; the status note follows, and the 30-second demo stays the second screen
• Quickstart: the commands take the Keeper URL from the line make up prints (eval "$(make shell-env)") instead of a fixed http://localhost:8080; the port table and the port troubleshooting row list GUIDE_HOST_PORT (8095), HERALD_HOST_PORT (8091) and the scale-out ports; the MCP section sets HERALD_DOMAIN next to HERALD_ALLOW_ANONYMOUS; the semantic search example writes query.json under ${TMPDIR:-/tmp}
• CONTRIBUTING.md, the README and the validation and demo pages state the CI policy as it is: the checks run on demand and for release tags, not on every pull request
• Kubernetes and deployment say that the 0.3.0 images are linux/amd64 only and that 0.3.1 and later are published for linux/amd64 and linux/arm64
• Releasing: a section's heading reads X.Y.Z - unreleased until the tag, whose date then replaces unreleased; the page follows the workflow's jobs and adds the MCP registry step
• The Release workflow runs the CI workflow on the tagged commit first; images, the GitHub release and PyPI wait for it, so a tag whose checks fail publishes nothing. After the release it publishes the documentation site built from the tag (the github-pages environment must allow v* tags to deploy)
• The compose defaults of Postgres (pgvector/pgvector:0.8.7-pg16), Ollama, LiteLLM, the OpenTelemetry collector, Jaeger, Prometheus, Grafana and the OIDC mock, and the Helm default of Ollama, are pinned by version and digest instead of latest, main-stable or pg16; the *_IMAGE variables still override them, and tests/tooling/test_dockerfile_pins.py checks the pins
• CI: the integration job may run 35 minutes instead of 20, and the BM25 integration job is part of the aggregate CI check
• The release jobs no longer keep the checkout token on the runner
• .env.example sets POSTGRES_IMAGE, OLLAMA_IMAGE, LITELLM_IMAGE, OTEL_COLLECTOR_IMAGE, JAEGER_IMAGE, PROMETHEUS_IMAGE and GRAFANA_IMAGE to the version and digest pins of deploy/compose/docker-compose.yaml instead of floating tags, so a .env created by make env keeps the pinned images.

Fixed

• A Scribe worker slot ended for good, and nothing reported it, when the database failed while the worker recorded a job's outcome: with the default concurrency the process stayed up with one slot, or none, while the queue grew. The slot now stays on, a slot task that ends with an error is logged and started again after scribe.worker.poll_interval_s, and lorebase.scribe.worker.slots shows it (Scribe)
• lorebase with no server configured said pass base_url or set LOREBASE_URL, an argument of the Python client the command line does not have; it now names the three ways a person has (--url, LOREBASE_URL, or url in a profile of the config file) and the path of that file
• lorebase-herald --help and --version started the stdio server, printed nothing and exited 0 when the host closed the pipe
• The audit chain heads and the merge attempts queries used a DISTINCT ON spelling that SQLAlchemy 2.1 deprecates (an SADeprecationWarning in every integration run); they use the dialect's distinct_on, and the test suite fails on any such warning from now on
• The 30-second demo excerpts in the README, the documentation home and manual setup quoted Revision ids that examples/merge_demo.py no longer prints; they are regenerated, and tests/tooling/test_merge_demo.py now fails when a quoted line differs from the output
• Stale documentation: the comparison listed LinkML import and export as roadmap work, the FAQ named the 1M-Record baseline as missing and left out the MySQL/MariaDB and SQL Server Sources, and examples/README.md linked a quickstart anchor that moved; tests/tooling/test_docs_links.py now checks the links and anchors of the README, CONTRIBUTING.md, SECURITY.md and examples/README.md
• scripts/quickstart.sh up resolves the variables of its exposure and port checks the way docker compose does: a variable set in the shell wins over the environment file, then the default. KEEPER_BIND=0.0.0.0 KEEPER_ALLOW_INSECURE=true make up with a loopback .env no longer starts an exposed Keeper with authentication off, and KEEPER_PUBLISH_INSECURE set in the shell (true or false) wins over the file.

Not done yet

• The chart wires no probe for Scribe by default; the scribe values show the exec probe to set.
• The MCP registry entry (server.json) is submitted by hand after the release; the registry lists nothing until then.

Upgrade notes

No migration. Rebuild or pull the images: the compose scribe service gains a healthcheck that reads the liveness file, with nothing to configure. The new settings have defaults; set scribe.worker.liveness_file and an exec probe to give another deployment the same check.

Lorebase 0.3.0

Knowledge that stays correct: a reconciliation queue for contradictions between Sources, corrections that survive Source syncs, document versions, ingestion modes and stewards, Traits and lenses in search and context packs, the merge of text values, and the resolution of words to schema and values with consolidation from use and coverage signals; the web app gains a Reconciliation screen and a passage-by-passage Conflict resolver. Pre-1.0: this minor version changes interfaces; read the breaking changes before upgrading.

Ten migrations (0038 to 0047). New configuration sections keeper.reconciliation, keeper.stewards, scribe.reconciliation, scribe.consolidation, store.values, store.resolve_misses, store.coverage, search.resolve and read.coverage, and new keys store.resolve_misses.caller_key, store.storage.hnsw_m, store.storage.hnsw_ef_construction, search.semantic.iterative_scan and search.semantic.max_scan_tuples. New endpoints POST /domains/{d}/resolve and /domains/{d}/reconciliation; responses add coverage to POST /query and POST /resolve, semantic_truncated to POST /search, can.settings to GET /domains/{d}, and source_key, version, replaces and status to documents. A Source sync no longer overwrites a value someone corrected on the target, and automatic merges into a protected branch wait for steward and role approvals (below, Breaking changes). The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Breaking changes

• Source syncs no longer put back a value someone corrected on the target: an auto_when_clean Source that merged silently stops at an open Proposal with a DV1 or DV2 Conflict when upstream changes such a value, and on its first run after migration 0040. Set local_edits: source_wins on the Source to keep the old behavior (below, Changed and Upgrade notes).
• Source syncs with auto_when_clean or auto_with_authority and extraction in direct mode no longer merge into a protected branch whose rules include required_approvals_from_stewards or required_roles; the Proposal stays open (merge_blocked: approvals, direct_blocked: approvals). Installations without those rules see no change (below, Changed).
• A text Attribute edited in different sentences on two Branches merges automatically (resolver policy:text_merge) instead of raising a Conflict. Set text_merge: false in the Domain's merge_policy setting to keep the Conflict (below, Changed).
• Semantic search reads the HNSW index at any size: unfiltered and broad-filter lists are approximate where they were exact (recall 0.92 to 0.99 at 300,000 vectors); a one-Kind filter stays exact (below, Changed).

Added

• Resolution of words to schema and values (design 8.9, AC-39). POST /domains/{d}/resolve {text, kinds?, levels?, query_vector?, limit?, branch?, at?} returns ranked candidates the words of a text may stand for: Kinds, Attributes (by name or alias, with their type) and stored values with a ready-to-use where predicate and the count of Records holding them, each with its level (exact, synonym, trigram, embedding), score and the words matched, plus the words nothing matched. It never interprets the question, runs a query or calls a model; the embedding level only uses a caller's query_vector when no other level matched. Candidates never name a Kind or Attribute the caller may not read. Herald lorebase_resolve, CLI lorebase resolve TEXT, Python client records.resolve (sync and async). Settings search.resolve (agents).
• Value index: mat_values holds per branch the distinct short values of selected Attributes, kept by triggers on mat_records in the writing transaction (migration 0045, additive). Enums are selected always, plain strings while their distinct values stay few (store.values); the Scribe value_index pass (scribe.scheduler.value_index_interval_s, default 300 s) and lorebase-scribe value-index DOMAIN [--rebuild] decide and backfill the selection. Sensitive and vault values are never indexed. A pass that changes the selection fences only its own Domain's writers (a transaction advisory lock taken shared by the triggers), for at most store.values.pass_timeout_ms; a schema change that makes an Attribute ineligible removes its values in the same commit. At 100,000 Records resolution answers in 11.2 ms at p95 (levels 1-2); the triggers cost about 90 µs per statement where nothing is held and 0.3 ms per updated Record whose held values change (performance, operations).
• Value index (design 8.9): an Attribute's distinct-value limit now holds between value_index passes. The limit is max_values: hard_max_distinct for enums and resolvable: true, max_distinct × hysteresis for plain strings. The write that takes a branch past it marks the Attribute overflowed in the same transaction: resolution stops returning its values at once, later writes stop indexing it, and the next pass drops it. A schema commit or merge on the default branch applies a new limit in the same commit. Reads also skip a branch past its limit that is not yet marked.
• Migration 0047 adds mat_value_sizes, the per-branch count of held values kept by the value triggers, and max_values and overflowed to mat_value_attributes. Passes read the sizes instead of counting mat_values. On upgrade, existing selections get hard_max_distinct and their sizes are counted; the first pass of each Domain counts its sizes again and sets its own limits. Trigger cost: about 90 µs per statement where nothing is held, as before; 0.3 to 0.4 ms per updated Record whose held values change, against 0.29 ms before (measured with tests/perf/resolve_bench.py on a loaded machine; performance).
• Schema: aliases on Kinds and Attributes, value_aliases (stored value to other words) and resolvable: true|false on Attributes; absent by default, so existing schemas keep their hashes. Validation refuses resolvable and value_aliases on types other than string and enum (alone or in arrays), on sensitive and on link Attributes, and value aliases of an enum that name no value (schema.attribute_flag_invalid). The web Modeler keeps them on round trips.
• Herald lorebase_query now documents counting with "aggregate": {"group_by": attr} and points to lorebase_resolve.
• Consolidation from use (design 8.10, AC-40): with store.resolve_misses.enabled (off by default), Keeper keeps the phrases POST /resolve matched to nothing per Domain, branch and phrase (counts, first and last seen, digests of distinct callers; migration 0046), off the request path and only for head reads of callers holding search.resolve.misses.permission Domain-wide; requests opt out with record_misses: false, and searches narrowed by kinds or levels are not recorded. One caller adds and holds a bounded number of phrases; a full Domain evicts its least missed phrase. Phrases expire after retention_days (30) and go with their Domain, and so do the vocabulary suggestions quoting them (their share removed, a suggestion left without phrases deleted, decided or not). A Scribe vocabulary_consolidation pass (scribe.scheduler.consolidation_interval_s, CLI lorebase-scribe consolidate DOMAIN) turns frequent misses into schema suggestions of the new types value_alias, attribute_alias, kind_alias (near values and names; never for sensitive, vault, derived or non-resolvable Attributes, never a phrase that already names something) and, with scribe.consolidation.llm.enabled, attribute_description (task schema_consolidation; the model sees names and phrases only). They are reviewed like other suggestions; accepting one opens a schema change Proposal. Vocabulary suggestions have their own cap (schema.suggestions.max_vocabulary_tracked) so they cannot crowd out extraction's; they are withheld from reviewers who may not read their Kind's Records or Attribute's values; an alias that would name two things is refused at acceptance. lorebase-scribe misses purge DOMAIN deletes a Domain's misses and vocabulary suggestions.
• Coverage signals (design 8.11, AC-41): POST /query and POST /resolve answer with a coverage block (coverage: false leaves it out; default read.coverage.enabled): the share of Records missing each Attribute used, Source freshness (last successful sync, stale_after exceeded, Records unseen), the phrases unknown in the Domain, and one note per signal. It reads per-Kind statistics of the default branch head (mat_coverage, migration 0046) counted by a Scribe coverage pass (scribe.scheduler.coverage_interval_s, CLI lorebase-scribe coverage DOMAIN) from one snapshot per Kind, Records unseen by Sources recounted every pass; the last successful run of a Source is one probe of the partial index ix_source_runs_last_succeeded; masked Attributes and Kinds outside the caller's scope are never described, and other branches and Revisions get no statistics. The two lookups take p95 3 ms at 100,000 Records and 600,000 Source runs.
• Python client Coverage on QueryResult and ResolveResult, coverage on QueryRequest and ResolveRequest, record_misses on ResolveRequest, the vocabulary suggestion types and attribute, value, description on SchemaSuggestion; CLI lorebase query and lorebase resolve print the coverage notes; Herald lorebase_query and lorebase_resolve end with coverage: lines.
• Web app, Review > Schema suggestions: ...

Lorebase 0.2.3

A governed bulk delete: delete_records removes many Records of one Kind in one Operation execution, one Revision and one rate-limit token. No migration, no breaking change.

Added

• Built-in bulk delete delete_records (kind, keys, optional skip_missing): deletes many Records of one Kind in one Operation execution, one Revision and one writes rate-limit token, instead of one delete_{kind} call (and Revision) per Record. It is governed like the other built-ins: protected branches and review, dry_run, expected_head, Idempotency-Key, and Kind-scoped grants as for delete_{kind}; it needs the Kind's delete built-in enabled. Links of the deleted Records follow each Relation's on_delete (cascade or restrict) as a single delete does. Unknown keys fail the call (record.not_found per key) unless skip_missing leaves them out with warnings. At most keeper.operations.max_bulk_delete_keys keys per call (default 1000). Python client operations.delete_records, CLI lorebase record delete KIND [KEY]... [--keys-file FILE] [--skip-missing], and Herald lists it among the generic Operations (design 5.7, operations).

Lorebase 0.2.2

Kind-scoped invisibility completes full RBAC: a caller whose grants are narrowed to Kinds no longer learns of the other Kinds, and gets 404 or 422 where it got 403 (keeper.auth.authz.invisible_kinds: false keeps the earlier answers). Lexical search can rank with BM25 through ParadeDB pg_search, opt-in and off by default. Seven fixes for running Lorebase locally: compose rebuilds its images and migrate refuses a database ahead of the code, Keeper settings forwarded from .env, a clear message for reserved Attribute names, herald.http.allow_anonymous, Herald as the compose profile mcp on 127.0.0.1:8091, CORS expose_headers and an additive allow_headers, and the web app README for live mode. Migration 0037 adds functions only.

This release was built and published locally, without the release workflow: the images on GHCR, the checksums and the release assets are not signed with cosign, and the images carry no SBOM attestation (the CycloneDX SBOMs are attached to the release as files). cosign verify does not apply to 0.2.2.

Breaking changes

• Kind-scoped invisibility is on by default: principals whose grants are narrowed to Kinds get 404 or 422 instead of 403 for Kinds they may not read, and no longer see those Kinds in the schema, counts, search or context packs. Clients that told a forbidden Kind from an unknown one by the 403 must stop relying on it, or set keeper.auth.authz.invisible_kinds: false to keep the earlier answers (design 13.3).
• Docker compose builds the backend, backend-ml and web images from the checkout on every up; set LOREBASE_PULL_POLICY=missing to keep using prebuilt images (deployment).

Added

• Kind-scoped invisibility, the last part of full RBAC. A caller whose grants are narrowed to Kinds no longer learns of the other Kinds: GET /schema (JSON and YAML) and schema elements return the schema without them and without the Relations, path Derivations, derived and ref Attributes, Traits, Operations and Sources that would tell about them, and every request naming one of them answers with the 404 or 422 of a name that does not exist, never a 403. Counts, Link summaries, neighborhoods, search hits and typo corrections, health reports, context packs and agent trace refs cover the visible Kinds only. Such callers can now read the Domain, its branches and tags, so the web app, Herald and the Guide work for them. On by default; keeper.auth.authz.invisible_kinds: false keeps the 403 naming the Kind (design 13.3, threat model).
• Optional BM25 ranking of lexical search through ParadeDB pg_search, for searches of very common terms on large Domains: search.backend: bm25 ranks with a BM25 index of the existing search vectors, built by python -m lorebase_store.storage apply with store.storage.bm25_index: true on a server that preloads pg_search. Matches, typo corrections, partial matching, Kind filters and masking are those of the default ranking; scores are relative to the best hit. Without the extension or a valid index search keeps the default ranking and /readyz reports search_backend: degraded. pg_search is licensed under AGPL-3.0 and runs only where you choose its image: the compose overlay deploy/compose/search-bm25.yaml and the Helm values file ci/search-bm25-values.yaml set it up; the default stack, chart and images are unchanged. At 1M Records very common words are 20 to 35% faster at p50 and never truncated; rare words are about 3 ms slower, overlays slower, and scans slow down after writes until VACUUM (search, performance).
• Migration 0037: lb_search_terms, lb_bm25_query_valid and the security definer functions lb_search_bm25 and lb_search_bm25_overlay, granted to the application role like the other read functions. They need nothing of the extension until it is installed.
• make perf-up PERF_SEARCH=bm25 runs the perf Postgres on the ParadeDB image, and tests/perf/bm25.py compares both rankings, the index build and the commit cost on a seeded database.
• herald.http.allow_anonymous (default false): Herald over Streamable HTTP forwards calls without a credential to Keeper without one instead of refusing them, for local stacks whose Keeper runs with auth mode none. Keeper still authenticates each call, so under any other mode they keep failing with [unauthorized] (MCP).
• Herald in docker compose: the optional profile mcp runs it over Streamable HTTP on port 8090 in the container, published on 127.0.0.1:8091 (HERALD_HOST_PORT; 8090 is the web app's), wired to the stack's Keeper, with DNS rebinding protection for 127.0.0.1, localhost and herald. make up PROFILES=mcp starts it with the default services (MCP).
• Docker compose forwards documented Keeper settings from .env to the Keeper services when they are set there: LOREBASE__KEEPER__AUTH__MODE, ...__PROPOSALS__ALLOW_SELF_APPROVAL, ...__PROPOSALS__REQUIRED_APPROVALS, ...__OPERATIONS__DIRECT_COMMIT_ON_PROTECTED and the ...__CORS__ origins, headers and credentials. Unset, the YAML file decides as before (deployment).

Changed

• Kind-narrowed callers get 404 or 422 instead of 403 for Kinds they may not read, see the cache entries and ETags of their own projection, and no longer see a Kind whose keys reference an invisible Kind or whose titles or full-text index are computed from one (a searchable count of its Records); derived Attributes reading an invisible Kind, such as supplier_count and single_sourced for a reader of Parts only, are left out of their Records and of Derivation previews. Agent trace steps of other principals no longer carry result counts, query plans or a truncated flag for them, Proposal extraction checks no longer count items for callers who do not see every Conflict, and resolving a Conflict outside their Kinds answers 404 like an unknown one. Branch responses leave out the progress of background copies for them. Proposal Conflicts about a reference to an invisible Kind are left out for them, and Record history, lineage, explanations and context packs leave out origins, messages and Proposal titles naming something invisible and the changes that touched only what is left out.
• Documentation: the validation record gains the five-day scenario's CI figures, the rounds and findings of the PII vault and outbox security reviews; the README and the documentation landing page describe v0.2.1; the roadmap keeps open only the reference-hardware benchmark and the API stability guarantees (validation, roadmap).
• The standby of the compose profile scale-out passes the arguments of its service command to Postgres, so an overlay can add settings to it.
• Documentation: the web app README explains live mode against a Keeper on another port or host, through the dev proxy (LOREBASE_KEEPER_PROXY) or cross-origin (VITE_KEEPER_BASE_URL), and how Keeper allows the app's origin with LOREBASE__KEEPER__CORS__ALLOW_ORIGINS as a JSON list (apps/web).

Fixed

• Keeper CORS: keeper.cors.expose_headers (default ETag, Lorebase-Revision) makes conditional reads work for a web app on another origin, which could not read the ETag and so never got a 304. keeper.cors.allow_headers now adds to the headers Keeper always allows (Authorization, Content-Type, Idempotency-Key, If-None-Match, the request id, session CSRF and agent trace headers) instead of replacing them, and the trace headers are allowed even with tracing off, since clients send them anyway (configuration).
• A Kind Attribute named after a reserved api name (key, kind, ref, revision) is refused with a message that says why and proposes another name, such as "`key` is a reserved name, choose another (e.g. `code`)", instead of api name 'key' is reserved; the schema documentation lists the reserved names (design 5.4).
• Docker compose no longer starts a stale local image: the backend, backend-ml and web images are rebuilt from the checkout on every up (pull_policy: build; LOREBASE_PULL_POLICY=missing keeps prebuilt images), and migrate refuses a database that a newer Lorebase already migrated with an error naming the database revision, the newest revision of the code and the fix, instead of Alembic's Can't locate revision and a Keeper that restarts in a loop (deployment).

Upgrade notes

• keeper.cors.allow_headers lists further headers only; its default is now empty. A configuration that repeats the earlier default keeps working, and one that listed fewer headers now allows the required ones as well.
• Clients of Kind-narrowed principals that relied on 403 to tell a forbidden Kind from an unknown one now get 404 or 422 for both. Set keeper.auth.authz.invisible_kinds: false to keep the earlier answers.
• Migration 0037 creates functions only and runs on any Postgres; apply it as usual. Search is unchanged unless you set search.backend: bm25 and build the index.
• From 0.2.1: replace the images (or the packages) with 0.2.2, run the migration to 0037, then restart.
• From 0.2.0: as from 0.2.1; the notes of 0.2.1 add nothing to do.
• From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.2.2.

Not done yet

• The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory ([pe...

Lorebase 0.2.1

Patch release for the storage profiles of 0.2.0. python -m lorebase_store.storage apply now converges when commits run during the lineage rewrite, the downgrade of migration 0036 no longer deadlocks against commits, and a repeated apply skips the full lineage scan once a pass is recorded complete (--verify forces it). The performance documentation replaces the 10M-Record storage profile estimate with measured numbers. No migration, configuration key or API response changes.

Added

• make perf-up starts the Postgres of the performance harness without running a benchmark; PERF_VM=16g sizes its memory for a Docker VM of 16 GiB (shared_buffers 6GB, effective_cache_size 12GB) instead of the default PERF_VM=8g. Each PERF_PG_* value can still be set on its own; these are harness settings, not product defaults (performance).
• python -m lorebase_store.storage apply --verify scans every lineage map row again and resets what is recorded (performance).

Changed

• storage apply no longer scans every lineage map row again once a rewrite has finished: it records the pass per partition in storage_tasks with a watermark, and a later run with the same profile only looks at the rows written since. At 10M Records the full scan took 13.5 minutes even with nothing to rewrite. Rows written while the triggers were turned off out of band, or frozen by VACUUM before a run looked at them, are only found by --verify (performance).
• The storage profiles at 10M Records are measured instead of estimated (Docker VM of 16 GiB): the hot set goes from 57 GB to 44.1 GB in balanced and 43.3 GB in compact, the cold 2-hop neighborhood from 122 to 100 ms at p50 (96 in compact), and a commit of 1,000 Part updates takes 2,711 ms at p50 in balanced and 2,332 in compact under the same load (performance, validation).

Fixed

• storage apply no longer leaves lineage map rows a commit held during the rewrite in the other form: the page ranges they are in are rewritten again after the pass, and while rows stay held the step is deferred (exit status 3) and the next run finishes them (performance).
• The downgrade of migration 0036 takes the locks of mat_records and mat_links up front, in the order commits write them, each wait shorter than deadlock_timeout: a commit waiting for it is never chosen as a deadlock victim, and neither is the downgrade.

Not done yet

• The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory (performance).
• The performance targets of the design at 10M Records: the 2-hop neighborhood, full-text search (searches of very common words above all), Record history and lineage are over budget, and a commit of 1,000 changes takes more than 2 s (performance).
• Full RBAC: Kind-scoped invisibility in queries (roadmap).
• The stability guarantees for /api/v1 bind from 1.0 (API stability).
• Packages on PyPI: the wheels and sdists are attached to the GitHub release.

Upgrade notes

• From 0.2.0: replace the images (or the packages) with 0.2.1 and restart; there is no migration and no configuration to change. If an earlier storage apply stopped or left lineage map rows in the other form, run it again as the maintenance role: it finishes them, and it exits with status 3 while commits still hold rows, in which case run it again later. The first run of 0.2.1 makes the full lineage pass once and records it; later runs skip it.
• From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.2.1.

Lorebase 0.2.0

Second pre-alpha release. Records now carry their history and lineage, values tagged pii, confidential or restricted are masked and vault values can be erased, Scribe computes embeddings for semantic and hybrid search, path Derivations work, storage profiles decide what the database keeps besides the rows, and Guide, an optional exploration assistant outside Keeper, reads a Domain with the caller's own credential. The web app signs in through Keeper and its Modeler edits Operations, Derivations and Sources, and its copy no longer names the data of any one Domain. Pre-1.0: this minor version changes interfaces; read the breaking changes before upgrading. The database schema moves from migration 0012 to 0036, read.history.lineage_max_changes is removed, Scribe and Guide run on a new image ghcr.io/aantenore/lorebase-ml, and POST /search adds truncated_candidates. The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Breaking changes

• Webhook signing keys are sealed by default: Keeper refuses to create a webhook unless webhooks.encryption_key is set (webhooks.allow_unsealed_keys allows it for local development only), make env generates LOREBASE_WEBHOOK_KEY for the compose stack, and the Helm chart requires secrets.values.webhookEncryptionKey or an existing Secret entry.
• The Helm chart no longer runs Keeper and Scribe as the owner of the tables. As on compose, it takes two DSNs: db-maintenance-dsn (secrets.keys.dbMaintenanceDsn, the owner) for the migration Job and for database.maintenance_dsn of Scribe and Keeper, and db-dsn for the application role (database.appRole.name, default lorebase_app) that Keeper and Scribe connect as, so row-level security, the append-only audit_log and the ban on DDL apply to them. A db-roles init container of the migration Job creates the role when missing (database.appRole.create, needs CREATEROLE) or waits for one created elsewhere, refuses SUPERUSER, BYPASSRLS or the owner, and applies the grants of the compose script; the migration runs with database.application_role. With the in-cluster Postgres the operator manages the role (postgresql.cluster.roles) with the password of secrets.values.dbAppPassword. The chart refuses one user for both DSNs unless database.allowSharedRole: true (development). Upgrading: add db-maintenance-dsn with the old DSN and point db-dsn at the application role; an in-cluster release upgrades in one step (next line) (Kubernetes).
• Kubernetes, in-cluster database: one helm upgrade now adds the application role (kubernetes). A pg-roles pre-upgrade hook Job declares the roles of postgresql.cluster.roles on the CloudNativePG Cluster and waits for the operator before the migration Job runs; upgrading a release that used one DSN no longer takes two upgrades. It needs access to the API server when egress is restricted (networkPolicy.egress.kubernetesApi); database.setup.clusterRoles.enabled: false turns it off.
• Scribe refuses to start when scribe.worker.heartbeat_interval_s is more than half of store.jobs.stale_after_seconds. Jobs run at least once: a job is run again after its worker is lost, so handlers must tolerate a second run (the built-in ones do).
• Reading values of Attributes tagged pii, confidential or restricted needs pii.read, confidential.read or restricted.read; other callers see them masked (design 10.2). Feeding vault values through imports and Sources needs the new Domain permission pii.import (only admin among the built-in roles), and vault key rotation needs vault.manage (installation administrators only). Give the read permissions through a custom role in keeper.auth.authz.custom_roles.
• Sensitive Attributes are left out of full-text search for every caller (see Changed): a search that matched a sensitive word no longer finds the Record.
• read.history.lineage_max_changes is removed (lineage no longer reads a window of changes). Configuration files refuse unknown settings: delete it.
• lorebase admin erase --value V refuses a literal value, which stayed in the shell history: use --value - or --value @FILE. A value erased before is stored erased when written again; vault.erased_values: allow keeps the old behaviour.
• Lexical search ranks a different set of matches by default: within search.common_term_docs (2,000) every match is ranked, beyond it the first search.common_term_candidates (1,000) in index order, and POST /search answers truncated_candidates: true when that cut applies. Before, the first store.reads.search_candidates (1,000) matches were ranked for every query. search.common_term_docs: 0 keeps the old behaviour. Keeper needs migration 0035.
• Compose and Helm run Scribe and Guide on the new image ghcr.io/aantenore/lorebase-ml (844 MB against 453 MB; fastembed and a baked embedding model). Deployments that mirror or pin images need it too; Helm imageMl.repository: "" keeps Scribe and Guide on image (kubernetes).
• Guide needs a writable state directory while its root file system is read-only: compose mounts the volume guide-data at /var/lib/lorebase/guide and Helm guide.persistence (an emptyDir by default); elsewhere the default is ~/.local/state/lorebase/guide, so mount a volume there or set guide.mode_memory_file: null (guide).
• Compose: make up, make logs, make down, make clean-stack and scripts/quickstart.sh add the files listed in COMPOSE_OVERLAYS after their own and stop before compose runs when one does not exist; make down stops every profile make up can start (quickstart).
• Migration 0036 (shared lineage maps) leaves the lineage maps of an existing install inline until python -m lorebase_store.storage apply runs as the maintenance role, so processes of 0.1.0 keep reading them during a rolling upgrade; the space comes back only with apply --compact --reindex (store schema notes). A new install shares maps from the start.
• store.storage.profile (compact, balanced, fast; compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile) decides which secondary indexes exist. The default builds the same indexes as before; compact drops the GIN index on Record values, the indexes of indexed-only Attributes and the lexicon's trigram index when storage apply runs (performance).
• Web app: the supply presets and dataset-specific placeholders are gone, example questions come only from domain.examples of the schema, and the "planned screen" placeholders (Explorer, Modeler, Review, Sources, Settings, the Record page's Run Operation and the Canvas diff overlay tab) are removed. The Retrieval lab no longer embeds the Explore panel; it links to Explore. Every user-facing string moved to apps/web/src/messages, so local text patches must move with it.

Added

• Search ranks every match only when it can afford to (design 8.4, performance). Lexical search without filters reads the word lexicon's counts of the query's lexemes, which bound its matches: within search.common_term_docs (2,000) every match is ranked; when every term is more common, only the first search.common_term_candidates (1,000) matches in index order are, and POST /search answers truncated_candidates: true so that callers narrow the query (Herald and Guide say so to the agent). Before, at most store.reads.search_candidates (1,000) matches were ranked for every query, without saying when that cut the ranking. search.common_term_docs: 0 keeps that behaviour. Measured at 1M and 10M Records with realistic text, 20,000 and 5,000 raised the p95 of title-word searches 3.4 times at 1M and 4.4 times at 10M, hence the lower defaults. Migration 0035 adds the security definer functions lb_search_bounded and lb_search_bounded_overlay, which report the cap.
• values: false on POST /neighborhood (agents): nodes come back as Record headers (kind, key, title) and edges as Link endpoints, for callers that need the structure only. The Python client takes values on NeighborhoodRequest; Herald's lorebase_neighborhood asks Keeper for it unless its values is true (its default leaves values out, as before, now Link values too); Guide's neighborhood tool takes values (default true).
• --vocabulary zipf for examples/supply/scripts/make_supply_data.py: names, titles, summaries and reasons drawn from 40,000 terms with Zipf frequencies plus product codes, the rest of the dataset unchanged. The 1M and 10M benchmarks use it (seed databases perf_seed_1m_zipf and perf_seed_10m_zipf; PERF_VOCABULARY=demo measures the earlier ones); tests/perf/focus.py compares search with and without the lexicon-sized ranking and the neighborhood with and without values on a seeded database.
• Benchmarks at 1M and 10M Records (performance): make perf-1m and make perf-10m start a Postgres of their own (tests/perf/compose.yaml, compose project lorebase-perf), seed the dataset once into a kept database and measure it in place; make perf-down deletes it. The 10M fixture loads a tenth through the commit pipeline and copies its materialized rows with renumbered keys (--replicas); an interrupted load resumes. The harness also times the agent trace through Keeper (--trace-iterations) and can bound the measured branch copy (--copy-batches, --no-pair-copy).
• COMPOSE_OVERLAYS (in .env or the environment): extra compose files, space separated, that make up, make logs, make down, `make c...

Lorebase 0.1.0

First release. Lorebase keeps a typed ontology in Postgres and lets teams change it on branches, review the change in a Proposal and merge schema and data together, with Conflicts resolved per Attribute. Agents read through a REST API that never calls a model, or through the MCP server, and write only through Operations; a write that needs review becomes a task branch and a Proposal. Documents are extracted into Proposals whose items reviewers accept or reject. Pre-alpha and pre-1.0: /api/v1 is not frozen and interfaces may change in a minor version. The database schema is at migration 0012. The lines of this first section link the documentation of each feature; from the next release they link the pull request or issue.

Added

Schema and versioning (design, schema as code)

• A Domain is a YAML schema document (format: lorebase.schema/v1) with Kinds, Attributes, Relations, Traits, Operations, Derivations and Sources, checked statically and applied as one schema change set: POST /api/v1/domains/{domain}/branches/{branch}/schema/changes, lorebase schema validate|diff|apply. Attribute ids are stable, so a rename keeps its history.
• Content-addressed Revisions, branches, tags, diff and time-travel reads (?at=<revision>, lorebase diff, lorebase record get --at).
• Three-way merge of schema and data with typed Conflicts (attribute value, delete or modify, add or add, missing value, type coercion, cardinality, dangling Link and more), resolved with ours, theirs, base or a custom value, one by one or from a file (lorebase proposal conflicts|resolve). Merge policies prefer, cascade and defaults. The merged data is validated against the merged schema before the commit.
• Proposals with reviews, required approvals (keeper.proposals), Conflict resolution and merge over REST and CLI (lorebase proposal create|review|merge|close). The default branch of a Domain is protected (keeper.domains.protect_default_branch): changes reach it through a Proposal.
• Expressions in CEL for Operations (preconditions, steps, review policies) and for expression and aggregate Derivations, type checked when a schema is applied (schema.expressions_enabled) and recomputed in the materialized head on every commit. path Derivations are not computed.

Reads for people and agents (agents, performance)

• Records, Links, the JSON query DSL, traversal, neighborhoods and lexical search over REST (lorebase record, query, search), with ETags and the serving Revision in every response. Full-text search reads GIN indexes on (domain_id, branch, search) and (domain_id, branch, title gin_trgm_ops) (migration 0011, btree_gin), so a search reads only its own branch's matches, however many branches share the partition. Semantic search takes vectors supplied by the caller; nothing computes embeddings.
• Context packs within a token budget (POST .../context, lorebase context), Record explanations with the origin of each value (GET .../records/{key}/explain, lorebase explain) and data health reports (GET .../health, lorebase health), all computed without a model.
• A change feed as Server-Sent Events and as polling (GET .../branches/{branch}/events, .../changes) and signed webhooks (POST .../webhooks, Lorebase-Signature HMAC).
• Overlay branches (store schema notes, migration 0010): a branch created with mode: overlay (POST .../branches, default store.materialize.branch_mode, always for names under agent/, extract/ and source/, and for agent task branches per keeper.agent_writes.branch_mode) stores only the rows it changes and reads its base as of the fork point. Creating one does not depend on the size of the Domain; it is promoted to a full copy above store.materialize.overlay_max_rows rows or overlay_max_base_lag_revisions generations of base lag, before a schema change and before its base is deleted. lorebase branch create --mode copy|overlay chooses the mode from the command line. Overlay reads go through _overlay variants of the security definer functions with the same argument and scope checks. An overlay gets the per-Attribute indexes of its schema on its own rows, built as store.index jobs when it is created, and its neighborhood looks Links and Records up with one index scan over both branches (migration 0012); on the 100k-Record baseline search, attribute queries, Links of a Record and the neighborhood stay within 30% of a copy at p50 (performance).
• Per-Attribute indexes and security definer read functions that keep Domain isolation under row-level security. At 99,900 Records and 302,940 Links on one laptop, the measured p95 of every agent read stays within its budget (performance); the 1M-Record baseline is not measured.

Writes and ingestion (quickstart, scribe, inference)

• Built-in Operations import_records, import_links and import_batch with dry run and idempotency keys, and custom Operations declared in the schema (POST .../operations/{operation}/execute).
• lorebase import records|links|batch load CSV, TSV, JSON Lines or JSON (Parquet with the parquet extra): types are converted from the schema, a Record travels with its Links, files are cut into batches of one Revision each.
• Schema inference from CSV, TSV, JSON, JSON Lines (.jsonl, .ndjson) and Parquet uploads, chosen by suffix (keeper.schema_inference.file_formats), and from Postgres tables (POST .../schema:infer): Kinds, typed Attributes, keys, Relations with cardinality, a confidence and the evidence for every decision. It proposes and writes nothing. Parquet uploads need the lorebase-keeper[parquet] extra; without it they are refused with a 422 naming it. A record profiler (lorebase_store.profiling.profile_records, RecordProfiler) and profile_source in Scribe profile any Source through its connector; limits are in store.profiling.records.
• CSV and Postgres Sources with snapshot and incremental syncs run by Scribe, the worker and scheduler on the Postgres job queue. A sync commits on a Source branch, opens a Proposal and applies the Source's merge policy (proposal, auto_when_clean, auto_with_authority); every changed value keeps its row as Lineage (lorebase sources sync|runs).
• JSON, JSON Lines, Parquet and HTTP JSON Sources (scribe): the json and jsonl types and the records_path, flatten_separator and http options (headers as secret references, cursor, page, offset or Link header pagination, a watermark parameter and a page cap). Files are opened under the allowed roots without following links, URLs are checked against private addresses, JSON Lines and Parquet are streamed, and an HTTP JSON run that passes max_pages fails instead of committing part of a snapshot. Parquet needs the lorebase-scribe[parquet] extra.
• Document intake and tiered extraction into Proposals (extraction): POST .../documents takes Markdown, plain text, PDF and DOCX (PDF and DOCX with the scribe documents extra); Scribe parses and chunks them and proposes Records, Links and values with verified evidence, from rules (tier 0, no model), local GLiNER and GLiREL (tier 1) or any OpenAI-compatible model with JSON schema output (tier 2, cached, token budget per job). Reviewers accept, edit or reject items one by one or by filter (GET .../proposals/{p}/items, POST .../items:decide, lorebase documents upload, lorebase proposal items|decide); nothing reaches the default branch without a merge. Derived and sensitive Attributes are never extracted. GET .../documents/{id}/text (lorebase documents text) serves the parsed text the evidence offsets index. The web app has a Documents screen (intake, library, a viewer that highlights the evidence in the parsed text) and an Items tab on extraction Proposals to accept, edit or reject items one by one or by filter.
• Inference providers behind one OpenAI-compatible interface for chat and embeddings, plus local fastembed, GLiNER and GLiREL (optional extras), and compose profiles for Ollama and a LiteLLM proxy. Extraction uses them; nothing computes embeddings yet.

Agents and access (MCP, operations)

• Herald, the MCP server, over stdio and Streamable HTTP with 11 tools (lorebase_context, lorebase_query, lorebase_execute with dry run and more), two resources and one prompt. It is a thin layer over Keeper and holds no database credentials. lorebase_execute passes intent, auto_propose and auto_merge to Keeper also when the call names no task, and returns Keeper's problem text unchanged; auto_propose: true without a task is refused as validation-failed.
• Agent writes that need review (agents): an Operation refused with 409 review-required (protected branch or review policy) runs, when the call names a task, on a task branch agent/{principal}/{task} and opens or updates one Proposal, merged at once only when keeper.agent_writes.allowed_roles_for_auto_merge allows the caller's role, nothing conflicts and the required approvals are in. Without a task the 409 carries next hints. Settings keeper.agent_writes, overridable per Domain.
• API keys (hashed with a server-side pepper), OIDC bearer tokens, role grants per Domain, Kind, Operation and branch, and a hash-chained audit log (lorebase apikey, GET .../audit). python -m lorebase_keeper bootstrap creates the first installation administrator key from the database settings and prints it once; it refuses when an administrator already holds a usable key, unless --force.
• Rate limits per principal and request class (RateLimit-* h...