Lorebase 0.2.2

Kind-scoped invisibility completes full RBAC: a caller whose grants are narrowed to Kinds no longer learns of the other Kinds, and gets 404 or 422 where it got 403 (keeper.auth.authz.invisible_kinds: false keeps the earlier answers). Lexical search can rank with BM25 through ParadeDB pg_search, opt-in and off by default. Seven fixes for running Lorebase locally: compose rebuilds its images and migrate refuses a database ahead of the code, Keeper settings forwarded from .env, a clear message for reserved Attribute names, herald.http.allow_anonymous, Herald as the compose profile mcp on 127.0.0.1:8091, CORS expose_headers and an additive allow_headers, and the web app README for live mode. Migration 0037 adds functions only.

This release was built and published locally, without the release workflow: the images on GHCR, the checksums and the release assets are not signed with cosign, and the images carry no SBOM attestation (the CycloneDX SBOMs are attached to the release as files). cosign verify does not apply to 0.2.2.

Breaking changes

• Kind-scoped invisibility is on by default: principals whose grants are narrowed to Kinds get 404 or 422 instead of 403 for Kinds they may not read, and no longer see those Kinds in the schema, counts, search or context packs. Clients that told a forbidden Kind from an unknown one by the 403 must stop relying on it, or set keeper.auth.authz.invisible_kinds: false to keep the earlier answers (design 13.3).
• Docker compose builds the backend, backend-ml and web images from the checkout on every up; set LOREBASE_PULL_POLICY=missing to keep using prebuilt images (deployment).

Added

• Kind-scoped invisibility, the last part of full RBAC. A caller whose grants are narrowed to Kinds no longer learns of the other Kinds: GET /schema (JSON and YAML) and schema elements return the schema without them and without the Relations, path Derivations, derived and ref Attributes, Traits, Operations and Sources that would tell about them, and every request naming one of them answers with the 404 or 422 of a name that does not exist, never a 403. Counts, Link summaries, neighborhoods, search hits and typo corrections, health reports, context packs and agent trace refs cover the visible Kinds only. Such callers can now read the Domain, its branches and tags, so the web app, Herald and the Guide work for them. On by default; keeper.auth.authz.invisible_kinds: false keeps the 403 naming the Kind (design 13.3, threat model).
• Optional BM25 ranking of lexical search through ParadeDB pg_search, for searches of very common terms on large Domains: search.backend: bm25 ranks with a BM25 index of the existing search vectors, built by python -m lorebase_store.storage apply with store.storage.bm25_index: true on a server that preloads pg_search. Matches, typo corrections, partial matching, Kind filters and masking are those of the default ranking; scores are relative to the best hit. Without the extension or a valid index search keeps the default ranking and /readyz reports search_backend: degraded. pg_search is licensed under AGPL-3.0 and runs only where you choose its image: the compose overlay deploy/compose/search-bm25.yaml and the Helm values file ci/search-bm25-values.yaml set it up; the default stack, chart and images are unchanged. At 1M Records very common words are 20 to 35% faster at p50 and never truncated; rare words are about 3 ms slower, overlays slower, and scans slow down after writes until VACUUM (search, performance).
• Migration 0037: lb_search_terms, lb_bm25_query_valid and the security definer functions lb_search_bm25 and lb_search_bm25_overlay, granted to the application role like the other read functions. They need nothing of the extension until it is installed.
• make perf-up PERF_SEARCH=bm25 runs the perf Postgres on the ParadeDB image, and tests/perf/bm25.py compares both rankings, the index build and the commit cost on a seeded database.
• herald.http.allow_anonymous (default false): Herald over Streamable HTTP forwards calls without a credential to Keeper without one instead of refusing them, for local stacks whose Keeper runs with auth mode none. Keeper still authenticates each call, so under any other mode they keep failing with [unauthorized] (MCP).
• Herald in docker compose: the optional profile mcp runs it over Streamable HTTP on port 8090 in the container, published on 127.0.0.1:8091 (HERALD_HOST_PORT; 8090 is the web app's), wired to the stack's Keeper, with DNS rebinding protection for 127.0.0.1, localhost and herald. make up PROFILES=mcp starts it with the default services (MCP).
• Docker compose forwards documented Keeper settings from .env to the Keeper services when they are set there: LOREBASE__KEEPER__AUTH__MODE, ...__PROPOSALS__ALLOW_SELF_APPROVAL, ...__PROPOSALS__REQUIRED_APPROVALS, ...__OPERATIONS__DIRECT_COMMIT_ON_PROTECTED and the ...__CORS__ origins, headers and credentials. Unset, the YAML file decides as before (deployment).

Changed

• Kind-narrowed callers get 404 or 422 instead of 403 for Kinds they may not read, see the cache entries and ETags of their own projection, and no longer see a Kind whose keys reference an invisible Kind or whose titles or full-text index are computed from one (a searchable count of its Records); derived Attributes reading an invisible Kind, such as supplier_count and single_sourced for a reader of Parts only, are left out of their Records and of Derivation previews. Agent trace steps of other principals no longer carry result counts, query plans or a truncated flag for them, Proposal extraction checks no longer count items for callers who do not see every Conflict, and resolving a Conflict outside their Kinds answers 404 like an unknown one. Branch responses leave out the progress of background copies for them. Proposal Conflicts about a reference to an invisible Kind are left out for them, and Record history, lineage, explanations and context packs leave out origins, messages and Proposal titles naming something invisible and the changes that touched only what is left out.
• Documentation: the validation record gains the five-day scenario's CI figures, the rounds and findings of the PII vault and outbox security reviews; the README and the documentation landing page describe v0.2.1; the roadmap keeps open only the reference-hardware benchmark and the API stability guarantees (validation, roadmap).
• The standby of the compose profile scale-out passes the arguments of its service command to Postgres, so an overlay can add settings to it.
• Documentation: the web app README explains live mode against a Keeper on another port or host, through the dev proxy (LOREBASE_KEEPER_PROXY) or cross-origin (VITE_KEEPER_BASE_URL), and how Keeper allows the app's origin with LOREBASE__KEEPER__CORS__ALLOW_ORIGINS as a JSON list (apps/web).

Fixed

• Keeper CORS: keeper.cors.expose_headers (default ETag, Lorebase-Revision) makes conditional reads work for a web app on another origin, which could not read the ETag and so never got a 304. keeper.cors.allow_headers now adds to the headers Keeper always allows (Authorization, Content-Type, Idempotency-Key, If-None-Match, the request id, session CSRF and agent trace headers) instead of replacing them, and the trace headers are allowed even with tracing off, since clients send them anyway (configuration).
• A Kind Attribute named after a reserved api name (key, kind, ref, revision) is refused with a message that says why and proposes another name, such as "`key` is a reserved name, choose another (e.g. `code`)", instead of api name 'key' is reserved; the schema documentation lists the reserved names (design 5.4).
• Docker compose no longer starts a stale local image: the backend, backend-ml and web images are rebuilt from the checkout on every up (pull_policy: build; LOREBASE_PULL_POLICY=missing keeps prebuilt images), and migrate refuses a database that a newer Lorebase already migrated with an error naming the database revision, the newest revision of the code and the fix, instead of Alembic's Can't locate revision and a Keeper that restarts in a loop (deployment).

Upgrade notes

• keeper.cors.allow_headers lists further headers only; its default is now empty. A configuration that repeats the earlier default keeps working, and one that listed fewer headers now allows the required ones as well.
• Clients of Kind-narrowed principals that relied on 403 to tell a forbidden Kind from an unknown one now get 404 or 422 for both. Set keeper.auth.authz.invisible_kinds: false to keep the earlier answers.
• Migration 0037 creates functions only and runs on any Postgres; apply it as usual. Search is unchanged unless you set search.backend: bm25 and build the index.
• From 0.2.1: replace the images (or the packages) with 0.2.2, run the migration to 0037, then restart.
• From 0.2.0: as from 0.2.1; the notes of 0.2.1 add nothing to do.
• From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.2.2.

Not done yet

• The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory (performance).
• The performance targets of the design at 10M Records: the 2-hop neighborhood, full-text search (searches of very common words above all), Record history and lineage are over budget, and a commit of 1,000 changes takes more than 2 s (performance).
• The stability guarantees for /api/v1 bind from 1.0 (API stability).
• Packages on PyPI: the wheels and sdists are attached to the GitHub release.
• Signed images and assets: 0.2.2 was published without the release workflow (see above).
• BM25 search at 10M Records is not measured (make perf-up PERF_VM=16g PERF_SEARCH=bm25, then tests/perf/bm25.py --database perf_seed_10m_zipf), and storage apply does not tune autovacuum for it: with BM25 on and frequent writes, set a lower autovacuum_vacuum_scale_factor on the mat_records partitions yourself (performance).