Lorebase 0.2.0

Second pre-alpha release. Records now carry their history and lineage, values tagged pii, confidential or restricted are masked and vault values can be erased, Scribe computes embeddings for semantic and hybrid search, path Derivations work, storage profiles decide what the database keeps besides the rows, and Guide, an optional exploration assistant outside Keeper, reads a Domain with the caller's own credential. The web app signs in through Keeper and its Modeler edits Operations, Derivations and Sources, and its copy no longer names the data of any one Domain. Pre-1.0: this minor version changes interfaces; read the breaking changes before upgrading. The database schema moves from migration 0012 to 0036, read.history.lineage_max_changes is removed, Scribe and Guide run on a new image ghcr.io/aantenore/lorebase-ml, and POST /search adds truncated_candidates. The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Breaking changes

• Webhook signing keys are sealed by default: Keeper refuses to create a webhook unless webhooks.encryption_key is set (webhooks.allow_unsealed_keys allows it for local development only), make env generates LOREBASE_WEBHOOK_KEY for the compose stack, and the Helm chart requires secrets.values.webhookEncryptionKey or an existing Secret entry.
• The Helm chart no longer runs Keeper and Scribe as the owner of the tables. As on compose, it takes two DSNs: db-maintenance-dsn (secrets.keys.dbMaintenanceDsn, the owner) for the migration Job and for database.maintenance_dsn of Scribe and Keeper, and db-dsn for the application role (database.appRole.name, default lorebase_app) that Keeper and Scribe connect as, so row-level security, the append-only audit_log and the ban on DDL apply to them. A db-roles init container of the migration Job creates the role when missing (database.appRole.create, needs CREATEROLE) or waits for one created elsewhere, refuses SUPERUSER, BYPASSRLS or the owner, and applies the grants of the compose script; the migration runs with database.application_role. With the in-cluster Postgres the operator manages the role (postgresql.cluster.roles) with the password of secrets.values.dbAppPassword. The chart refuses one user for both DSNs unless database.allowSharedRole: true (development). Upgrading: add db-maintenance-dsn with the old DSN and point db-dsn at the application role; an in-cluster release upgrades in one step (next line) (Kubernetes).
• Kubernetes, in-cluster database: one helm upgrade now adds the application role (kubernetes). A pg-roles pre-upgrade hook Job declares the roles of postgresql.cluster.roles on the CloudNativePG Cluster and waits for the operator before the migration Job runs; upgrading a release that used one DSN no longer takes two upgrades. It needs access to the API server when egress is restricted (networkPolicy.egress.kubernetesApi); database.setup.clusterRoles.enabled: false turns it off.
• Scribe refuses to start when scribe.worker.heartbeat_interval_s is more than half of store.jobs.stale_after_seconds. Jobs run at least once: a job is run again after its worker is lost, so handlers must tolerate a second run (the built-in ones do).
• Reading values of Attributes tagged pii, confidential or restricted needs pii.read, confidential.read or restricted.read; other callers see them masked (design 10.2). Feeding vault values through imports and Sources needs the new Domain permission pii.import (only admin among the built-in roles), and vault key rotation needs vault.manage (installation administrators only). Give the read permissions through a custom role in keeper.auth.authz.custom_roles.
• Sensitive Attributes are left out of full-text search for every caller (see Changed): a search that matched a sensitive word no longer finds the Record.
• read.history.lineage_max_changes is removed (lineage no longer reads a window of changes). Configuration files refuse unknown settings: delete it.
• lorebase admin erase --value V refuses a literal value, which stayed in the shell history: use --value - or --value @FILE. A value erased before is stored erased when written again; vault.erased_values: allow keeps the old behaviour.
• Lexical search ranks a different set of matches by default: within search.common_term_docs (2,000) every match is ranked, beyond it the first search.common_term_candidates (1,000) in index order, and POST /search answers truncated_candidates: true when that cut applies. Before, the first store.reads.search_candidates (1,000) matches were ranked for every query. search.common_term_docs: 0 keeps the old behaviour. Keeper needs migration 0035.
• Compose and Helm run Scribe and Guide on the new image ghcr.io/aantenore/lorebase-ml (844 MB against 453 MB; fastembed and a baked embedding model). Deployments that mirror or pin images need it too; Helm imageMl.repository: "" keeps Scribe and Guide on image (kubernetes).
• Guide needs a writable state directory while its root file system is read-only: compose mounts the volume guide-data at /var/lib/lorebase/guide and Helm guide.persistence (an emptyDir by default); elsewhere the default is ~/.local/state/lorebase/guide, so mount a volume there or set guide.mode_memory_file: null (guide).
• Compose: make up, make logs, make down, make clean-stack and scripts/quickstart.sh add the files listed in COMPOSE_OVERLAYS after their own and stop before compose runs when one does not exist; make down stops every profile make up can start (quickstart).
• Migration 0036 (shared lineage maps) leaves the lineage maps of an existing install inline until python -m lorebase_store.storage apply runs as the maintenance role, so processes of 0.1.0 keep reading them during a rolling upgrade; the space comes back only with apply --compact --reindex (store schema notes). A new install shares maps from the start.
• store.storage.profile (compact, balanced, fast; compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile) decides which secondary indexes exist. The default builds the same indexes as before; compact drops the GIN index on Record values, the indexes of indexed-only Attributes and the lexicon's trigram index when storage apply runs (performance).
• Web app: the supply presets and dataset-specific placeholders are gone, example questions come only from domain.examples of the schema, and the "planned screen" placeholders (Explorer, Modeler, Review, Sources, Settings, the Record page's Run Operation and the Canvas diff overlay tab) are removed. The Retrieval lab no longer embeds the Explore panel; it links to Explore. Every user-facing string moved to apps/web/src/messages, so local text patches must move with it.

Added

• Search ranks every match only when it can afford to (design 8.4, performance). Lexical search without filters reads the word lexicon's counts of the query's lexemes, which bound its matches: within search.common_term_docs (2,000) every match is ranked; when every term is more common, only the first search.common_term_candidates (1,000) matches in index order are, and POST /search answers truncated_candidates: true so that callers narrow the query (Herald and Guide say so to the agent). Before, at most store.reads.search_candidates (1,000) matches were ranked for every query, without saying when that cut the ranking. search.common_term_docs: 0 keeps that behaviour. Measured at 1M and 10M Records with realistic text, 20,000 and 5,000 raised the p95 of title-word searches 3.4 times at 1M and 4.4 times at 10M, hence the lower defaults. Migration 0035 adds the security definer functions lb_search_bounded and lb_search_bounded_overlay, which report the cap.
• values: false on POST /neighborhood (agents): nodes come back as Record headers (kind, key, title) and edges as Link endpoints, for callers that need the structure only. The Python client takes values on NeighborhoodRequest; Herald's lorebase_neighborhood asks Keeper for it unless its values is true (its default leaves values out, as before, now Link values too); Guide's neighborhood tool takes values (default true).
• --vocabulary zipf for examples/supply/scripts/make_supply_data.py: names, titles, summaries and reasons drawn from 40,000 terms with Zipf frequencies plus product codes, the rest of the dataset unchanged. The 1M and 10M benchmarks use it (seed databases perf_seed_1m_zipf and perf_seed_10m_zipf; PERF_VOCABULARY=demo measures the earlier ones); tests/perf/focus.py compares search with and without the lexicon-sized ranking and the neighborhood with and without values on a seeded database.
• Benchmarks at 1M and 10M Records (performance): make perf-1m and make perf-10m start a Postgres of their own (tests/perf/compose.yaml, compose project lorebase-perf), seed the dataset once into a kept database and measure it in place; make perf-down deletes it. The 10M fixture loads a tenth through the commit pipeline and copies its materialized rows with renumbered keys (--replicas); an interrupted load resumes. The harness also times the agent trace through Keeper (--trace-iterations) and can bound the measured branch copy (--copy-batches, --no-pair-copy).
• COMPOSE_OVERLAYS (in .env or the environment): extra compose files, space separated, that make up, make logs, make down, make clean-stack and scripts/quickstart.sh add after their own, so make up PROFILES=assistant can start Scribe with a hosted model and its key file (deploy/compose/hosted-model.yaml) and an overlay of your own (quickstart, inference providers). A listed file that does not exist stops the command before compose runs. make logs (SERVICES="scribe guide" narrows it) follows the logs of the stack.
• Job metrics lorebase.jobs.reclaimed (by type and outcome requeued or failed) and lorebase.jobs.released (by type), and the alert LorebaseJobsReclaimed (operations) when running jobs lose their worker.
• Guide and Scribe embed text out of the box in compose and Helm. They run on a new image, lorebase-ml (target backend-ml of deploy/docker/backend.Dockerfile; ghcr.io/aantenore/lorebase-ml in releases): the backend image plus fastembed (ONNX Runtime, no PyTorch) and the embedding model EMBEDDING_MODEL (default BAAI/bge-small-en-v1.5) downloaded at build time, so POST /embed and Scribe's embedding jobs work offline with the read-only root file system. It is 844 MB against 453 MB for the backend image, so Keeper, the migrations and Herald stay on the small one. Helm: imageMl (empty imageMl.repository puts Scribe and Guide on image). The supply example schema gives Part and Supplier an embedding template, so semantic search over the sample returns hits once Scribe has run its first pass (quickstart).
• make up PROFILES=assistant (or make up-assistant) starts Guide and Scribe with the quickstart stack through the new overlay deploy/compose/assistant.yaml, which points the web app at /guide and lets Guide accept calls while Keeper has no authentication. make down now stops every profile make up can start.
• make example-documents (scripts/quickstart.sh documents) extracts the contract C-3099 and the four example documents with the rules tier and lists the schema suggestions. One document supports each suggestion once, below schema.suggestions.min_support (2), so a single upload listed nothing; five documents list ends, sole_source_parts and buyer (extraction).
• Record lineage names the Revision behind every value, however many changes came after it (agents): materialized Records keep that Revision per Attribute (migration 0032, mat_records.lineage_revisions), and rows written earlier fall back to the branch history. Lineage at a branch head is one statement, about 25% faster at 100k Records (performance). GET .../records/{key}/lineage takes at and answers from a snapshot of that Revision, like Record history; so do lorebase record lineage --at, the Python client and the MCP tool lorebase_lineage.
• The Modeler previews Derivations and Source expression mappings before they are saved (tour). POST /domains/{d}/derivations:preview checks the edited schema like a schema change and evaluates a candidate expression or aggregate Derivation on sample Records of its Kind, with Record-read permissions and masking (read.derivation_preview); POST /domains/{d}/sources:preview (schema.read) evaluates a Source's expression mappings on the rows sent (read.source_preview_max_rows). Nothing is written. Scribe now maps Attributes by expression (CEL over the row, row["name"] for any column) instead of refusing them. Python client: preview_derivation, preview_source.
• Path Derivations are now followed wherever Relations are: neighborhoods (when named in relations), Link counts of a Record, Operation links(), links.<path> in Derivation expressions and aggregate Derivations, which are updated incrementally when a Link of any step changes. Paths crossing a Kind the caller may not read are left out of neighborhoods and Link counts. mat_schema.path_fanout records the store.traverse.path_fanout each branch was computed with; a commit or a starting job worker that finds a different value rewrites the materialized paths and the derived values reading a path (design 5.8).
• Structured output modes are remembered per chat endpoint across restarts (inference providers). Scribe keeps them in the table inference_endpoint_modes; Guide keeps them in a locked JSON file (guide.data_dir, guide.mode_memory_file). Entries are keyed by an HMAC of the endpoint, model and output settings (inference.mode_memory_key is mixed in when set) and expire after inference.mode_memory_ttl_s (7 days). lorebase-scribe modes clear and lorebase-guide modes clear forget them at once.
• Schema suggestions from the CLI: lorebase schema suggestions list|accept|reject|merge (extraction). merge (also POST /domains/{d}/schema/suggestions:merge, schema.write, client merge_suggestions) folds near-duplicate Attribute suggestions already stored, with the same similarity as new ones. A Kind suggestion is no longer hidden by a same-named Attribute unless that Attribute is free text no Relation models and the evidence shows no independent entity (schema.suggestions.kind_independent_instances, kind_independent_attributes). An optional dedicated discovery pass (scribe.extraction.discovery.llm_pass, off by default) proposes Kinds and Attributes through the inference.llm.tasks.schema_discovery route.
• Reads at a past Revision (at) support filters, ordering, includes, aggregates, Link and lineage summaries, traversal, neighborhoods, search, Record history, health reports, context packs and explanations (design 8.6). They are served from a snapshot of the Revision, materialized on first use with the code of a branch rebuild and shared by later reads, so they answer what the head answered when that Revision was the head. read.time_travel.snapshot_max_entries (100000) bounds the size of a snapshot, larger Revisions are refused with 422 time_travel, and read.time_travel.snapshot_max_per_domain (4) how many are kept. History, explain and the health routes take at as a query parameter, traverse, neighborhood, search and context in the body.
• An overlay for trying browser sign-in on the compose stack (quickstart): deploy/compose/oidc-mock.yaml starts a mock OpenID Connect provider in Keeper's network namespace, turns on keeper.auth.sessions and puts the web app in session mode (WEB_AUTH_MODE=session); set LOREBASE_SESSION_KEY in .env. The provider signs in anyone: local use only. One sign-in, session, CSRF-protected sign-out and expiry were checked through the web service (validation).
• guide.http.default_origins, the browser origins Guide accepts while allowed_origins is empty. The compose file sets it to the origins of its web service, so GUIDE_ALLOWED_ORIGINS=[] now behaves as unset (Guide logs a warning) instead of refusing the web app.
• A values schema for the Helm chart (deploy/helm/lorebase/values.schema.json): every value of values.yaml is typed, settings with a fixed set of values are enumerated, and unknown keys are refused except under config and postgresql, so helm lint, template, install and upgrade reject a misspelled key or a wrong type. CI lints and renders the guide profile and checks that the schema refuses bad values.
• Connectors: SQL Server query Sources can end in ORDER BY (Scribe appends OFFSET 0 ROWS inside the derived table), and rowversion watermarks are kept as hex in the run checkpoint and bound as bytes. POST /domains/{d}/schema:infer accepts named MySQL, MariaDB and SQL Server sources (mysql, mssql), read through a bounded sample under the same allowed-source rules and profiled like records (new lorebase-keeper[mysql] and lorebase-keeper[mssql] extras, store.profiling.sql settings). JSON uploads can hold their records under a key with the records_path form field (Python client infer_files(records_path=...), web "Start from data"). The new POST /domains/{d}/schema:preview (schema.write, nothing stored; client preview_files) returns the first rows of uploaded files, and the web Source preview uses it for Parquet files.
• Explore in the web app (guide): a screen next to the Retrieval lab, and the same panel under the lab, that asks Guide a question about the Domain and shows each event of POST /explore as it streams in (reasoning notes, tool calls with compact arguments, result summaries and counts, errors, the answer and the totals). The request carries the caller's own credential (Bearer token, or the session cookie and CSRF header when Guide is served from the app's origin; guideUrl may be a path such as /guide). Stop aborts the request, a stream longer than VITE_EXPLORE_TIMEOUT_MS is stopped, and a 429 waits for Retry-After. Model output is rendered as plain text only; citations become links to Record pages when their Kind and key are valid, "View trace" opens the session in Agent activity, and the cited Records can be drawn on the Canvas. The supply Domain has five preset questions. Against the mock API the panel plays scripted explorations. New settings VITE_EXPLORE_MAX_STEPS, VITE_EXPLORE_TIMEOUT_MS and VITE_EXPLORE_MOCK_STEP_MS.
• Guide, an optional exploration assistant outside Keeper (guide). The new service lorebase-guide (package lorebase_guide) answers POST /explore with a stream of server-sent events: a small read-only agent reads the Domain through Keeper (schema, search, query DSL, Record, links, neighborhood, traversal, history, lineage and context packs), streams each tool call, a short result summary and the model's reasoning notes, and ends with an answer citing the Records it was shown, or "Not found in the graph." Every Keeper request carries the caller's own credential (Bearer token or API key, or the web session cookie with its CSRF token, checked against Keeper), so RBAC and masking apply exactly as for the user; Guide holds no credential and has no write tool. Each exploration is an agent trace session (X-Lorebase-Session, tool header, reasoning posted as notes). Budgets for steps, tokens per call, tokens per exploration and wall-clock time, per-credential rate and concurrency limits, and an Origin allow-list are in the new guide settings. POST /embed returns the vector of a query or document text with the configured embedding model and prefixes, the same vector Scribe stores for the same document text. Besides Scribe, Guide is the only package that may use lorebase_inference; the import contracts keep Keeper and Herald model-free and forbid them to import Guide. It runs from the backend image (compose profile assistant, Helm guide.enabled, off by default), and the web app reads its address from the runtime configuration guideUrl (VITE_GUIDE_URL).
• The Python client refuses empty, . and .. path segments (Domain ids, Kind names, Record keys, relation names) and branch names with such parts, raising ValueError before any request: an HTTP client resolves them, which sent the request to another route.
• A quickstart that works from a fresh clone (quickstart). make env writes .env with generated database passwords, make up builds the image from the checkout and waits until Keeper is healthy, make example-data loads the supply sample and merges it into main, make down and make clean-stack stop the stack and delete its data. The quickstart adds an agent connected over MCP, a real model (Ollama in compose or on the host, or a hosted endpoint whose key is mounted from a file by deploy/compose/hosted-model.yaml), cleanup, a troubleshooting table and measured times. .env.example has COMPOSE_PROJECT_NAME, so two stacks on one machine no longer share containers and volumes, and Scribe and inference-check resolve host.docker.internal on Linux too.
• Ranked partial matching in lexical and hybrid search (design 8.4). When q as written (every term) finds fewer than search.partial.min_results Records (1), Records holding some of its terms follow, after every full match: the words and quoted phrases of q joined by OR, negations kept for each, stop words of search.partial.stop_words (english) left out, scored weight * (ts_rank_cd + term_bonus * terms held) with typo corrections applied per term. Each hit reports match (all or partial, absent for hits found by vector only); the field is new and optional. Descriptive queries such as "ring that stops oil leaking" found nothing before, so hybrid search ranked them by the vector alone; the O-rings now come first lexically too (validation). Measured at 100k Records in performance.
• Chat endpoint options for hosted providers (inference providers): inference.llm.headers sends extra HTTP headers (values accept secret:// references, for example an anthropic-workspace-id), extra_body merges extra request fields (reasoning_effort, thinking, ...), json_schema_strict sets response_format.json_schema.strict, and temperature: null leaves the temperature out for models that refuse it. All four also work per task in inference.llm.tasks; a task on another base_url never inherits the headers. Header values are scrubbed from error text like the key. Anthropic's OpenAI-compatible endpoint and local Ollama models were run against the example documents; setup and results are in validation.
• Agent trace (agent trace, keeper.trace, off by default). Requests that opt in (X-Lorebase-Trace: on, or a listed principal or API key, or trace_all, which leaves out the web app's own requests) become steps of the session they name (X-Lorebase-Session) or, without one, of an inferred session (same principal, credential and user agent, calls less than inferred_session_idle_s apart, marked inferred) in the new table agent_trace_steps (migration 0023): tool (X-Lorebase-Tool), route, status, latency, response cache use, the query as structured data (filter values kept only on Attributes that are not sensitive, capture_filter_values), result counts, search corrections, context pack budget and the Records and Links the answer touched, never values; with X-Lorebase-Trace: debug and the new permission trace.explain, query plan summaries. Agents attach notes with POST /api/v1/trace/sessions/{id}/notes; GET /api/v1/trace/sessions and GET /api/v1/trace/sessions/{id} list sessions and replay one. A principal sees its own sessions; the new permission trace.read (held by admin) shows other principals' steps in a Domain, with their query and note text only to readers holding every sensitivity clearance there (withheld otherwise). Steps are written off the request path in batches, capped per session and deleted after retention_days (7). Herald forwards the session id of the MCP client or session (none when there is none), the tool name and a traceparent on every Keeper call; the Python client takes session_id, trace and tool and has a trace resource; the CLI has lorebase trace list|show; the web app has an Agent activity screen with the timeline and the touched subgraph on the Canvas, step by step.
• Retrieval lab in the web app (retrieval lab): one question, two parameter sets side by side (mode, weights, thresholds, limits), lexical results with scores and corrections, semantic and hybrid ones when a query vector is pasted or an embedding service is configured (guideUrl in the web app's runtime configuration; the browser calls it, never Keeper), the context pack an agent would get with why each entry is in it and its tokens against the budget, and the involved Records on the Canvas. POST /search takes tuning (lexical_weight, semantic_weight, rrf_k, trigram_weight, and the work bounds typo_candidates and candidates, which may only be lowered, and trigram_threshold, which may only be raised) in place of the search settings for one request.
• Schema suggestions from document extraction (extraction). Extraction jobs report what they see outside the schema: **Label:** value fields and table columns no Attribute matches, headings naming a thing by a code no Kind knows, spans and relations of the new scribe.extraction.discovery.span_labels and relation_labels, and, at tier 2, an unlisted list in the answer schema plus records, values and links outside the schema, all with verified evidence. Observations are matched against the schema (normalized singular names, synonyms, similarity), never kept when they match a sensitive Attribute, kept without evidence when the quote holds a sensitive value, and aggregated per Domain by fingerprint into suggestions with support, documents, examples, samples and, for Attributes, the type schema inference gives the samples; extracting a document again does not count it twice. GET /api/v1/domains/{domain}/schema/suggestions lists them (permission document.read; pending ones from schema.suggestions.min_support, at most max_suggestions) and POST .../schema/suggestions:decide rejects, resets or accepts them: accepted suggestions become one schema change set on a new branch (keeper.schema_suggestions.branch_prefix) proposed into the default branch, new Kinds and Relations as drafts. The web app has a Schema suggestions tab in Review and the Python client schema.suggestions, iterate_suggestions and decide_suggestions. New settings schema.suggestions, scribe.extraction.discovery and keeper.schema_suggestions; migration 0021 adds the schema_suggestions table. The tier 2 cache key changes when discovery is on, so the first run after upgrading calls the model again.
• lorebase schema import --from linkml --apply merges into the current schema of the Domain instead of replacing it: Operations, Derivations, Sources and the fields LinkML cannot carry (derivation, authority, indexed, ...) are kept, elements match by id or name, and Kinds, Attributes, Relations and Traits missing from the LinkML file are kept with a warning unless the new --prune flag removes them; --dry-run shows the resulting change set (LinkML).
• LinkML schema import and export (LinkML). lorebase schema export --format linkml translates the schema of a Domain, or a local document with --file, into a LinkML schema: Kinds become classes, Traits mixin classes, Attributes class attributes with their range, required, identifier, constraints and ifabsent, enums LinkML enums, and each Relation a pair of slots linked by inverse whose multivalued gives the cardinality. Element ids, sensitivity tags, search settings and the types LinkML lacks travel as lorebase.* annotations, Operations, Derivations and Sources as schema annotations, so importing the export gives back the same schema. lorebase schema import FILE --from linkml prints the Lorebase document (or writes it with -o), --apply and --dry-run send it to a branch like schema apply; what it leaves out (Operations, Derivations and Sources, is_a inheritance, which it flattens, abstract and tree_root classes, inlined objects, rules, any_of, enum meanings, custom type constraints) is reported as warnings with their path, and classes without slot descriptions or a title attribute are imported as draft Kinds. The translation needs PyYAML, in the new optional extra lorebase-cli[linkml].
• Path Derivations work (design 5.8). A path Derivation is a virtual Relation (A -r1-> B -r2-> C read as A -> C) usable by its api_name or inverse_api_name wherever a Relation name is accepted from a Kind: Links of a Record and of a Relation, via and count predicates, include and traversal. By default its Links are composed at read time from the steps' Links (overlays and row-level security included); every later step follows at most store.traverse.path_fanout Links (100) per intermediate Record, smallest far keys first. With materialized: true the Links are stored in mat_links under the Derivation id and kept current in the commit transaction: only the start Records a changeset can affect are recomputed (above store.materialize.derivation_max_incremental_records, the whole path), schema changes and rebuilds rewrite them, and verify(..., derived=True) checks them. Both modes give the same Links, also at past Revisions, where they are composed from the tree. Following a path needs read access to every Kind it crosses. Path Links carry no values.
• Trait queries at a past Revision (at): POST /query with trait lists the Records of every implementing Kind from the tree, in the head's order and with the head's cursors; filters, ordering by other fields, includes and aggregates stay refused there (422 time_travel), as for Kinds.
• MySQL/MariaDB and SQL Server Sources: the types mysql_table, mysql_query, mssql_table and mssql_query read a table or a query like the Postgres ones (snapshot or incremental with a watermark compared in SQL, batches of fetch_rows, connection by allowed secret:// reference, statement_timeout_ms and connect_timeout_s in the new scribe.sources.mysql and scribe.sources.mssql settings) and can be profiled for schema inference. The drivers are optional extras and a run without one fails naming it: lorebase-scribe[mysql] (asyncmy; the session is read-only) and lorebase-scribe[mssql] (pymssql in worker threads, no ODBC driver to install; use a login that can only read, since SQL Server has no read-only transaction). Schema validation asks the new query types for a query, and the web Modeler lists the types and keeps them on a round trip (scribe).
• Operation side effects (design 10.5, agents). A stored Operation's side_effects (unique snake_case names of at most 64 characters, now checked by schema validation as schema.api_name_invalid or schema.api_name_duplicate) make each committed execution write one operation.<name> outbox event in the transaction of its Revision, invocation record and audit entry, so the event exists exactly when the execution committed; dry runs, refusals, executions that need review (agent task writes that open a Proposal included) and idempotent replays write none. The payload names the execution, carries the parameters as recorded (possibly sensitive ones ***) and up to keeper.operations.side_effect_max_edits (100) edits. Webhooks subscribe with operation.<name> or operation.*; a webhook created without events now gets revision.committed and operation.*. Sensitive values in events reach a webhook only when it was created with matching clearances (pii, confidential, restricted; creating one needs that clearance permission on the Domain), others get webhooks.masked_value; vault values are stored as references and opened when the delivery is sent, so an erased one arrives as webhooks.erased_value. Every attempt of an event carries Idempotency-Key: evt_<sequence> (webhooks.idempotency_header): delivery is at least once and unordered. Each consumer of the outbox now keeps its own delivery state (outbox_pending, filled by a trigger in the writing transaction), so a second consumer no longer loses events to the webhook worker, and an event whose transaction commits late is not skipped. Outbox events are pruned once every consumer handed them off and they are older than the new store.outbox.retention_s (7 days; webhooks.retention_s now applies to deliveries only), or after store.outbox.max_age_s (30 days) in any case, store.outbox.prune_batch (10,000) at a time. New metrics lorebase.outbox.handoffs, lorebase.outbox.pruned and lorebase.operations.events. Migration 0020 adds outbox_consumers, outbox_pending and webhooks.clearances, moves undelivered events to the webhooks backlog and drops outbox.delivered_at and outbox.attempts.
• Rate limits shared by every Keeper: keeper.limits.rate.backend: postgres keeps the token buckets in the application database (the UNLOGGED table rate_limit_buckets), so several Keepers enforce one limit instead of each its own. A Keeper leases a few tokens per query (postgres.lease_tokens, at most a quarter of a bucket) and serves requests from them, so most requests cost no query and the Keepers together never exceed a limit. Failed-authentication buckets are written on every failure and fail closed (429) when the database does not answer within postgres.timeout_s; the other buckets follow postgres.on_error (local, the default, enforces them in the process meanwhile; allow; deny). The default backend is now called local (memory still works). The Helm chart sets postgres when keeper.replicaCount is above 1 or the HPA is on (keeper.rateLimitBackend overrides it); compose reads KEEPER_RATE_LIMIT_BACKEND, to set with the scale-out profile. New metric lorebase.ratelimit.shared.queries (operations).
• Auditing of reads (keeper.auth.audit.reads, off by default). Successful reads of authenticated callers are recorded in audit_read_log, a table with hash chains of its own (per Domain and installation-wide) so that sampled, droppable read records never touch the chains of actions. sample_rate, principal_types and kinds choose which reads are kept, except that a read that showed or masked a sensitive value (pii, confidential, restricted, vault values included) is always recorded, with sensitive: [shown|masked]. Records are queued and written in batches off the request path; a full queue or a failed batch drops records, counted in lorebase.audit.reads.dropped. python -m lorebase_keeper audit verify --reads verifies the read stream and the daily audit_verify job checks both streams (operations).
• Signed checkpoints of the audit chains outside the database (scribe.audit_checkpoints, off by default). An audit_checkpoint job, every interval_s (one hour), signs the newest entry of every chain with an Ed25519 key and stores it in a directory (never replacing a file), at an HTTP endpoint or in an S3-compatible bucket (PUT signed with Signature Version 4), with the address checks and pinning of webhook targets. python -m lorebase_keeper audit verify --checkpoints PATH --checkpoint-key PUB reports checkpointed entries that are missing or changed, so a table owner who cuts the end of a chain or recomputes one is detected. New metric lorebase.audit.checkpoints (operations).
• The five-day proof scenario runs in CI. The End-to-end workflow builds the backend image from the checkout (Docker layers cached), starts the demo stack with the deterministic fake inference provider and runs examples/five_day_demo/run.sh on pushes to main, nightly, on demand and on pull requests that change examples/, packages/, deploy/compose/ or deploy/docker/; on failure it uploads the stack logs and the run transcript. Every day of the demo now ends with assertions (demo.py assert-*: schema applied, import counts, merge with resolved Conflicts, Source runs and webhook, agent writes through Proposals and the audit log, restore equals live), a failed one exits non-zero, and the run prints the seconds each day took. Day 2 gained a document: a contract extracted with the deterministic tiers into a Proposal of items that is reviewed and merged, and the agent of day 5 now also opens a Proposal of its own with task and intent (its key gets the editor role on agent/*). DEMO_NO_BUILD=1 makes stack.sh up use a prebuilt image (demo).
• Personal data vault and erasure (design 13.5). An Attribute tagged pii with vault: true stores its values encrypted (AES-256-GCM) in the vault table, and Record objects hold a token {"$vault": "vlt_..."} instead; hashes, diffs and merges compare tokens, which stay the same for the same value of the same Attribute of the same Record. POST /api/v1/domains/{domain}/vault/erase (new permission pii.erase, admin only among the built-in roles), lorebase admin erase KIND --key K or --attribute A --value V, and vault.erase_record / erase_value in the Python client erase a data subject's values on every branch and Revision, in one transaction with an audit entry that never holds the value; erasing again changes nothing, and reads show keeper.reads.sensitivity.erased_value ([erased]). Keys come from the new vault settings (token_key, encryption_key, key_id, previous_keys), the same for every process that commits; without them, writes of vault values are refused (value.vault_unavailable). Vault Attributes cannot be unique, indexed, derived, references, titles, primary keys or foreign keys, and vault storage cannot be turned on or off on an existing Attribute (schema.vault_invalid). References are accepted in writes only for the Record they belong to, and Conflicts over vault values are resolved by picking a side. Bundles carry tokens, never vault values, and their import refuses plain values in vault Attributes. lorebase admin erase --value - reads the value from standard input. Migration 0019 adds the columns that bind a vault row to its Record. The Modeler keeps vault when it edits a Kind.
• Sensitivity masking on reads (design 10.2). Values of Attributes tagged pii, confidential or restricted are shown only to callers holding the new permissions pii.read, confidential.read or restricted.read (Domain-wide, on the branch read, or narrowed to the Kind; admin holds them, custom roles grant them); other callers get keeper.reads.sensitivity.masked_value (***), or nothing with omit: true. This covers Records, Links, query includes, traversals, neighborhoods, history and lineage values, diffs and changesets, explain and context packs, Conflicts and stored resolutions, and Operation edits, so the CLI, the Python client and the MCP server see the same. Filtering, ordering or grouping by a sensitive Attribute needs the clearance (403 sensitive_attribute), and by a vault Attribute is refused (422 vault_attribute). Responses are cached and tagged per clearance. Clearances come from Domain-wide grants or grants narrowed to the Kind (by its id on the default branch), not from grants narrowed to a branch; the tags of the default branch, carried through Derivations, apply to every Revision and branch read; removing a tag on any branch, by a schema change or a merge, needs its clearance (403 sensitivity_lowered), and writers without a clearance check (agent auto-merge, Source merges) cannot. Derived Attributes and the step values of Operations must carry the tags of what they read, preconditions cannot read sensitive Attributes (schema.sensitivity_inherited), sensitive Attributes cannot be primary or foreign keys, and uniqueness violations on them no longer name the other Record. Extraction items are masked too, Operation invocation records and Proposal descriptions no longer keep parameter values that may be sensitive, and Source run errors are shown only to callers cleared for every tag of the schema.
• Copies and deletions of large branches run in the background, so they no longer hit database.statement_timeout_ms (at 1M Records a copy took 85 to 95 s and failed through the API). A copy branch whose source holds more than store.materialize.async_copy_min_rows Records and Links (10,000) is created at once as an overlay of its source, readable and writable right away, and a store.materialize job copies its rows in batches of job_batch_rows, one transaction each, resuming from its cursor after a failure, then switches the branch to a full copy. Commits on the branch and on its source stay correct during the copy and wait for at most one batch; a schema change on the branch is refused with 409 branch-materializing and Retry-After until it is done. Deleting a branch with more than async_delete_min_rows rows hides it at once and a store.branch_delete job deletes its rows in batches. At most job_concurrency of these jobs run at once across workers. Branch responses have materialization_status (pending, running, done, failed) and materialization_progress; lorebase branch create --mode copy --wait waits for the copy, and lorebase branch list|get show it. Overlays promoted because their base moved too far are copied the same way. New metrics lorebase.store.branch_jobs.rows, .runs and .batch.duration. Migration 0017 adds refs.deleted_at and the branch_materializations table (operations, store schema notes).
• The embedding model of a Domain is advertised to agents: GET /api/v1/domains/{domain}/schema has a search block with the search modes and, when a Kind has search.embedding_template, embedding (model, dims) read from search.semantic.model and store.embeddings.dims; lorebase_schema shows it as a SEARCH line, and the Python client has SchemaDocument.search.
• inference.embeddings.query_prefix and document_prefix (empty by default) for models trained with prefixes, such as E5: lorebase-inference embed --query TEXT puts the query prefix before the text, and Scribe puts the document prefix before every embedded Record text. Changing the document prefix makes the next pass embed every text again. lorebase-inference embed TEXT without --query no longer adds anything, which is what it did before prefixes existed (inference providers).
• Garbage collection deletes the embeddings no branch head references (for example the vectors of Records that only a deleted branch held); python -m lorebase_store.gc --dry-run and the gc job report their count as embeddings (operations).
• The Modeler edits search.embedding_template and search.embedding_sensitive on the Kind form, checks the placeholders against the Kind's Attributes (unknown, derived and sensitive ones) before a change is sent, and follows an Attribute rename in the template.
• Embedding jobs for semantic and hybrid search. A Kind opts in with search.embedding_template in the schema ({attribute} placeholders by api name, plus {_kind} and {_key}; sensitive Attributes only with search.embedding_sensitive: true; invalid templates are reported as schema.embedding_template_invalid). Scribe's scheduler queues an embeddings job per Domain every scribe.scheduler.embeddings_interval_s (60 s): it embeds, branch by branch, the Records that lack a vector of the configured inference.embeddings model, re-renders the Kinds whose template changed, skips texts whose hash did not change and copies the vector of an identical text instead of calling the model. Vectors stay keyed by content hash, so overlay and copied branches only add work for the Records they change. Batching, a row budget per job with resumable continuations, call pacing and retries come from the new scribe.embeddings settings; the model's dimension is checked against store.embeddings.dims before the first call. lorebase-scribe embeddings DOMAIN [--branch B] [--reindex] [--prune] backfills, reindexes and, after a model switch, deletes the vectors of other models. New metrics lorebase.scribe.embeddings.records and lorebase.scribe.embeddings.call.duration. Migration 0014 adds embeddings.text_hash and the embedding_cursors table (scribe).
• Query vectors on the caller's side, since Keeper and Herald never call a model: lorebase-inference embed TEXT prints the vector of the configured embedding model, and lorebase search TEXT --vector-file FILE (- for standard input) sends it for hybrid search (inference providers).
• Per-Record history and lineage: GET /api/v1/domains/{domain}/kinds/{kind}/records/{key}/history lists the Revisions that changed a Record on a branch, newest first, with the changed Attributes, their provenance and, with values=true, the values before and after; GET .../records/{key}/lineage tells per Attribute where the current value came from (provenance, Revision, Source, Operation, extraction evidence, Proposal). Both read one index range of the branch history, are authorized and cached like Record reads, and are bounded by the new read.history settings. Also in the Python client (records.history, records.lineage), the CLI (lorebase record history|lineage), the MCP server (lorebase_history, lorebase_lineage) and the History tab of the Explorer Record page.
• Merge policies authority and newest_wins (design 6.10). authority (globally or per Attribute in authority_attributes) is a ranked list of origins such as source:<id>, principal:<id>, actor:<id> or origin:extraction: on an attribute value Conflict the side whose last change comes from the higher ranked origin wins, and schema_authority: true ranks an Attribute's schema authority Source first. newest_wins (or newest_wins_attributes) takes the side whose last change has the later provenance time, equal times going to the greater provenance id. Both read the Lineage of both branches, leave the Conflict to a person when they cannot decide, and record why they decided in the attempt's auto_resolutions[].details, which the Review screen shows. Set them in the Domain's merge_policy setting; Source merges apply them too.
• Sign-in for the web app with Keeper as backend-for-frontend (keeper.auth.sessions, off by default; needs an oidc auth mode). GET /api/v1/auth/login runs the OpenID Connect authorization code flow with PKCE against the configured provider, GET /auth/callback checks state, nonce and the ID token and sets an opaque HttpOnly, SameSite=Lax session cookie (__Host- prefixed and Secure unless cookie_secure: false), GET /auth/session returns the user and the session's CSRF token, and POST /auth/logout ends the session and returns the provider's end-session URL. Sessions live in the new web_sessions table (migration 0013; only a SHA-256 of the session id is stored, provider tokens are encrypted with sessions.encryption_key) with idle and absolute timeouts, a new id on every sign-in and refresh-token checks at the provider. A session maps to the same principal, groups and grants as the user's bearer tokens; POST, PUT, PATCH and DELETE with the cookie need the CSRF header, a trusted Origin and no cross-site Sec-Fetch-Site. API keys, bearer tokens and auth.mode: none work as before. The web app gets VITE_AUTH_MODE=session (authMode in /config.js): a sign-in screen at /login, a sign-out button, cookie requests with the CSRF header, and a redirect to sign-in on 401 (deployment).
• Operation, Derivation and Source editors in the web app Modeler (tour). Operations: typed parameters, preconditions, steps with their Kind or Relation and values, review policy, permissions, side effects, rate limit, and a dry run of the saved Operation on the Branch. Derivations: expression, aggregate and path variants, with the computed Attribute picked from the schema. Sources: the fields of each type (records_path, flatten_separator, http headers and pagination for http_json), secret:// references only for connections and credential headers, key, Attribute and Relation mapping, sync mode with watermark, schedule and merge policy. Expressions are edited in a CEL editor, and the findings of Keeper's dry run, the CEL type check included when schema.expressions_enabled is on, appear next to the field. Edits go through the same change set, Branch and Proposal flow as the other editors, and keys a form does not edit stay as the YAML has them.
• Helm chart values for the web sign-in: config.keeper.auth.sessions settings, the OIDC client secret and the session encryption key read from the Secret (secrets.keys.oidcClientSecret, secrets.keys.sessionEncryptionKey, or secrets.values.* for a chart-managed one) and wired as secret://env references, web.config.authMode: session, render-time checks, and the ci/sessions-values.yaml overlay (Kubernetes).
• Audit chain verification (operations). python -m lorebase_keeper audit verify DOMAIN|--installation|--all recomputes the hash chains of the audit log with the maintenance DSN, prints the first entry that does not verify (or JSON with --json) and exits 1 when there is one; --after-seq resumes from the last_seq of an earlier run. GET /api/v1/domains/{domain}/audit/verify (permission audit.read) and lorebase audit verify (exit code 2 on a broken chain) check one Domain's chain, and the Python client has audit.verify. Entries are read in batches, so memory does not grow with the log. Scribe queues an audit_verify job of every chain each scribe.scheduler.audit_verify_interval_s (one day; 0 disables it), counted in the new metric lorebase.audit.verifications by outcome, and the LorebaseAuditChainBroken alert fires when a chain is broken.
• A threat model: STRIDE per trust boundary (web sign-in, API keys and OIDC, authorization in Keeper, Scribe connectors and document intake, Herald, Postgres roles and row-level security, webhooks, inference providers, the release pipeline), each mitigation with the file that implements it, and the residual risks in order of priority.
• An API stability policy for /api/v1: what is stable, which changes are compatible or breaking, the deprecation process with Deprecation and Sunset headers (none is deprecated yet, so Keeper sends neither), client compatibility, MCP tool stability and the lorebase.schema/v1 format. It binds from 1.0; before it, breaking changes are named in the changelog.
• Webhook clearances can be changed: PATCH /domains/{domain}/webhooks/{id} with {"clearances": [...]} replaces them, audited as webhook.update with the clearances before and after. Each clearance added needs its clearance permission on the whole Domain, as at creation; removing one needs only domain.settings. The Python clients have webhooks.set_clearances (agents).
• Storage profiles (performance): store.storage.profile (compact, balanced, fast; compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile and ci/storage-<profile>-values.yaml with matching Postgres settings) decides what the database keeps besides the rows: shared or inline lineage maps, the GIN index on Record values, the per-Attribute indexes of indexed Attributes, the trigram index of the typo lexicon, half or full precision for the HNSW index, TOAST compression and the fill factor of mat_records; every other key of store.storage overrides one choice. Every feature works in every profile. python -m lorebase_store.storage status|apply|prewarm shows, applies (batched, resumable, reads go on; --compact --reindex moves rows into free space and gives pages back) and prewarms (pg_prewarm, autoprewarm with shared_preload_libraries). The sizing guide gives the memory that keeps the agent read set resident at 1M, 10M and 100M Records per profile. At 1M Records the hot set of the agent read path went from 12.5 GB to 5.5 GB (balanced, after apply --compact --reindex); at 10M Records the hot set is estimated at 49 GB against 57 GB measured before (the 10M after-run is pending).
• Shared lineage maps (migration 0036, store schema notes): mat_records.lineage, lineage_revisions and mat_links.lineage hold a reference to a map stored once per Domain (mat_lineage_maps) instead of the map; readers resolve it with lb_lineage. Record rows are about 55% smaller. tests/perf/working_set.py measures relation sizes, pages per read and buffer residency; tests/perf/halfvec_recall.py the recall of a half-precision HNSW index.
• Example questions in the schema: domain.examples of lorebase.schema/v1 lists optional {label, question} pairs (at most 20, unique labels; design 5.13). They are part of the Domain element, so changing them is a schema change, and GET /domains/{d}/schema returns them in its domain block. The web app offers them in Explore and the Retrieval lab only when present. Schemas without them load and hash as before. examples/supply/schema.yaml declares the five questions of the example agent.
• VITE_DOCS_URL (docsUrl in /config.js): the documentation site behind the web app's "Learn more" links (default https://aantenore.github.io/lorebase, none hides them). LOREBASE_GUIDE_PROXY makes pnpm dev and pnpm preview proxy /guide to a guide service.
• pnpm screenshots --live --keeper <url> --domain <id> --guide <url> takes the tour pictures from a running stack read only: the existing Domain as it is, a real Explore answer and its trace (apps/web).

Changed

• .env.example no longer sets GUIDE_ALLOW_ANONYMOUS=false (the compose default is still false); with make up PROFILES=assistant it defaults to true. A value set in an existing .env is kept. The UV_SYNC_ARGS comment no longer lists fastembed, which the lorebase-ml image carries.
• Agent trace: trace_all no longer traces the web app's own requests (agent trace). The web app sends X-Lorebase-Client: web on every Keeper request; Keeper leaves out of trace_all the requests carrying it (keeper.trace.client_header, web_client) and those authenticated by a browser session cookie that name no session. keeper.trace.include_web: true traces them again. Guide and Herald sessions, header opt-ins and listed principals or keys are traced as before. The header is added to the CORS allowed headers when tracing is enabled.
• Agent trace: values compared by filters are kept on Attributes that are not sensitive (agent trace), so a timeline shows single_sourced = true instead of {"redacted": "boolean"}. The new keeper.trace.capture_filter_values (non_sensitive by default, or none) decides; a field is kept only when the read's own field check finds it not sensitive under the effective tags used for masking (through Traits and Derivations; title and key by their title and primary key Attributes). Kept values are scalars or lists of scalars, cut to max_text_chars and the new max_filter_value_items (20); they are shown like the rest of the query, to the caller and to readers cleared in the Domain only. Set capture_filter_values: none for the earlier behavior.
• Snapshot builds for reads at a past Revision are bounded: one build per Domain at a time, read.time_travel.snapshot_max_concurrent_builds per process with snapshot_build_wait_ms of waiting (503 with Retry-After), snapshot_builds_per_minute and snapshot_build_burst per principal (429), snapshot_max_history_revisions of history (422), and a snapshot_min_age_s grace period before a snapshot may be dropped for another; requests are validated before a build, and snapshots use the sensitivity tags of their Revision's schema.
• Large branch copies never run in one transaction any more (store schema notes). A copy above store.materialize.async_copy_min_rows that cannot start as an overlay (a copy of an overlay, from an older Revision, or with the overlay cap reached) is created building (migration 0031) and a store.materialize job writes its rows from the tree of its start Revision in batches; meanwhile its head is read like a past Revision and commits into it answer 409 branch-materializing. Overlays over a base above that size are no longer promoted inline: their limits queue the batched copy, and a rebuild of the base detaches them and builds them from their trees.
• Schema changes on an overlay whose background copy is queued or running are accepted instead of answering 409 branch-materializing: the change applies to the rows the commit writes and the copy job reconciles derived values, path Links, titles and search vectors in batches after its switch. A schema change on an overlay over a large base queues that copy instead of promoting the overlay inline.
• Background copy and deletion batches size themselves by the time they hold the branch lock: from store.materialize.job_batch_min_rows (100) up to job_batch_rows, aiming at job_batch_target_ms (200 ms) and at job_batch_contended_ms (50 ms) once a commit waited for the lock, so a commit on a branch being copied no longer waits up to 0.5 s for a batch. New metric lorebase.store.branch_jobs.batch.rows.
• Schema suggestions no longer keep evidence or samples that hold a vaulted value of any vault Attribute of the Domain (extraction). Extraction compares keyed fingerprints of the quote's word sequences with the live vault rows and opens nothing. It fails closed: a quote past discovery.vault_match_max_text_words, vault_match_max_text_chars or vault_match_max_candidates, or any quote of a Domain with vaulted values when Scribe has no vault keys, loses its evidence. A suggestion without evidence looks like one that never had any; the job report no longer carries suggestion_evidence_withheld, and operators get the counter lorebase.extraction.evidence_withheld.
• mypy no longer carries a project-wide override for pyarrow (reported as an unused section when only some packages are checked): the three optional pyarrow imports ignore its missing type information where they are.
• Sensitive Attributes are left out of full-text search for every caller: their words are no longer in search vectors, the lexicon, corrections or snippets (a searchable sensitive Attribute gets the warning schema.sensitive_attribute), and a Kind whose title Attribute is sensitive has Records without a title. Semantic and hybrid search use the vectors of a Kind that embeds sensitive Attributes only for callers cleared to read them. Rows materialized before keep their titles and search vectors until the Kind's definition changes on that branch (any edit of the Kind, such as its description, recomputes them); materialize.verify reports such rows until then.
• Validation messages of schema migrations no longer quote values of sensitive Attributes.
• Deleting a branch above store.materialize.async_delete_min_rows rows returns 204 at once while its rows are deleted in the background; until then its name stays taken, and creating a branch with it answers 409 already-exists with Retry-After. Copies above async_copy_min_rows answer with materialization: overlay until their copy is done. Set both thresholds very high to keep copies and deletions in the request transaction.
• Typo-tolerant search matches words instead of whole titles. Each branch keeps, per Kind, the distinct words of its search vectors (mat_lexicon, kept by triggers on mat_records in the writing transaction; an overlay adds its base's), and a query word the searched Kinds do not hold is corrected to their closest words by trigram similarity (negated words are left alone), then searched as (word | correction ...). Hits of the query as written rank first, the others after them at search.trigram_weight times their rank. POST /search responses have corrections (word, replacements; empty when nothing was corrected), also in the Python client (SearchResult.corrections) and in lorebase_search. search.trigram_threshold is now the least similarity of a correction (default unchanged, 0.3), and search.typo_candidates (3; 0 turns it off) and search.typo_min_length (4) are new. Words are lexemes of the text search configuration, so a stemming configuration reports stems. Migration 0018 adds the table, its triggers and backfill (writes to mat_records wait while it reads every search vector), the read functions lb_search_corrections and lb_search_corrected (grant them to an application role created later with grant_read_functions, as before), and drops the title trigram index of 0011. At 1M Records typo-tolerant search takes 7.6 to 14 ms at p95 instead of 116 to 123 ms; the triggers add about 5% to a commit of 1,000 changes at 1M Records and about 12% at 100k (performance, design 8.4).
• The performance harness (tests/perf/bench.py) reports p99 next to p50 and p95, times Record history and lineage, and with --seed-database NAME keeps the loaded dataset in its own database and measures a copy of it, so a run at 1M Records loads once. The 1M-Record run is in performance: every read is within budget except typo-tolerant title search (about 120 ms at p95 against 50 ms), the 2-hop neighborhood on branches other than main (29 to 34 ms against 30 ms) and lineage against the 10 ms proposed for it; creating or deleting a copy branch at that size takes longer than the default database.statement_timeout_ms.
• The web app Canvas lays large graphs out in a web worker instead of on the main thread: the graph is drawn at once and refined while it is on screen, the layout stops when it settles or at a time budget and can be paused, and large graphs use Barnes-Hut, hide edge labels and most node labels, and hide edges while panning. On a generated graph of 5,000 nodes and 10,000 edges the first render takes about 0.6 s instead of 1.7 s (performance). The limits are configurable (VITE_CANVAS_*), and pnpm perf:canvas with VITE_MOCK_PERF_NODES measures a generated graph.
• Herald gives each Keeper request its own span: a request hook on the Keeper client sends a traceparent with a fresh span id whose parent is the span of the MCP tool call, in the caller's trace, so the requests of one tool call are told apart in the agent trace and in tracing backends. When an OpenTelemetry span is active in Herald, each request is a child of that span instead; with herald.trace.enabled off no traceparent is sent. The Python clients take event_hooks, passed to httpx (agent trace).
• Operation executions that need review now emit their operation.* events when their Proposal merges (agents). An agent task write keeps its events in operation_deferred_events (migration 0028); the merge of a Proposal from the task branch writes those whose Revision the merged source head contains to the outbox and deletes them, in the merge transaction, so each is written once. They name the target as branch, the merged head, source_branch and proposal, and are counted in lorebase.operations.events when the merge commits. Events of a task branch deleted unmerged are dropped, and so are those older than store.outbox.max_age_s.
• The web app is domain-agnostic and its copy is shorter. No screen names the Kinds, Records or questions of a dataset (the supply presets, placeholders such as a question about bearings in Torino, and example keys are gone; mock data stays in the mock fixtures), and no description puts the Domain's name into its text. Every user-facing string lives in apps/web/src/messages: a heading and at most one short line per screen, with "Learn more" links into the docs, consistent terms and no internal jargon. src/test/domain-agnostic.test.ts fails on dataset identifiers or an interpolated Domain name outside the mock fixtures and tests. The "planned screen" placeholders (Explorer, Modeler, Review, Sources, Settings, the Record page's Run Operation and the Canvas diff overlay tab) are removed.
• Web app layout: shared primitives (PageHeader, Section, Card, CardGrid, FormRow, FormAction) give every screen the same spacing, equal-height cards and buttons on the line of their inputs; the navigation becomes a scrolling row below 768px, tab lists scroll instead of overflowing, and tables, long keys and JSON no longer push screens wider than a phone. The Explore panel is no longer repeated in the Retrieval lab: the lab links to Explore ("Ask the guide") when a guide service is configured, and its parameter fields have readable labels.
• The tour pictures, the README gallery and docs/tour.md show the current app on a demo stack with real data, a real Explore answer and its Agent activity trace, Record history, the Derivation preview, typo-tolerant and hybrid search, extraction items and schema suggestions; four pictures come from the mock and say so.

Fixed

• run_index_jobs (lorebase_store.indexes) sends job heartbeats while a build runs, so a long index build is no longer reclaimed as lost after store.jobs.stale_after_seconds. Releasing a job on SIGTERM and reclaiming one after SIGKILL were checked in containers (validation).
• Jobs of a Scribe that is killed or loses the database no longer stay running for an hour: a job whose worker sent no heartbeat for store.jobs.stale_after_seconds, now 90 seconds (was 3600, measured from the last heartbeat; Scribe sends one every scribe.worker.heartbeat_interval_s, now 30 seconds, was 60), is reclaimed by the scheduler leader at its next tick: queued again, or failed once it used store.jobs.max_attempts, so a job that keeps killing its worker is not run forever (scribe). A worker that lost a job this way no longer overwrites the outcome of the run that replaced it (its run counts as lost in lorebase.scribe.jobs), and python -m lorebase_store.jobs work now sends heartbeats too.
• A stopping Scribe (SIGTERM) leaves no job running even when a handler does not stop when cancelled: shutdown_cancel_s (scribe.worker, 5 seconds) after shutdown_grace_s its jobs are put back in the queue anyway, without counting the attempt.
• Agent trace: calls without a session id of one client no longer split into many one-call inferred sessions when they run concurrently (agent trace). A step carries the time its request started but is written when the request finishes, and the grouping only looked back from each step, in write order: a call that started before the last one written found no session and opened its own (the web app's parallel reads opened a new session every few milliseconds). Inferred steps of a batch are now grouped in time order and join the session of the nearest step of their client within inferred_session_idle_s on either side.
• Guide in the compose stack and the Helm chart has a writable state directory (/var/lib/lorebase/guide, volume guide-data; Helm guide.persistence, an emptyDir by default) while its root file system stays read-only, so it remembers structured output modes across restarts and no longer logs a warning at each start. The example agent reports the keys of the edits and the Proposal ids from the response to the committing call, not those of its dry run.
• Hybrid search ranks its results by fused score again: Records found only by the semantic list were listed before every lexical match.
• Guide's query tool told the model a where and order_by shape Keeper's query DSL refuses ({attr: value}, {attr: asc|desc}), so in live runs most failed tool calls were refused queries and many explorations ran out of steps. The description now gives the DSL's predicate forms and a valid example (checked against the DSL in tests), and the history, neighborhood and traverse descriptions state their bounds (validation).
• A Guide exploration whose token budget ran out after several useful reads answered Not found in the graph.: only the step budget left a turn to answer. A tool step is now taken only when the budget still covers it, the next call with a full tool result and an answer; otherwise that turn is the final one (guide). The input estimate takes its characters-per-token ratio from the endpoint's count of the previous call (JSON tool results ran at about 2.2 on Claude models, not 4), and the answer of the final turn is kept when the endpoint counted more than estimated. The instructions now ask for a partial answer with citations rather than found: false when part of the question is answered, and treat a checked negative result (no Record meets the conditions) as an answer. The tokens of a model answer that does not parse now count towards the exploration's total, and the model call failed log line names the schema problems (paths and keywords, never answer text).
• Guide's neighborhood and traverse tools dropped the Link Attributes Keeper returned (a stocked quantity, a lead_time_days), so the model answered that the graph held no quantities. Edges now carry their values.
• The example process agent sent temperature: 0 on every request, which current Claude models refuse with HTTP 400. --temperature none (AGENT_TEMPERATURE) leaves it out and --extra-body (AGENT_EXTRA_BODY) passes options such as thinking (process agent).
• Herald no longer logs every Keeper request URL, Record keys included, at INFO through the HTTP client's logger; over stdio those lines went to the agent's standard error.
• Record lineage stays bounded on Records with long histories: changes are counted up to read.history.lineage_count_max (10000; changes_truncated reports more), and rows without lineage revisions search only the newest read.history.lineage_scan_changes changes (1000). The new store.lineage_backfill job fills lineage_revisions for rows materialized before migration 0032. The creating change falls back to the history when the Revision stored for it has no history row, maps that are not JSON objects read as empty, and a merge attributes only the values it changes, so values it leaves keep the Revision that set them (agents).
• Record lineage after a fast-forward names the Revisions of the source branch that set each value, as a rebuild does, instead of the source head; rebuilds and backfills attribute a merge only the values it changed, and python -m lorebase_store.jobs lineage-backfill --recompute (job payload "recompute": true) corrects rows written by merges before that fix. Workers starting together queue the lineage backfill once.
• After a fast-forward, the branch history holds one row per change of each fast-forwarded Revision, as a rebuild writes it, so Record history and the lineage change count match a rebuild (store.materialize.fast_forward_history_max_revisions, 1000; a longer range keeps one row per element at the source head). lineage-backfill --recompute rewrites the history, Records and Link lineage of earlier fast-forwards the same way.
• A background copy that keeps failing no longer holds up the deletion of its base for good: each failed job is retried after a delay doubling from job_retry_seconds (at most copy_retry_max_seconds), and after copy_max_failures failures the overlay is detached from its base and built from its tree instead. New outcomes retried, fallback and given_up of lorebase.store.branch_jobs.runs and alerts LorebaseBranchCopyFallback and LorebaseBranchBuildFailed (operations).
• Derivation and Source previews (tour). A previewed value computed from sensitive Attributes of another Kind is masked unless the caller is cleared on the Derivation's Kind and on every Kind the values come from, and a masked item reports a failure without its error code (error.code is now nullable). keys are bounded by read.derivation_preview_max_key_length (1024). POST /sources:preview accepts a schema.read grant narrowed to the Source's target Kind and answers 403 for other Kind-narrowed callers before validating the change set; it refuses rows of more than read.source_preview_max_columns columns (200) and requests over read.source_preview_max_cells evaluations (1000, rows times expression mappings) with 422, and evaluates off the event loop. Expressions over rows are compiled once over every column of the Source, so a column a row lacks is null in Scribe and in the preview alike (it was an unknown-variable error), and very large decimals keep their exponent form.
• Path Derivations: values over the inverse direction stay current when a Link pushes another far key out of an intermediate Record's store.traverse.path_fanout bound. Job workers rewrite branches computed with another fan-out in the background next to their job loop instead of before it, and a branch that fails or is being copied in the background no longer stops or blocks the sweep. Operation links() over a path reads its first step no further than store.reads.max_page_rows, and neighborhoods following a composed path report truncated only when Links were left out.
• Embedding vectors are keyed by the fingerprint of the embedding template as well as by content hash and model (migration 0024). Branches whose templates differ for the same content keep a vector each instead of overwriting one another, and Keeper's semantic and hybrid search read the vectors of the searched branch's template. Vectors of identical texts are still copied without a model call, now across templates too. The upgrade sets the fingerprint wherever every branch holding the content agrees on one template, and the next pass re-renders each text once to fill in the rest. Garbage collection also deletes vectors of templates no branch uses any more. Scribe resolves embedding templates once per schema hash and keeps the last scribe.embeddings.template_cache_size schemas.
• Database sources of schema inference are bounded further: at most store.profiling.sql.max_tables tables (now 50), total_timeout_s for a whole request, and max_value_chars per value, with unbounded text columns cut and binary or unknown-type columns not read. A schema must be a plain name found in the catalog, and keeper.schema_inference.allowed_schemas can restrict a source to some of them. A failed or misconfigured source answers a generic 503. Previews cut long text (preview_max_cell_chars).
• LorebaseGcNotRun no longer fires for a gc schedule set to dry runs (scribe.scheduler.gc_dry_run: true). Scribe counts the collections it schedules in lorebase_gc_scheduled_total{dry_run}, and the alert needs real scheduled collections (dry_run="false") that never complete within 3 days. The Scribe dashboard shows scheduled runs next to completed ones.
• lorebase_jobs_queue_depth reports jobs waiting for a retry delay (queued with a run_after still ahead) as status="retrying", so queued counts only claimable jobs and LorebaseJobQueueStalled no longer counts jobs no worker could claim yet.
• LinkML: path Derivations are exported as slots carrying their steps, inverse name and materialized as annotations, and schema import --apply reads them back, so a round trip keeps them. Applying to a Domain no longer warns that Operations, Derivations and Sources are not imported when the merge keeps them, and --prune without --apply or --dry-run is refused instead of doing nothing.
• Large commits no longer slow down with the number of Links they write. Looking existing Links up by (from, to) pairs planned one index scan per pair (about half a millisecond of planning per pair, so 0.5 s for a chunk of 1,000) and now runs as one semi-join on two arrays (13 ms for the same chunk); cardinality and required-Link checks count the batch's Links per key once instead of once per key; canonical JSON writes strings without characters to escape in one step. Loading 28,000 Records and 84,000 Links in 15 commits went from 67 to 48 s on the same machine.
• Commits no longer slow down with the size of the Kinds they write. The search vectors of a commit's Records were refreshed by filtering every Record of the Kind on the commit's Revision (7 s per statement for a Kind of 200,000 Records) and are now refreshed by key; unique values were checked with an expression the Attribute's unique index does not match (a minute per thousand values at 160,000 Suppliers) and now go through the index.
• The background copy of a branch no longer fails at 10M Records before copying anything: its first step counted the base branch's rows for the progress report (85 million rows, past the statement timeout). rows_total is now the planner's estimate until the copy switches, then the rows read.
• Schema suggestions no longer list one fact twice under two names: Attribute suggestions of the same Kind whose names share at least schema.suggestions.merge_similarity of their words (0.6) and whose samples share a value type are folded into the better supported one (renewal_notice_days and renewal_proposal_notice_days, termination_notice_days and termination_for_cause_notice_days). Kinds named by a generic document or workflow word (schema.suggestions.kind_stoplist) or by an Attribute (Buyer beside buyer) are not suggested, and a proposed Kind is listed once it has kind_min_attributes (2) Attribute suggestions or as many observations.
• Tier 2 extraction asks for link Attributes: a Relation with link Attributes has its own branch in the answer schema with the link's values (evidence for each, the required ones required), and the prompt asks for the required ones. Link values are verified and grounded like record values and reach the Link item; before, stocked_at Links came without quantity. A Link item still missing a required link Attribute stays pending, with the names in match.missing.
• A structured output mode an endpoint refuses is remembered for the process, per base_url, model and output settings, instead of being tried again at the start of every extraction job (two refused requests per job with a strict-only endpoint before).
• A tier 2 chunk whose answer stays invalid after the repair attempts no longer fails the whole extraction job: the chunk is skipped with a warning, counted as model_invalid_answers, and its tokens still count toward the job budget and usage. An answer cut off before any output says that a reasoning model may have spent the token limit on reasoning.
• Tier 2 timestamp values written without an offset (2024-01-23, 2024-01-23T10:15) are read as UTC instead of being discarded as invalid, and timestamps are grounded by their day in the quote ("approved on 23 January 2024").
• A webhook receiver that disappears no longer holds a Scribe worker. Each delivery attempt, host lookup and every checked address included, is now cut off after webhooks.timeout_s; before, the budget applied to each address in turn, the lookup had none, and one pass sent up to webhooks.batch deliveries one after the other, so a receiver that accepted connections and never answered, or dropped packets, kept a webhook_delivery job running for many minutes while its heartbeat kept it from being requeued. Connections give up after the new webhooks.connect_timeout_s (5 s) and try the next checked address; a pass sends webhooks.concurrency (8) deliveries at a time, starts none after webhooks.pass_timeout_s (60 s) and releases the rest to the next pass without counting an attempt; a pass still running after webhooks.lease_s fails its job attempt. webhooks.lease_s must now exceed pass_timeout_s + timeout_s (the defaults do). Timed-out attempts are retried with backoff and marked dead after max_attempts as before, and counted in the new metric lorebase.webhooks.timeouts (operations).
• The compose db-roles step grants the application role EXECUTE on all eight security definer read functions of a database migrated before the role existed (it granted only the four of revision 0005), reading them from the catalog like the Helm chart's role setup does.
• Guide no longer overshoots its token budget by up to the steps times max_result_chars: before each model call it estimates the input (prompt, messages and tool results so far) and calls only when used plus estimated input fits in guide.agent.max_total_tokens, with the completion cap cut to what remains after the estimate. The session event now names the chat model and the effective max_steps of the run (guide).
• Outbox pruning keeps up with high event rates: a pass deletes batches of store.outbox.prune_batch events, each in its own transaction, until nothing is left or the new store.outbox.prune_budget_s (5 s) is spent, instead of one batch per pass. The new gauge lorebase.outbox.prune.backlog reports the events still prunable after a pass (counted up to 100,000) and the alert LorebaseOutboxPruneBacklog fires when it stays above 50,000 for 30 minutes (operations).
• The per-session step cap of the agent trace holds across Keeper replicas, and listing sessions no longer scans the steps of the retention period (agent trace). Migration 0029 adds agent_trace_sessions, whose step count the trace writer reads and raises under the row lock in the transaction writing the steps (steps over keeper.trace.max_steps_per_session are dropped and counted as session_full), and agent_trace_session_domains, a summary per session and Domain (first and last step, calls, notes, tools, Proposals; indexed on principal and on Domain with the last step) that sessions are listed from under the same visibility rules as steps. Pruning lowers both and recomputes the summaries from the steps left; both tables are filled from the existing steps.

Security

• Derived values whose sensitivity is inherited from other Kinds are masked unless a Kind-narrowed clearance covers every source Kind, on every read path: Records, lists, queries, search, history, lineage, diffs, explain and context packs, as in the Derivation preview. Filtering, ordering or grouping by such a value needs the same clearance (403 sensitive_attribute). Domain-wide clearances are unaffected; webhook clearances are Domain-wide and change feed events carry no values (design 13.5).
• The structured output mode memory stores only keyed digests, never an endpoint address, and its table grants the application role no TRUNCATE, REFERENCES or TRIGGER. Closing the evidence gap for vaulted values leaves one residual inference: a reader of suggestions can tell that a quote they chose matched from the missing evidence (threat model).
• Vault keys rotate (design 13.5). lorebase admin vault rotate (POST /api/v1/vault/rotate, permission vault.manage, held by installation administrators only) queues the vault.rotate job, which seals every live vault row of every Domain again under vault.encryption_key and makes its fingerprint again under the current token key, vault.rekey_batch rows per transaction, continuing where it stopped; lorebase admin vault status (GET /api/v1/vault/keys) counts rows per key and names the earlier keys no row uses. vault.token_key now rotates too: give the new key a token_key_id and keep the old one in previous_token_keys; tokens already stored keep working and a value written again finds its old token, so nothing reads as changed.
• Erasure covers what it missed. Erasure by value compares matching forms (case, compatibility characters and white space ignored), so Ada@Example.com and ada@example.com are erased together. Erasure by key follows key changes: a commit that deletes a Record and writes another of its Kind holding its vault references (a key changing Operation, say) now stores those values again under the new key instead of being refused (once per reference, into a Record that does not exist yet), and records the lineage that erasure follows both ways. Erased values leave a tombstone (their fingerprint): a later write of one by a Source sync, extraction or an import (vault.tombstone_origins; writes of people and agents are not checked, so they cannot test whether a guessed value was erased) is stored erased (vault.erased_values: erase, the default), refused (refuse, issue value.vault_erased) or stored again (allow, the old behaviour). Commits that store vault values and erasures take the Domain's vault lock (shared, waiting at most vault.lock_timeout_ms, and exclusive), so a value is written either before an erasure, which erases it, or after, which finds its tombstone. The audit entry of an erasure by key holds a keyed digest of the key, no longer the key.
• The parameter digest of Operation invocations is always keyed: under vault.digest_key, else the vault token key (as before), else a random secret that migration 0027 stores in installation_secrets; it was a plain SHA-256 without vault keys.
• Health reports no longer list which Records share a sensitive unique or title value to callers without the clearance: they count the groups and Records (duplicates.hidden). Validation issues about values the caller may not read carry the generic value.invalid code. A vault reference nested inside another Attribute's value is refused (value.vault_reference). The NDJSON diff stream and Operation edits show vault values, [erased] or the mask, as other reads do, instead of raw references.
• Operation sensitivity checks resolve the fields an expression reads against the Kind the type checker infers (comprehension variables over links() or lists of Records, conditional results, nested traversals, record($step[n])), so a same-named sensitive Attribute on another Kind no longer refuses a harmless Operation; only fields of untyped (dyn) values are still matched by name. Evaluation errors are shown in full to callers cleared for the Attributes the expression reads on the Kinds that hold them, not only to callers cleared Domain-wide.
• Branches forked before the default branch tagged an Attribute can take its sensitivity tags: POST /domains/{d}/branches/{b}/rebase-tags (lorebase branch rebase-tags BRANCH) commits the tags (never removing one) and recomputes the titles and search vectors the branch stored before them. With rows_only, or automatically for every other branch when the default branch tags an Attribute (store.materialize.refresh_branches_on_tag, on by default), a store.sensitivity_refresh job rewrites those rows in resumable batches. Extraction decides what to extract by the effective tags.
• Evaluation errors of Operation steps (a value that does not fit its Attribute, a created Record that exists already) are redacted when the step reads or writes values the caller may not read, and an expression the redactor cannot parse is redacted rather than shown.
• python -m lorebase_store.bundle export --include-vault-values writes the vault values of the exported Records in plain text (or erased) in vault.jsonl, listed in the manifest, readable by its owner only and audited as bundle.export_plain without the values; import seals them under the target's vault keys after checking that each token is held by that Record and Attribute in the bundle. A restore from a bundle no longer has to lose vault values.
• Read auditing gains keeper.auth.audit.reads.on_overflow: reject, which answers a read with 503 and Retry-After when its record cannot be queued and keeps retrying failed batches (turning readiness failed) instead of dropping them, so the read stream can serve as proof.
• Operation edits resolve only vault Attribute values bound to their own Kind and Attribute, on a copy of the changes; revalidation issues on values the caller may not read are counted per Attribute instead of listed by Record; markdown context packs and explanations resolve vault references from structured data, never by matching text; writes that show sensitive values say so in their audit entry; Conflicts show vault values instead of references.
• Sensitivity refresh jobs follow the effective tags (derived title and searchable Attributes included), also run for tags on Attributes new to the default branch and for copies of stale branches, are no longer swallowed by a running job, and are queued only for branches with stale Kinds; the tag rebase lists added tags only to callers who may read the default branch's schema.
• Erasure takes the Domain's vault lock first (vault.erase_lock_timeout_ms, then 503) and reads the key changes under it, so a key change committing meanwhile is erased too; it follows any number of key changes, in batches, and flags an unusually long lineage in its audit entry instead of refusing. A token a key change took over is marked moved (migration 0033) and can be neither taken over again nor claimed back.
• Bundle exports with --include-vault-values carry the vault's key changes and erasure tombstones, read under the Domain's vault lock and audited at start and end; import refuses tombstones made under a token key it does not have, checks values against the bundle's schema and against tombstones as vault.erased_values says, and treats a Domain with leftover vault rows as not empty.
• New Domain permission pii.import (only admin among the built-in roles): values of vault Attributes can be fed through the built-in import and through Sources (defining or mapping such a Source, triggering its sync) only with it, checked before anything is written, so callers without it cannot test whether a value was erased.
• Vault bundles carry the moved_to of key-changed tokens and legacy tombstones (as <token_key_id>/1, noted in the manifest), accept only key changes between Records the bundle holds, record each vault import in a bundle.import_vault audit entry with the count of oversized key-change groups, and hold the vault lock only while the export's snapshot is taken.
• Erasure makes outdated fingerprints current before taking the vault lock and checks again under it; a key change replayed on another branch is allowed as the same move, a later rename carries the pointer of earlier moved tokens along, and erased references move without being marked moved.
• Reviewer edits of extraction items that set a vault value need pii.import, Scribe never auto-accepts extraction items holding vault values, and python -m lorebase_keeper sources vault lists Sources that map vault Attributes so ones defined before pii.import can be confirmed or disabled.
• lorebase admin erase --value takes - (standard input, asked without echo on a terminal) or @FILE; a value on the command line is refused, as it stayed in the shell history and the process list.
• Keeper sends X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Content-Security-Policy: frame-ancestors 'none', Referrer-Policy: no-referrer and, over https (directly or from a proxy in keeper.trusted_proxies with X-Forwarded-Proto: https), Strict-Transport-Security; keeper.security_headers configures or turns them off. The web image sends a Content Security Policy whose connect-src holds the Keeper origin of the runtime configuration (LOREBASE_WEB_CSP_CONNECT_SRC adds sources, LOREBASE_WEB_CSP replaces the policy, LOREBASE_WEB_HSTS adds HSTS), and the Helm chart has the same through web.security. The web image's Referrer-Policy changes from same-origin to no-referrer (deployment).
• The application role can no longer change the audit log: migration 0015 revokes UPDATE, DELETE and TRUNCATE on audit_log from PUBLIC, from database.application_role and from every other role but the table owner, and the compose db-roles script no longer grants them. Entries are still appended and read as before; nothing in Lorebase prunes the audit log, and an operator who must do so works as the owner (deployment).
• Document intake by URL no longer trusts the allowlisted host's DNS: Keeper resolves the host once, refuses it with 422 unless every address is global unicast (loopback, private, link-local, shared and other non-global ranges are refused, as for webhook targets), and connects only to the checked addresses, so a changed DNS answer cannot reach an internal service. keeper.documents.url_allow_private_targets (false) turns the check off for hosts on an internal network.
• Scribe parses PDF and DOCX documents in a child process without network access, stopped after scribe.extraction.parser.timeout_s (60 s) of wall-clock time, cpu_s (60) of CPU time or memory_mb (1024 MiB, where the system enforces address-space limits), and refuses documents above max_input_bytes (50 MiB). A crafted document that hits a limit fails its extraction job with a message naming the setting instead of holding the worker; isolated: false returns to parsing in a worker thread (extraction).
• Webhook signing keys can be stored encrypted: with webhooks.encryption_key (a secret:// reference to at least 32 characters, the same for Keeper and Scribe) new keys are sealed with AES-256-GCM, the delivery worker seals the keys stored before, and previous_encryption_keys lets the key change without breaking deliveries. Receivers change nothing. Migration 0016 adds webhooks.sealed_key; its downgrade refuses while sealed keys exist (agents).
• Supply chain: the base images of the backend and web images (and the compose proxy image) are pinned by digest with the tag kept readable, and moved by hand (release); the build arguments PYTHON_TAG, NODE_TAG and NGINX_TAG are replaced by PYTHON_IMAGE, NODE_IMAGE and NGINX_IMAGE. A CI workflow (audit.yaml) runs pip-audit on the locked Python dependencies and pnpm audit --prod on every pull request and weekly, failing on advisories not accepted in .github/audit-allowlist.yaml; make security uses the same check.

Not done yet

• The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory (performance).
• The performance targets of the design at 10M Records: the 2-hop neighborhood, full-text search (searches of very common words above all), Record history and lineage are over budget, and a commit of 1,000 changes takes 3.3 s (performance).
• Full RBAC: groups from the OIDC groups claim, Kind-scoped grants on reads and sensitivity masking are done; Kind-scoped invisibility in queries is not (roadmap).
• The stability guarantees for /api/v1: the policy is written and binds from 1.0; Keeper sends no Deprecation or Sunset header yet, since nothing is deprecated (API stability).
• Packages on PyPI: the wheels and sdists are attached to the GitHub release.

Upgrade notes

In this order: configuration and secrets, then the migrations, then the restart and the one-off jobs. Keeper and Scribe of 0.1.0 must not run against the migrated database, except as noted for rolling upgrades.

Before upgrading: configuration and images

• Compose: run make env once to add LOREBASE_WEBHOOK_KEY to .env; values already set there are kept. Helm: set secrets.values.webhookEncryptionKey (or the webhook-encryption-key entry of an existing Secret), or config.webhooks.allow_unsealed_keys: true for development; the chart refuses to render otherwise. Elsewhere, set the same webhooks.encryption_key for Keeper and Scribe.
• Helm database roles: give the Secret db-maintenance-dsn (the owner, the old db-dsn) and point db-dsn at the application role (lorebase_app by default; the migration Job creates it when the owner has CREATEROLE). With postgresql.enabled, set secrets.values.dbAppPassword (or create the lorebase-db-app basic-auth Secret); one helm upgrade is enough, as the pg-roles hook declares the role before the migration Job. With restricted egress, allow the API server (networkPolicy.egress.kubernetesApi). database.allowSharedRole=true keeps one DSN, for development only.
• Delete read.history.lineage_max_changes from configuration files.
• Choose a storage profile with store.storage.profile (compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile with the Postgres settings of ci/storage-<profile>-values.yaml); without one, the indexes stay as they are.
• Scribe: keep scribe.worker.heartbeat_interval_s at most half of store.jobs.stale_after_seconds (a deployment that sets only the heartbeat above 45 seconds must lower it or raise the threshold), and SCRIBE_STOP_GRACE (compose) or scribe.terminationGracePeriodSeconds (chart) above scribe.worker.shutdown_grace_s plus shutdown_cancel_s (35 seconds by default; the defaults are 45 and 60).
• Callers that read sensitive values need pii.read, confidential.read or restricted.read through a custom role in keeper.auth.authz.custom_roles. To use the vault, set the same vault.token_key and vault.encryption_key (secret references, 32+ characters) for Keeper and Scribe; rotate the token key with token_key_id and previous_token_keys, never by replacing it.
• With more than one Keeper outside Helm, set keeper.limits.rate.backend: postgres.
• Images: make ghcr.io/aantenore/lorebase-ml reachable where Scribe and Guide run, or set Helm imageMl.repository: "". Give Guide a writable state directory (see Breaking changes).
• Web image: a VITE_KEEPER_BASE_URL given only as a build argument does not reach the Content Security Policy; set it at run time or list the Keeper origin in LOREBASE_WEB_CSP_CONNECT_SRC.

Migrations 0013 to 0036

Run them (alembic upgrade head; the compose migrate service and the Helm hook do it) before restarting Keeper and Scribe, with database.application_role set when not on compose or Helm, so that the grants of the application role follow. On compose, db-roles keeps UPDATE and DELETE on audit_log and audit_read_log revoked on every start.

• 0013 web sessions; 0014 embedding jobs.
• 0015 makes audit_log append-only for the application role (revokes UPDATE, DELETE and TRUNCATE); revoke them yourself if the migration runs without database.application_role.
• 0016 adds webhooks.sealed_key; its downgrade refuses while sealed keys exist.
• 0017 background branch copies; 0018 the per-branch search lexicon (its functions stay for Keepers of an earlier version after 0035).
• 0019 vault bindings; 0020 outbox consumers; 0021 schema suggestions.
• 0022 read audit log and shared rate limit buckets. python -m lorebase_keeper audit verify --json gains a stream member and the audit_verify job result a reads member.
• 0023 agent trace; 0024 keys embedding vectors by template fingerprint; 0025 the snapshot ref kind used by reads at past Revisions.
• 0026 restores the plain reverse Link index, built concurrently per partition, only on a database that ran its withdrawn first version.
• 0027 adds vault.fingerprint_key and the tables vault_tombstones, vault_lineage and installation_secrets (with a random digest secret). Operation invocations recorded without vault keys keep their plain digest, so replaying one of their idempotency keys answers a conflict.
• 0028 deferred Operation events.
• 0029 agent trace session rows. A Keeper of an earlier version writes and prunes steps without updating them: turn keeper.trace.enabled off on old Keepers during a rolling upgrade.
• 0030 mat_schema.path_fanout. Branches with materialized path Derivations, or derived values reading a path, are recomputed once by the first commit on the branch or when a job worker starts.
• 0031 the building materialization and the build, retry and reconcile columns of branch_materializations; Scribe workers of an earlier version must not run store.materialize jobs afterwards.
• 0032 mat_records.lineage_revisions; until the backfill below, lineage reads the missing Revisions of earlier rows from the branch history.
• 0033 the moved_to of key-changed vault tokens.
• 0034 inference_endpoint_modes; downgrading drops it and the modes are learned again from the next rejections.
• 0035 lb_search_bounded and lb_search_bounded_overlay, which lexical search calls; the candidates are capped by store.reads.definer_ceilings.candidates (100,000).
• 0036 shared lineage maps (mat_lineage_maps, read through lb_lineage). On an existing install it shares nothing yet (see After the restart); its downgrade writes every map back into its rows.

After the restart

• Run a job worker for store.lineage_backfill (Scribe runs it by default; python -m lorebase_store.jobs work too): it fills mat_records.lineage_revisions of the rows materialized earlier, branch by branch in batches of store.materialize.job_batch_rows.
• Once every process runs 0.2.0, run python -m lorebase_store.storage apply as the maintenance role (compose: docker compose --profile maintenance run --rm storage apply) to share the lineage maps and apply the storage profile, then apply --compact --reindex to give the space back. Both are batched, resumable and leave reads running; the 1M-Record benchmark database took 26 and 7 minutes on a loaded machine. python -m lorebase_store.storage status shows what is left and prewarm loads the read set into memory.
• Run lorebase admin vault rotate once: rows stored before keep the fingerprint of their exact value until rotated (an erasure by value rotates the rows it looks at first).
• Run python -m lorebase_keeper sources vault (--check exits 1 when any is listed): Sources that map vault Attributes and were defined before pii.import existed keep syncing on their schedule until you confirm or disable each one.
• Existing webhooks keep working; their signing keys are sealed by the next delivery passes.
• Branches forked before the default branch tagged an Attribute take its sensitivity tags with POST /domains/{d}/branches/{b}/rebase-tags.