Repository navigation
v1.2.0
Release Notes v1.2.0
Released: 2026-08-13
v1.2.0 changes what a memory is on the way in and what retrieval hands back on the way out. Memories are now cut into passages (verbatim spans of the parent body) that are first-class graph entities, writers can declare their own cut plan, exact-match keys, and self-verification probes, and readers can widen a result set with two bounded traversal verbs instead of asking for a bigger pack. Alongside that: memory scope becomes an indexed column with a reversible backfill, the graph loses several tablescans that were collapsing ingest, and every response now names the server version.
Highlights
Passages: a slice substrate for retrieval
Long memories are projected into EntityType.PASSAGE rows on the existing entity table, each one a verbatim span of its parent with a PART_OF edge back to it. Retrieval serves spans instead of whole fat bodies, and drops the parent only when the pack already holds every span of that projection (passage_covers_parent true and indices exactly range(total)). Partial coverage never hides a memory: if projection has to skip an oversized span, every surviving span is stamped passage_covers_parent: false and the parent stays servable.
The mechanical cutter (projection/slicing.py) packs siblings into a 600–1,200 char band with a 2,000 char hard ceiling, descends oversized subtrees one level, emits an oversize childless leaf whole rather than bisecting a literal, and merges a stranded sub-floor tail back into the slice before it. Slices partition the body lines exactly.
Writers declare structure: spans, retrieval keys, probes
remember / add and POST /entities accept four new declarations:
spans— an agent-authored cut plan of half-open{start, end, label}offsets into the stored body. Validated as verbatim and complete: 2 toMAX_AGENT_SPANS(64) spans, no gap, no overlap, each withinMAX_AGENT_SPAN_CHARS(17,768). An invalid plan is refused, not silently rewritten; a stale plan falls back to the mechanical cutter and is stamped as such.atomic— "this body is one unit," which suppresses cutting entirely.retrieval_keys— up toMAX_RETRIEVAL_KEYS(16) identifiers the memory answers to (error strings, symbols, flags, aliases). Matching is casefold plus internal whitespace collapse and nothing else:ERR_DB_TIMEOUTwill not matchERR_TIMEOUT. Stored as both display and normalized columns, element-indexed forCONTAINSANY.probes— up toMAX_PROBES_PER_MEMORY(5) questions this memory must answer later. Each probe is run through the live retrieval path at write time (REHEARSAL_LIMIT10,REHEARSAL_BUDGET_SECONDS6.0) and the per-probe rank-or-absent verdict is returned onEntityResponse.probe_rehearsal. Retrievability becomes a measurement instead of an assumption.
A nightly job (replay_memory_probes, cron 0 5 * * *, 168-hour window, 200 memories per run) re-asks those probes and records probe_last_replay, so a memory that becomes unreachable as the graph evolves shows up as drift instead of silence.
Bounded traversal verbs for clients
Two new verbs let an agent widen deliberately instead of over-fetching up front:
expand_neighbors(POST /api/search/expand) — walks the bounded neighborhood of up to 8 seeds,depth1–3,limitup to 24, preview content capped at 500 chars, withunresolvedreported identically for missing and unreadable seeds.fetch_slice(POST /api/search/slice) — reads one memory at span granularity, window default 3 up to 64 spans, returningparent_idfor citation pluscovers_parent,window_start, andpassage_total.
Both are stateless per call, clamp rather than reject, authorize every row for the calling reader, and authorize a span's parent before serving the span. They return adjacency and previews only; context keeps composition. Documented in docs/api/mcp-traverse.md.
Deterministic memory export and a per-project handbook
sibyl export memory materializes a project into a .sibyl/memory/ tree: README.md, index.md, recent.md, tasks.md, notes/<type>-<slug>.md, optional handbook.md, and a manifest.json carrying per-file SHA-256 plus a content_digest over the whole projection. Byte-identical output for an identical graph is enforced by three rules: no wall-clock values, a whitelist of stable metadata fields, and total ordering by id (server response order cannot leak in). Files land read-only unless --writable; the writer refuses to overwrite a populated directory that has no prior manifest unless --force. Flags include --project, --output, --notes, --tasks, --task-status, --type, --hydrate/--no-hydrate, --handbook/--no-handbook, and --json. See docs/cli/export.md.
The handbook (GET /synthesis/handbook?project=<id>) composes five fixed sections: Orientation, Decisions In Force, Current Work, Gotchas, and Key Artifacts. Every line cites its source, each source is claimed by exactly one section (the one it scores highest for, ties going to the earlier section), and an empty heading is reported as a true statement about the graph rather than padded. There is no model call in the path, so it is composed per request and never cached: it cannot go stale against the graph it describes.
Memory scope becomes queryable, with a reversible backfill
memory_scope is promoted from the attributes bag to an indexed entity column (schema v17), so scope absence is now a fact rather than an ambiguity, and a scopeless write can no longer unscope the row it replaces. sibyld migrate scope-backfill stamps rows that predate scope stamping: --org-id, --dry-run/--apply, --reverse. Recovery from the originating raw capture runs before derivation from project_id, so a genuinely private memory is never widened to project scope. Scopes no reader could satisfy (team, delegated, organization, shared, public) are skipped and counted rather than stamped, and every stamped row records scope_backfill_source plus a scope_backfill_prior receipt that --reverse restores exactly. A concurrent write can no longer forge that rollback record.
Query performance: tablescans retired
Three hot paths stopped scanning the entity and relates_to tables:
- Embedding-backfill
UPDATEnow addresses rows through an indexedgroup_id + uuidsubquery with the if-current guards intact: 2.83s → 20–130ms per statement at 43K rows. - Bulk-upsert
DELETEreplacesuuid IN $listwith per-uuid indexed deletes: 9.4s → 16ms for 700 uuids at 350K edges. - Record-id resolution maps per-uuid lookups instead of an IN-list: 3.3s → 128ms for 1,400 uuids at 167K entities.
idx_entity_labels moves to element form (labels.*, rebuilt CONCURRENTLY in schema v19) so label filters are index-served and correct on their own instead of scanning or silently returning nothing.
Retrieval and Ranking
- Exact-key arm. A mechanical identifier detector (quoted strings, letter/digit mixes, underscores,
::scope separators, dotted segments, hex runs, case transitions, long--flags; bare all-caps acronyms deliberately excluded) fires an exact-key lane overretrieval_keys_normalizedwhenRetrievalSignal.EXACT_KEYis planned, with a1.4boost applied once per candidate even when several lanes reach it. - Typed-overfetch KNN with exactness fallback.
knn_type_overfetch(0–32, off by default) walks an untyped HNSW pooloverfetch × limitdeep and filters types on materialized rows, capped atKNN_TYPE_OVERFETCH_CAP4,000 (warm-probed at 0.82s on a 95K-row namespace). If the filtered head fills the budget it is the typed KNN head; otherwise the classic typed bracket runs. - KNN effort floor.
knn_search_effortcomputesmax(1, configured_ef, k). Configured effort is a floor on quality, never a ceiling on requested rows, which fixes silent truncation whenef < k. - Per-lane seed budget scales with the request.
seed_candidates_per_signal(limit)returnsmax(2, min(limit, 50))instead of a flat per-lane depth. Seed budget is diversity budget, and once the unit is a passage a lane needs more rows to reach the same number of distinct sources. - A2 ranking levers ship opt-in and off by default:
semantic_prior_rescue_weight(a coverage-gated prior that only lifts candidates with little lexical overlap) and thetyped_entity_overlappool ordering. Both failed their gate (see Measurement). searchnow serves stored content rather than its derived blurb, and passages are always included in the requested type set.
Context Packs
- Character-bounded composition.
compose_operational_evidenceacceptschar_budget, admitting candidates in rank order while the running content total fits. Admission stops at the first candidate that does not fit rather than skipping ahead to a smaller one, so a pack is always a prefix of its ranking. Exposed aschar_budget(1–500,000) on the evidence request. An item count only bounds payload while item size is stable, and a passage is roughly a twelfth of the state it was cut from. - The token budget is spent, not just trimmed to.
markdown_token_budgetnow defaults toDEFAULT_MARKDOWN_TOKEN_BUDGET(4,000) and accepts up to 32,000. Given a budget, both breadth and per-item depth scale toward it (85% of the budget to content, the rest to structure) within hard ceilings. - Facet routing by parent type. A span of a decision is filed as decision material, not under a generic "passage" heading. Eligibility is settled before parent suppression, so a filtered-out span cannot suppress the memory it came from on its way out.
- Degraded packs keep fidelity. The fallback search path no longer inherits search's 500-char preview default; it uses
FALLBACK_SEARCH_CONTENT_MAX_CHARS(18,000). Degradation may cost freshness or ranking, but a caller cannot tell truncated content from short content. - Source expansion windows over adjacent passages, and each window run is sized by what that run actually holds.
Data Integrity and Reliability
- Entity content cap enforced.
MAX_ENTITY_CONTENT_CHARS(50,000) was declared in three layers and enforced in none; the live graph held a 520,802-character pasted transcript that skewed BM25 length normalization corpus-wide. Single and bulk writes now reject oversized bodies (rejection, not truncation); migrations can still reshape legacy rows. - Embedding backfill false negatives fixed. The currency fence now derives persisted text the same way hydration does (
content→metadata.content→ summary), so rows stored with an empty content column stop reporting as missing forever. Stale-but-present rows are reported asstale_entity_ids; only genuinely absent rows fail the batch. - Interrupted idempotency receipts recover on retry. A pending reservation observed under the serialization lock was interrupted, not in flight, so the claim is adopted and the handler re-executes instead of returning 409. A true in-flight duplicate still blocks at lock acquisition.
- Document search degrades to scan when the embedder is soft-failing and both vector and lexical lanes come back empty.
- Record identifiers stay in error messages. UUID scrubbing turned every "Task not found:
<id>" into a field-less "Invalid request data." Ids are the system's public vocabulary; SQL, POSIX path, and Windows path scrubbing remain. - New
unprocessable_entity()(422) factory passes the message through verbatim with an optionalfieldandremediation, so an agent can correct its own payload. - Passages are kept out of near-duplicate consolidation, re-cut when a memory's body changes (mint first, retire after, so readers see a stale span rather than a gap), retired when the memory is deleted, and restamped when only scope changes.
sibyl debug queryis usable for aggregates and builtins; work-item content stays in sync with its description; the dev script waits for the API only while its process is alive.
Versioning and CLI
- Every response carries
X-Sibyl-Version, plusX-Sibyl-Min-Clientwhen an operator configuresminimum_client_version. Clients sendX-Sibyl-Client-Version. - The CLI raises
ClientTooOldErrorwhen it is below the server's declared floor, and otherwise warns once per server version per 24 hours on stderr (! Server is <x>, this CLI is <y> — run sibyl update) so--jsonoutput stays clean. sibyl healthandsibyl updateshow the server version alongside the CLI and PyPI versions, flagging when the server is ahead.- API images self-identify:
SIBYL_VERSION,SIBYL_GIT_COMMIT, andSIBYL_GIT_DIRTYare passed as build args and promoted to runtime env, withVERSIONcopied into the runtime stage as a fallback. Without these the server reported0.0.0/unknownand drift detection could not work.
Measurement and Benchmarks
Every lever below was screened against a same-commit baseline with paired replication, and the verdicts ship with the code.
- A1 chunk geometry (shipped). New
benchmarks/longmemeval_v2_chunk_geometry/harness. Straddle rate is zero: 31,244 of 31,244 (question, phrase, carrier-state) triples land inside a single slice. A 3-adjacent-slice window reaches the fat-state ceiling exactly (95.5% enterprise / 100% web, versus fat 95.5% / 100%; single-slice 93.2% / 97.9%). Part confinement costs nothing: all three arms agree digit for digit at widths 1, 2, 3, 5 in both domains, with an emptyconfinement_lossesset and a positive control that separates the arms (2,555/2,555 and 1,623/1,623 versus 0). Oversize-leaf fallback is 114 of 83,850 enterprise slices (0.14%) and zero on web. Header tax is 19% / 12%, of which the breadcrumb is 11.5% on enterprise. - Stage-2 re-baseline anchor: combined 30.38% (enterprise 31.60%, web 29.31%) under 3-pass paired replication at post-merge HEAD.
- A2 (both levers dead, shipped off). Semantic-prior rescue recovered 0/8 misses and minted two new ones; entity-overlap typed-pool ordering produced byte-identical outcomes, and the historical −3.7pp was entirely the confounded reservation widening. The identifier-flank probe produced the structural finding: the benchmark corpora declare no retrieval keys, so the exact-key lane is inert there by construction, even though verbatim fulltext beats dense-alone +10pp at k=10 on identifier queries. Closing that flank needs key declaration at ingest, which is a corpus rebuild.
- A3 bounded agentic traversal (screened, closed for 1.2). The arm allows at most 2 widening rounds on top of the seed pass (inside the verbs' three-round budget), one follow-up search, and an 8s gathering deadline under the gate's 15s p95 latency ceiling. Over all 211 enterprise questions with activity receipts in every row (traversal armed 211/211, widening used in 209): pack coverage −4 questions against a gate of ≥ +6 with a −2 regression floor, paired latency +28.4s median against a ≤ +4s screen line, and the sufficiency stop never fired (0 early stops in 211) — the loop gathers indiscriminately. Screen FAIL, no paid run made.
- Usage-signal rerank feasibility (NO-GO). 14,740 events (14,523 exposure, 211 citation), 2,470 distinct exposed items, 154 cited, a 1.45% citation rate and a 425-second median exposure-to-citation lag. The usage-prior arm's CI straddles zero (
[-0.037, +0.043]), and the query is not recoverable from storage, so no query-conditioned reranker is possible without new instrumentation. - Team-scope trust receipt is now machine-generated (
tools/trust/team_scope_gate.py): 45 probes (29 deny, 16 allow) across six read surfaces, run through the real capture and read paths against ephemeral stores, with 14 independent failure sites including leaks, allow-direction blackouts, surface disagreement, stamped-scope mismatch, and owner forgery (5 offered, 0 surviving). The hand-maintained receipt it replaces contained manually typed zeros.
Deprecations and Removals
retrieval_mode=accurateis deprecated. It measured lower accuracy (−4pp enterprise) at roughly 2.5× the latency offast. The value still works for one release and now returns an in-bandretrieval_mode_deprecatednotice; the server logsretrieval_mode_accurate_deprecatedwith remediation. Move toretrieval_mode=fast(the default).sibyl_core.retrieval.query_planningremoved, along withplan_evidence_queries,plan_evidence_queries_with_usage,EvidenceQueryPlan,EvidenceQueryFacet,EvidencePlanningReceipt, andMAX_SUPPLEMENTAL_QUERIES. It was the orphaned LLM query planner for accurate mode with no live callers.bitnami/kubectl:1.30.7is gone from the SurrealDB chart, replaced byalpine/k8s:1.34.1. Docker Hub's bitnami catalog purge removed the versioned tags, so the old image can never be pulled again.
Infrastructure and CI
- SurrealDB subchart 0.4.0 → 0.5.0, which fixes the duplicate
storageClassName/selectorrendering the wrapper used to work around.terminationGracePeriodSecondsis set to 120 so RocksDB can finish flushing on node drains instead of losing the compaction tail to the 30s default. - The SurrealDB image is stated explicitly instead of inheriting a default, and quickstart/Ansible compose image tags move to
1.2.0. - Container scanning is fixed for multi-arch: Trivy is told the platform (
TRIVY_PLATFORM=linux/<arch>) instead of defaulting tolinux/amd64and failing on arm64 before reading any packages, and the workflow now expects 4 SBOMs (one per image per architecture) rather than 2. - Dataset fetches fail at the download rather than later at the checksum. Release notes are generated with Opus 5.
- Toolchain moves: ruff 0.16 formatting adopted, ty 0.0.63 inference satisfied,
mcppinned to>=1.29.0,<2because FastMCP is removed in 2.0, plus typer 0.27, anthropic 0.120.2, greenlet 3.5.4, TypeScript 7.0.2, and a full workspace dependency refresh that clears the nanoid advisory. - New docs:
docs/cli/export.md,docs/api/mcp-traverse.md, the fresh-store schema path withsibyld db init,benchmarks/usage_rerank/FEASIBILITY.md,benchmarks/longmemeval_v2_chunk_geometry/README.md, plus the 1.1 implementation plan and the 1.3 strategy skeleton.
Upgrade Notes
- Schema migrates 16 → 19 on start: the
memory_scopecolumn and index (v17), retrieval-key columns with an element index (v18), and theidx_entity_labelselement-form rebuild (v19). The label index rebuild runsCONCURRENTLYbecause a single-transaction rebuild fails on RocksDB when every row carries labels; expect it to complete in the background on large graphs. - Backfill scope on existing graphs. Run
sibyld migrate scope-backfill --dry-runfirst, then--apply. Read the skip counts: rows with no readable scope equivalent are deliberately left unstamped rather than having their readers revoked. A non-zerostampable_remainingmeans concurrent writes landed behind the cursor, so re-run against a quiet graph.--reverserestores exactly what the forward pass overwrote. - Stop sending
retrieval_mode=accurate. It is deprecated in this release and slated for removal. - Drop any import of
sibyl_core.retrieval.query_planning. The module is gone. - Rebuild custom API images with the version build args (
SIBYL_VERSION,SIBYL_GIT_COMMIT,SIBYL_GIT_DIRTY), otherwise the server reports0.0.0/unknownand client drift detection cannot fire. - Helm and compose: re-pull chart dependencies for SurrealDB 0.5.0, confirm your values do not still reference
bitnami/kubectl, and bump image tags to1.2.0if you pin them. - Reindexing note for writers: passage ids are deterministic per parent and index, and they regenerate when a body changes. Cite
parent_idfromfetch_slice, not the span id. - Optional new levers are off by default.
knn_type_overfetch,semantic_prior_rescue_weight, and thetyped_entity_overlappool all failed their gates on the benchmark corpora; enable them only with your own measurements.