Skip to content

v1.2.0

Choose a tag to compare

@github-actions github-actions released this 13 Aug 19:54
· 437 commits to main since this release
f967882

Release Notes v1.2.0

Released: 2026-08-13

v1.2.0 changes what a memory is on the way in and what retrieval hands back on the way out. Memories are now cut into passages (verbatim spans of the parent body) that are first-class graph entities, writers can declare their own cut plan, exact-match keys, and self-verification probes, and readers can widen a result set with two bounded traversal verbs instead of asking for a bigger pack. Alongside that: memory scope becomes an indexed column with a reversible backfill, the graph loses several tablescans that were collapsing ingest, and every response now names the server version.

Highlights

Passages: a slice substrate for retrieval

Long memories are projected into EntityType.PASSAGE rows on the existing entity table, each one a verbatim span of its parent with a PART_OF edge back to it. Retrieval serves spans instead of whole fat bodies, and drops the parent only when the pack already holds every span of that projection (passage_covers_parent true and indices exactly range(total)). Partial coverage never hides a memory: if projection has to skip an oversized span, every surviving span is stamped passage_covers_parent: false and the parent stays servable.

The mechanical cutter (projection/slicing.py) packs siblings into a 600–1,200 char band with a 2,000 char hard ceiling, descends oversized subtrees one level, emits an oversize childless leaf whole rather than bisecting a literal, and merges a stranded sub-floor tail back into the slice before it. Slices partition the body lines exactly.

Writers declare structure: spans, retrieval keys, probes

remember / add and POST /entities accept four new declarations:

  • spans — an agent-authored cut plan of half-open {start, end, label} offsets into the stored body. Validated as verbatim and complete: 2 to MAX_AGENT_SPANS (64) spans, no gap, no overlap, each within MAX_AGENT_SPAN_CHARS (17,768). An invalid plan is refused, not silently rewritten; a stale plan falls back to the mechanical cutter and is stamped as such.
  • atomic — "this body is one unit," which suppresses cutting entirely.
  • retrieval_keys — up to MAX_RETRIEVAL_KEYS (16) identifiers the memory answers to (error strings, symbols, flags, aliases). Matching is casefold plus internal whitespace collapse and nothing else: ERR_DB_TIMEOUT will not match ERR_TIMEOUT. Stored as both display and normalized columns, element-indexed for CONTAINSANY.
  • probes — up to MAX_PROBES_PER_MEMORY (5) questions this memory must answer later. Each probe is run through the live retrieval path at write time (REHEARSAL_LIMIT 10, REHEARSAL_BUDGET_SECONDS 6.0) and the per-probe rank-or-absent verdict is returned on EntityResponse.probe_rehearsal. Retrievability becomes a measurement instead of an assumption.

A nightly job (replay_memory_probes, cron 0 5 * * *, 168-hour window, 200 memories per run) re-asks those probes and records probe_last_replay, so a memory that becomes unreachable as the graph evolves shows up as drift instead of silence.

Bounded traversal verbs for clients

Two new verbs let an agent widen deliberately instead of over-fetching up front:

  • expand_neighbors (POST /api/search/expand) — walks the bounded neighborhood of up to 8 seeds, depth 1–3, limit up to 24, preview content capped at 500 chars, with unresolved reported identically for missing and unreadable seeds.
  • fetch_slice (POST /api/search/slice) — reads one memory at span granularity, window default 3 up to 64 spans, returning parent_id for citation plus covers_parent, window_start, and passage_total.

Both are stateless per call, clamp rather than reject, authorize every row for the calling reader, and authorize a span's parent before serving the span. They return adjacency and previews only; context keeps composition. Documented in docs/api/mcp-traverse.md.

Deterministic memory export and a per-project handbook

sibyl export memory materializes a project into a .sibyl/memory/ tree: README.md, index.md, recent.md, tasks.md, notes/<type>-<slug>.md, optional handbook.md, and a manifest.json carrying per-file SHA-256 plus a content_digest over the whole projection. Byte-identical output for an identical graph is enforced by three rules: no wall-clock values, a whitelist of stable metadata fields, and total ordering by id (server response order cannot leak in). Files land read-only unless --writable; the writer refuses to overwrite a populated directory that has no prior manifest unless --force. Flags include --project, --output, --notes, --tasks, --task-status, --type, --hydrate/--no-hydrate, --handbook/--no-handbook, and --json. See docs/cli/export.md.

The handbook (GET /synthesis/handbook?project=<id>) composes five fixed sections: Orientation, Decisions In Force, Current Work, Gotchas, and Key Artifacts. Every line cites its source, each source is claimed by exactly one section (the one it scores highest for, ties going to the earlier section), and an empty heading is reported as a true statement about the graph rather than padded. There is no model call in the path, so it is composed per request and never cached: it cannot go stale against the graph it describes.

Memory scope becomes queryable, with a reversible backfill

memory_scope is promoted from the attributes bag to an indexed entity column (schema v17), so scope absence is now a fact rather than an ambiguity, and a scopeless write can no longer unscope the row it replaces. sibyld migrate scope-backfill stamps rows that predate scope stamping: --org-id, --dry-run/--apply, --reverse. Recovery from the originating raw capture runs before derivation from project_id, so a genuinely private memory is never widened to project scope. Scopes no reader could satisfy (team, delegated, organization, shared, public) are skipped and counted rather than stamped, and every stamped row records scope_backfill_source plus a scope_backfill_prior receipt that --reverse restores exactly. A concurrent write can no longer forge that rollback record.

Query performance: tablescans retired

Three hot paths stopped scanning the entity and relates_to tables:

  • Embedding-backfill UPDATE now addresses rows through an indexed group_id + uuid subquery with the if-current guards intact: 2.83s → 20–130ms per statement at 43K rows.
  • Bulk-upsert DELETE replaces uuid IN $list with per-uuid indexed deletes: 9.4s → 16ms for 700 uuids at 350K edges.
  • Record-id resolution maps per-uuid lookups instead of an IN-list: 3.3s → 128ms for 1,400 uuids at 167K entities.

idx_entity_labels moves to element form (labels.*, rebuilt CONCURRENTLY in schema v19) so label filters are index-served and correct on their own instead of scanning or silently returning nothing.

Retrieval and Ranking

  • Exact-key arm. A mechanical identifier detector (quoted strings, letter/digit mixes, underscores, :: scope separators, dotted segments, hex runs, case transitions, long --flags; bare all-caps acronyms deliberately excluded) fires an exact-key lane over retrieval_keys_normalized when RetrievalSignal.EXACT_KEY is planned, with a 1.4 boost applied once per candidate even when several lanes reach it.
  • Typed-overfetch KNN with exactness fallback. knn_type_overfetch (0–32, off by default) walks an untyped HNSW pool overfetch × limit deep and filters types on materialized rows, capped at KNN_TYPE_OVERFETCH_CAP 4,000 (warm-probed at 0.82s on a 95K-row namespace). If the filtered head fills the budget it is the typed KNN head; otherwise the classic typed bracket runs.
  • KNN effort floor. knn_search_effort computes max(1, configured_ef, k). Configured effort is a floor on quality, never a ceiling on requested rows, which fixes silent truncation when ef < k.
  • Per-lane seed budget scales with the request. seed_candidates_per_signal(limit) returns max(2, min(limit, 50)) instead of a flat per-lane depth. Seed budget is diversity budget, and once the unit is a passage a lane needs more rows to reach the same number of distinct sources.
  • A2 ranking levers ship opt-in and off by default: semantic_prior_rescue_weight (a coverage-gated prior that only lifts candidates with little lexical overlap) and the typed_entity_overlap pool ordering. Both failed their gate (see Measurement).
  • search now serves stored content rather than its derived blurb, and passages are always included in the requested type set.

Context Packs

  • Character-bounded composition. compose_operational_evidence accepts char_budget, admitting candidates in rank order while the running content total fits. Admission stops at the first candidate that does not fit rather than skipping ahead to a smaller one, so a pack is always a prefix of its ranking. Exposed as char_budget (1–500,000) on the evidence request. An item count only bounds payload while item size is stable, and a passage is roughly a twelfth of the state it was cut from.
  • The token budget is spent, not just trimmed to. markdown_token_budget now defaults to DEFAULT_MARKDOWN_TOKEN_BUDGET (4,000) and accepts up to 32,000. Given a budget, both breadth and per-item depth scale toward it (85% of the budget to content, the rest to structure) within hard ceilings.
  • Facet routing by parent type. A span of a decision is filed as decision material, not under a generic "passage" heading. Eligibility is settled before parent suppression, so a filtered-out span cannot suppress the memory it came from on its way out.
  • Degraded packs keep fidelity. The fallback search path no longer inherits search's 500-char preview default; it uses FALLBACK_SEARCH_CONTENT_MAX_CHARS (18,000). Degradation may cost freshness or ranking, but a caller cannot tell truncated content from short content.
  • Source expansion windows over adjacent passages, and each window run is sized by what that run actually holds.

Data Integrity and Reliability

  • Entity content cap enforced. MAX_ENTITY_CONTENT_CHARS (50,000) was declared in three layers and enforced in none; the live graph held a 520,802-character pasted transcript that skewed BM25 length normalization corpus-wide. Single and bulk writes now reject oversized bodies (rejection, not truncation); migrations can still reshape legacy rows.
  • Embedding backfill false negatives fixed. The currency fence now derives persisted text the same way hydration does (content → metadata.content → summary), so rows stored with an empty content column stop reporting as missing forever. Stale-but-present rows are reported as stale_entity_ids; only genuinely absent rows fail the batch.
  • Interrupted idempotency receipts recover on retry. A pending reservation observed under the serialization lock was interrupted, not in flight, so the claim is adopted and the handler re-executes instead of returning 409. A true in-flight duplicate still blocks at lock acquisition.
  • Document search degrades to scan when the embedder is soft-failing and both vector and lexical lanes come back empty.
  • Record identifiers stay in error messages. UUID scrubbing turned every "Task not found: <id>" into a field-less "Invalid request data." Ids are the system's public vocabulary; SQL, POSIX path, and Windows path scrubbing remain.
  • New unprocessable_entity() (422) factory passes the message through verbatim with an optional field and remediation, so an agent can correct its own payload.
  • Passages are kept out of near-duplicate consolidation, re-cut when a memory's body changes (mint first, retire after, so readers see a stale span rather than a gap), retired when the memory is deleted, and restamped when only scope changes.
  • sibyl debug query is usable for aggregates and builtins; work-item content stays in sync with its description; the dev script waits for the API only while its process is alive.

Versioning and CLI

  • Every response carries X-Sibyl-Version, plus X-Sibyl-Min-Client when an operator configures minimum_client_version. Clients send X-Sibyl-Client-Version.
  • The CLI raises ClientTooOldError when it is below the server's declared floor, and otherwise warns once per server version per 24 hours on stderr (! Server is <x>, this CLI is <y> — run sibyl update) so --json output stays clean.
  • sibyl health and sibyl update show the server version alongside the CLI and PyPI versions, flagging when the server is ahead.
  • API images self-identify: SIBYL_VERSION, SIBYL_GIT_COMMIT, and SIBYL_GIT_DIRTY are passed as build args and promoted to runtime env, with VERSION copied into the runtime stage as a fallback. Without these the server reported 0.0.0/unknown and drift detection could not work.

Measurement and Benchmarks

Every lever below was screened against a same-commit baseline with paired replication, and the verdicts ship with the code.

  • A1 chunk geometry (shipped). New benchmarks/longmemeval_v2_chunk_geometry/ harness. Straddle rate is zero: 31,244 of 31,244 (question, phrase, carrier-state) triples land inside a single slice. A 3-adjacent-slice window reaches the fat-state ceiling exactly (95.5% enterprise / 100% web, versus fat 95.5% / 100%; single-slice 93.2% / 97.9%). Part confinement costs nothing: all three arms agree digit for digit at widths 1, 2, 3, 5 in both domains, with an empty confinement_losses set and a positive control that separates the arms (2,555/2,555 and 1,623/1,623 versus 0). Oversize-leaf fallback is 114 of 83,850 enterprise slices (0.14%) and zero on web. Header tax is 19% / 12%, of which the breadcrumb is 11.5% on enterprise.
  • Stage-2 re-baseline anchor: combined 30.38% (enterprise 31.60%, web 29.31%) under 3-pass paired replication at post-merge HEAD.
  • A2 (both levers dead, shipped off). Semantic-prior rescue recovered 0/8 misses and minted two new ones; entity-overlap typed-pool ordering produced byte-identical outcomes, and the historical −3.7pp was entirely the confounded reservation widening. The identifier-flank probe produced the structural finding: the benchmark corpora declare no retrieval keys, so the exact-key lane is inert there by construction, even though verbatim fulltext beats dense-alone +10pp at k=10 on identifier queries. Closing that flank needs key declaration at ingest, which is a corpus rebuild.
  • A3 bounded agentic traversal (screened, closed for 1.2). The arm allows at most 2 widening rounds on top of the seed pass (inside the verbs' three-round budget), one follow-up search, and an 8s gathering deadline under the gate's 15s p95 latency ceiling. Over all 211 enterprise questions with activity receipts in every row (traversal armed 211/211, widening used in 209): pack coverage −4 questions against a gate of ≥ +6 with a −2 regression floor, paired latency +28.4s median against a ≤ +4s screen line, and the sufficiency stop never fired (0 early stops in 211) — the loop gathers indiscriminately. Screen FAIL, no paid run made.
  • Usage-signal rerank feasibility (NO-GO). 14,740 events (14,523 exposure, 211 citation), 2,470 distinct exposed items, 154 cited, a 1.45% citation rate and a 425-second median exposure-to-citation lag. The usage-prior arm's CI straddles zero ([-0.037, +0.043]), and the query is not recoverable from storage, so no query-conditioned reranker is possible without new instrumentation.
  • Team-scope trust receipt is now machine-generated (tools/trust/team_scope_gate.py): 45 probes (29 deny, 16 allow) across six read surfaces, run through the real capture and read paths against ephemeral stores, with 14 independent failure sites including leaks, allow-direction blackouts, surface disagreement, stamped-scope mismatch, and owner forgery (5 offered, 0 surviving). The hand-maintained receipt it replaces contained manually typed zeros.

Deprecations and Removals

  • retrieval_mode=accurate is deprecated. It measured lower accuracy (−4pp enterprise) at roughly 2.5× the latency of fast. The value still works for one release and now returns an in-band retrieval_mode_deprecated notice; the server logs retrieval_mode_accurate_deprecated with remediation. Move to retrieval_mode=fast (the default).
  • sibyl_core.retrieval.query_planning removed, along with plan_evidence_queries, plan_evidence_queries_with_usage, EvidenceQueryPlan, EvidenceQueryFacet, EvidencePlanningReceipt, and MAX_SUPPLEMENTAL_QUERIES. It was the orphaned LLM query planner for accurate mode with no live callers.
  • bitnami/kubectl:1.30.7 is gone from the SurrealDB chart, replaced by alpine/k8s:1.34.1. Docker Hub's bitnami catalog purge removed the versioned tags, so the old image can never be pulled again.

Infrastructure and CI

  • SurrealDB subchart 0.4.0 → 0.5.0, which fixes the duplicate storageClassName/selector rendering the wrapper used to work around. terminationGracePeriodSeconds is set to 120 so RocksDB can finish flushing on node drains instead of losing the compaction tail to the 30s default.
  • The SurrealDB image is stated explicitly instead of inheriting a default, and quickstart/Ansible compose image tags move to 1.2.0.
  • Container scanning is fixed for multi-arch: Trivy is told the platform (TRIVY_PLATFORM=linux/<arch>) instead of defaulting to linux/amd64 and failing on arm64 before reading any packages, and the workflow now expects 4 SBOMs (one per image per architecture) rather than 2.
  • Dataset fetches fail at the download rather than later at the checksum. Release notes are generated with Opus 5.
  • Toolchain moves: ruff 0.16 formatting adopted, ty 0.0.63 inference satisfied, mcp pinned to >=1.29.0,<2 because FastMCP is removed in 2.0, plus typer 0.27, anthropic 0.120.2, greenlet 3.5.4, TypeScript 7.0.2, and a full workspace dependency refresh that clears the nanoid advisory.
  • New docs: docs/cli/export.md, docs/api/mcp-traverse.md, the fresh-store schema path with sibyld db init, benchmarks/usage_rerank/FEASIBILITY.md, benchmarks/longmemeval_v2_chunk_geometry/README.md, plus the 1.1 implementation plan and the 1.3 strategy skeleton.

Upgrade Notes

  1. Schema migrates 16 → 19 on start: the memory_scope column and index (v17), retrieval-key columns with an element index (v18), and the idx_entity_labels element-form rebuild (v19). The label index rebuild runs CONCURRENTLY because a single-transaction rebuild fails on RocksDB when every row carries labels; expect it to complete in the background on large graphs.
  2. Backfill scope on existing graphs. Run sibyld migrate scope-backfill --dry-run first, then --apply. Read the skip counts: rows with no readable scope equivalent are deliberately left unstamped rather than having their readers revoked. A non-zero stampable_remaining means concurrent writes landed behind the cursor, so re-run against a quiet graph. --reverse restores exactly what the forward pass overwrote.
  3. Stop sending retrieval_mode=accurate. It is deprecated in this release and slated for removal.
  4. Drop any import of sibyl_core.retrieval.query_planning. The module is gone.
  5. Rebuild custom API images with the version build args (SIBYL_VERSION, SIBYL_GIT_COMMIT, SIBYL_GIT_DIRTY), otherwise the server reports 0.0.0/unknown and client drift detection cannot fire.
  6. Helm and compose: re-pull chart dependencies for SurrealDB 0.5.0, confirm your values do not still reference bitnami/kubectl, and bump image tags to 1.2.0 if you pin them.
  7. Reindexing note for writers: passage ids are deterministic per parent and index, and they regenerate when a body changes. Cite parent_id from fetch_slice, not the span id.
  8. Optional new levers are off by default. knn_type_overfetch, semantic_prior_rescue_weight, and the typed_entity_overlap pool all failed their gates on the benchmark corpora; enable them only with your own measurements.