Skip to content

v2.8.0 "Strong memories stay whole"

Latest

Choose a tag to compare

@samvallad33 samvallad33 released this 05 Sep 09:18
· 3 commits to main since this release
56265ea

The release the Sep 4 deep audit asked for. The ingest gate stops merging new
notes into strong memories (the "weld" class behind two incidents this month),
failures push back on what was just retrieved, and every MCP tool now carries
honest behaviour hints on a leaner surface. Around that: the Memory PR
quarantine decides by subject instead of by word, usearch 2.26 and Rust 1.98.1
land with Windows and Android compiled in CI for the first time, libgit2's four
CVEs are closed, and the hardening pass from Aaron Garcia's independent audit
of v2.7.0 ships in full.

Security

  • libgit2 1.9.7 (libgit2-sys 0.18.3+1.9.2 to 0.18.8+1.9.7). libgit2 1.9.5
    fixed CVE-2026-53584 (submodule path escape), CVE-2026-53585 (unbounded
    allocation from a delta size header), CVE-2026-53586 (authentication callback
    host confusion) and CVE-2026-53587; 1.9.7 followed. Vestige builds git2
    with vendored OpenSSL, so the fix ships inside every release binary.

Fixed — The ingest gate now honours memory strength

  • smart_ingest computed whether the closest existing memory was strong
    (retrieval strength above 0.85) and never read it, so a confirmed,
    high-strength record could be appended to by any similar note, the same
    way a weak one could. That is the "weld" class behind the Sep 2 and Sep 4
    incidents. Yang, Duncan and Barense (2026) found prediction-error updating
    intrudes new material into weak memories only, never strong ones, with
    memory age irrelevant. The gate now stores a similar note as its own memory
    and links it to the strong one instead of merging into it
    (CreateReason::ProtectedStrongMemory, on by default through
    PredictionErrorConfig::protect_strong_memories). Near-identical content
    still reinforces, because reinforcing never touches content.

Added — Failures push back on what was just retrieved

  • Post-retrieval failure feedback (Heinbockel, Leicht, Wagner and Schwabe,
    eLife 2025). When a memory that reads as a failure is ingested, the memories
    retrieved in the previous thirty minutes of receipts lose retrieval strength
    in proportion to their rank in those receipts, at most 0.10 each, never
    below the floor, same scope only, once per failure and memory. Every delta
    is written to the new failure_feedback ledger (migration V33) and
    revert_failure_feedback undoes it exactly. This is the first mechanism by
    which a wrong memory leaves the top of recall without anyone demoting it by
    hand. Opt-in for this release: set
    VESTIGE_FAILURE_FEEDBACK=1. It demotes memories on its own, so it becomes
    the default only once real-store data shows the demotions land on the right
    ones.
  • Retroactive Salience Backfill now also runs live, through the same pipeline
    and receipts as the backfill tool, the moment a failure is ingested. The
    paper it is ported from (Zaki et al., Nature 2025) found offline
    co-reactivation linking stronger during wake than sleep, so leaving it to the
    consolidation pass alone was the weaker schedule. Same kill switch as the
    consolidation auto-fire, VESTIGE_BACKFILL_AUTOFIRE=0. Results of both hooks
    are returned under failureHooks in the ingest response.

Changed — MCP surface

  • Every advertised tool now carries a title and all four MCP behaviour hints
    (readOnlyHint, destructiveHint, idempotentHint, openWorldHint),
    serialised explicitly so clients never fall back on the spec defaults that
    assume the worst. recall, receipt, memory_status, and session_start
    are read-only (they record receipts and access statistics, never memory
    content); memory, intention, maintain, and dedup are destructive;
    only source_sync reaches outside the local store. graph stays writable
    because label records an outcome. A unit test pins the exact sets and the
    e2e suite checks the wire shape.
  • 45 tool and property descriptions were rewritten to say the same thing in
    fewer words (3,710 bytes removed), and codebase now advertises its
    verify action. With the hints added, tools/list went from 38,474 to
    35,019 bytes. The rest of the weight is schema structure and is tracked as
    its own issue.
  • recall with detail_level: "brief" now returns receiptId without the
    full receipt and without the contextReinstatement block. The receipt is
    persisted and one receipt call away. Default and full responses are
    unchanged.

Fixed — Memory PR gate

  • A single sensitive word deep inside a long note no longer quarantines the
    write. The sensitive_topic signal fires when a tag names the topic, a
    credential-shaped value sits in the content, the write is short (60 words
    or fewer), the topic leads the text (first 12 words), or two distinct
    topics appear. On the real store the incidental case produced most holds:
    ordinary engineering notes that mentioned "token" or "identity" once were
    held for review and their authors had to come back for them. Five tests
    cover the branches, including a fake token fixture checked against the
    secret scanner. docs/CONFIGURATION.md states the rule.

Added — Android (Termux) build profile

  • The no-embeddings configuration (--no-default-features --features connectors,cloud-sync) is now a first-class build (#145). It is the
    configuration Termux users build from source, and the first time it was
    checked it carried eight dead-code warnings behind cfg gates the default
    build never sees. Each item is now gated to match its callers, and a new CI
    job (no-embeddings-build) runs clippy with warnings as errors plus the core
    and server test suites in that configuration, so it cannot rot again.
  • CI cross-compiles that profile for aarch64-linux-android with cargo-ndk
    (android-build, API 24, static C++ runtime) and fails if the binary links
    anything beyond bionic's libc, libm, libdl, and liblog. The artifact it
    uploads is the binary a Termux user runs, so a phone can test a pull request
    before a release exists.
  • codebase-git feature, on by default. libgit2 (with OpenSSL and libssh2) is
    now optional; a build without it keeps the whole codebase tool and answers
    git-history questions with "git history is not available in this build"
    instead of failing to compile. The git data types moved to a shared module so
    both variants expose the same API.
  • Honest status in builds without an embedding runtime: vestige-cli health
    says "not compiled into this build" instead of "Not Ready", memory_status
    and the dashboard report embeddingBackend, and smart_ingest responses
    carry "dedup": "unavailable in this build" when the prediction-error gate
    is compiled out.
  • docs/INSTALL-TERMUX.md: the from-source recipe for Android.

Changed — Dependencies

  • usearch =2.23.0 is unpinned to 2.26. The MSVC compile break that forced
    the pin (2.24.0 referenced the POSIX MAP_FAILED macro, unum-cloud/USearch#746)
    was fixed upstream in April 2026, and the fp16lib feature that 2.23.0 needed
    no longer exists: 2.26 carries its own scalar fp16 conversion and its headers
    have no #warning directive, so the MSVC fatal-error path that made fp16lib
    load-bearing is gone. Default features stay off so release binaries do not
    ship NumKong's AVX2/FMA dispatch kernels (#71). No re-embedding is needed.
  • CI now compiles the default feature set on Windows MSVC (windows-msvc-build)
    on every pull request and on main. Until now Windows was compiled only when a
    release tag was cut, which is how an MSVC-only break could ship (#93).

Changed — Toolchain and dependencies

  • The pinned Rust is 1.98.1 (from 1.97.1), the version Termux ships for
    Android, so contributors building from source on a phone (#145) compile with
    the same compiler CI does. The lints 1.98 adds under -D warnings
    (chunks_exact_to_as_chunks, drain_collect, useless_format) are fixed at
    their eleven sites; slice::as_chunks has been stable since 1.88, so
    rust-version = "1.91" still holds.
  • cargo update within semver: tokio 1.53, rustls 0.23.43, axum 0.8.9, hyper
    1.11, serde 1.0.229, openssl-src 3.6.3, libgit2-sys 1.9.7 (the CVE fix from
    #207), and roughly two hundred transitive crates. ONNX Runtime (ort
    2.0.0-rc.11) and fastembed 5.13.2 are deliberately held back: cargo update
    moves them to rc.13 and 5.17 together, which changes the runtime that gets
    downloaded at build time and is its own upgrade (#214).

Fixed — Test isolation

  • cargo test -p vestige-core --lib vector no longer fails three peer_*
    tests on every run. with_vector_search_disabled set
    VESTIGE_DISABLE_VECTOR_SEARCH in the process environment, so every other
    test thread building a Storage in that window silently got no vector index;
    the full suite passed only by scheduling luck. Both env-gated test helpers
    (vector search and VESTIGE_AUTO_CONSOLIDATE_MERGE) now pin a thread-local
    override that the gates read in test builds, ENV_LOCK is gone, and two
    tests assert that a sibling thread never sees the override.
  • The vector-search gate and its "why is it off" report now parse the
    variable the same way. VESTIGE_DISABLE_VECTOR_SEARCH=0 leaves the index on
    and is reported as on; before, any value made the report say disabled while
    the index stayed on.
    The remaining sections are the hardening pass driven by Aaron Garcia's
    (@aaronukgarcia) independent audit of v2.7.0, verified finding by finding against main before anything changed.
    Confirmed items are fixed below; the audit's migration-idempotence and WAL
    starvation findings did not reproduce (every migration already runs inside one
    IMMEDIATE transaction with its own version bump, and wal_autocheckpoint is
    on), so those got regression tests and a checkpoint hook rather than rewrites.

Fixed — Vector index

  • A long-lived MCP server process now sees exactly the vectors its sibling
    processes wrote (#181). The refresh that shipped in 2.7.0 noticed that
    something had changed (PRAGMA data_version) but then re-read every
    vector row for the active profile on every external commit, and it skipped
    any id already in its index, so a peer re-embedding an existing memory left
    a stale vector behind until restart. Migration V32 adds vector_journal, an
    append-only, trigger-fed record of every insert, update and delete on
    embedding_profile_vectors, keyed by an AUTOINCREMENT sequence that is
    allocated inside the writer's transaction and never reused. The refresh reads
    only the rows past its watermark, replaces re-embedded vectors, and drops
    purged ones; a process whose watermark has fallen behind the pruned journal
    reconciles against the table instead of trusting it. The journal holds ids
    only. Six tests cover it, including two stores open on one file, and each
    guard was negative-tested by reintroducing the defect it prevents.

Fixed — Concurrency

  • purge_node released its SQLite writer guard only after touching the
    vector-index lock, while activate_embedding_profile takes those two locks
    in the opposite order. A purge racing a profile activation could deadlock
    the process with no timeout. The writer is now dropped before the index is
    touched, matching every other combined lock site, and a threaded regression
    test races the two paths under a 30 s watchdog.

  • The dream tool and the dashboard POST /api/dream held the shared
    CognitiveEngine mutex across the whole synchronous O(n²) pairwise scan, so
    one dream stalled every other tool that takes that lock (explore,
    predict, session_context, memory_unified, autopilot, and more) for
    the duration. The dreamer is now snapshotted under a short lock (its state is
    Arc-shared, so the engine still records the run) and the scan runs on the
    blocking pool. MemoryDreamer gained Clone and synchronous
    *_blocking entry points; the async wrappers are unchanged.

  • Every writer transaction now begins IMMEDIATE. A DEFERRED transaction
    that reads before it writes can fail with SQLITE_BUSY_SNAPSHOT when another
    process (the CLI beside the MCP server) commits in between, and SQLite does
    not consult busy_timeout for that upgrade. BEGIN IMMEDIATE takes the
    write lock up front, where the 5 s busy timeout applies, and SQLite then
    guarantees no SQLITE_BUSY until COMMIT. A source-policy test keeps it
    that way, across every storage module rather than one file: the first
    version of that lint read sqlite.rs alone, and two writers sat in its
    blind spot. The open-time foreign-key repair (repair_cascade_orphans) read
    PRAGMA foreign_key_list and then DELETEd inside one unchecked
    DEFERRED transaction, so a CLI writing beside the server could fail the
    store open; the memory-PR decide path in trace_store opened DEFERRED and
    skipped the retry helper. Both go through begin_write_transaction now, and
    the lint reports the offending file and line.

  • The cross-encoder reranker (Jina v1 Turbo, ~150MB) was loaded under the
    shared CognitiveEngine mutex, one second after every startup. The load is
    synchronous: it downloads the model on a fresh install and reads it from
    disk on every later start. So the same lock the dream finding was about
    was held for the length of a 150MB download, stalling explore, predict,
    session_context, memory_unified and the rest, unconditionally rather
    than on demand, and it blocked a tokio worker thread while it ran. The load
    is now a Reranker::load_cross_encoder associated function that runs on the
    blocking pool holding nothing; the lock is taken only to install the result.

  • The HTTP transport held the global sessions write lock across
    handle_request().await while answering initialize, so every other client
    on the multi-agent transport queued behind one client's handshake. The slot
    is now reserved under one write-lock acquisition (the MAX_SESSIONS check
    and the insert still cannot race), the lock is released, and the request is
    handled with no global lock held. A handshake that produces no usable
    protocol version releases its reservation instead of leaking a session.

Fixed — Determinism and honest claims

  • The Observatory brain print (vb1-…) folded dueForReview into the hashed
    shape payload and into the review-pressure trait. That count is
    next_review <= now, so an untouched store re-keyed every time a card
    crossed its due date, breaking "same store twice, identical print". Only
    stored shape is hashed now; vector lane 2 is reserved (always 0). Existing
    prints re-key exactly once.
  • The salience-rescue walk without a receipt announced root cause found
    with a receipt-shaped line, while the receipt-backed walk said
    candidate cause found. Both now say candidate cause found, and the
    heuristic verdict's provenance line reads heuristic, no receipt. Nothing in
    the field asserts a root cause automatically.
  • ObservatoryStage now defaults to chrome="none". The 'full' overlays
    render real memory labels and the export/capture paths mount this component,
    so a future mount that forgets the prop can no longer start exporting labels
    by accident; opt in to 'full' explicitly (both routes already did).
  • Loop-export comments no longer promise a "byte-identical" clip. The rendered
    frames are pixel-identical across machines; the H.264 bytes depend on the
    encoder.

Fixed — Silent fallbacks now speak

  • schema_version is validated before any migration decision: a non-integer,
    negative, or missing version on a populated database fails closed with a
    message that names the corrupted row instead of replaying early ALTER
    migrations into a duplicate column loop, and a store newer than the binary
    logs a warning instead of nothing.
  • Vector-index rebuilds and vector loads count and log the rows they skip
    instead of dropping unreadable or undecodable embeddings silently. That
    includes the legacy node_embeddings mirror inside get_all_embeddings,
    which is the branch a store still on the legacy profile actually takes, and
    which kept dropping rows silently while the profile-table half above it
    logged; one summary line now covers both sources.
  • review_mode.json with an unknown mode (a typo for fast, say) now logs a
    warning naming the accepted labels instead of silently running risk-gated.
    ReviewMode::try_from_label is the strict parser behind it.
  • vestige.toml larger than 1 MiB is ignored with a warning instead of being
    read into memory on every startup.

Fixed — Second pass: every remaining audit item, reproduced or not

  • Every ALTER TABLE ... ADD COLUMN in every migration now runs through the
    idempotent add_column_if_missing guard, whatever version it belongs to
    (the runner splits them out of the migration body and applies the rest as
    one batch). The per-migration transaction already made crash replay clean;
    this additionally absorbs a database whose columns were added by an older
    runner or a pre-release build. A test half-applies every column-adding
    migration and proves the replay succeeds.
  • V28's manifest count is authoritative on replay (INSERT OR IGNORE kept a
    row a pre-release build wrote), and the legacy repair pass reconciles a
    stale vector_count on open even when no vector needs repair.
  • schema_version with more than one row logs a warning instead of silently
    taking MAX().
  • Writer transactions go through one helper: BEGIN IMMEDIATE, then three
    logged retries (100/200/400 ms) on BUSY/LOCKED past the busy timeout.
  • Every remaining filter_map(|r| r.ok()) on a SQLite row stream logs the
    row it skipped and the operation that skipped it (15 sites).
  • purge_node_in_transaction now fails closed. Its three reference sweeps
    (insights.source_memories, memory_receipts.payload, agent_traces. payload) selected the rows that still name the memory being erased and then
    dropped any unreadable one with filter_map(|row| row.ok()), so a row the
    scrub could not read kept its reference while the caller was told the purge
    succeeded. On an erasure path that is the one failure that must never be
    silent, so an unreadable row now aborts the whole purge and the transaction
    rolls back with nothing half-scrubbed. A payload that parses as neither a
    Receipt nor a MemoryTraceEvent still falls through to the raw-text
    sweeps, but says so in the log instead of vanishing. Regression test seeds
    an insight whose TEXT primary key holds a BLOB: on the old code the purge
    returned Ok, on the fix it refuses and leaves the memory intact.
  • MemoryStoreSend::insert indexes a supplied embedding under the active
    profile (vector table, has_embedding, in-memory index, legacy mirror) or
    fails loudly; it can no longer accept a vector and leave the memory
    unsearchable. The persistence is shared with the embedder path.
  • The secret scanner has a work budget: 16 MiB scanned (the tail is logged,
    never silently skipped) and at most 64 distinct findings per kind.
  • OrdinalDirectory exhaustion is a graceful limit: ensure returns None
    and the reality index counts the unindexed memory instead of panicking.
  • The MCP server runs a PASSIVE WAL checkpoint every 60 s and warns when
    the WAL stays large afterwards, and sweeps expired agent traces at startup
    as well as during consolidation.
  • Dashboard WebSocket subscribers that lag receive an explicit
    EventsDropped { missed } event instead of resuming mid-stream, the
    closed-channel case ends the socket cleanly, and the broadcast buffer grew
    from 1024 to 4096 events. The client logs the gap.
  • Observatory pick readback runs inside try/finally (the staging buffer is
    destroyed on every exit), rapid clicks share one in-flight readback, the
    stage disposes its NodeRenderer on unmount and before a re-create, and a
    disposed renderer ignores late results.
  • The WebGPU engine recovers from device.lost: it releases device-bound
    state and re-acquires the device with exponential backoff (0.5 s to 8 s,
    five attempts), reporting a recovering status; the canvas re-fires
    onready so the stage re-registers and re-uploads. After the last attempt
    it stays in error with the original reason.
  • Regression tests pin the stage's chrome default to 'none' and forbid
    any route from mounting it with chrome="full".
  • Every writer transaction in the storage layer now goes through
    begin_write_transaction, not just the ones in sqlite.rs. Thirteen
    writers in synaptic_store, replay_store and attestation_store began
    IMMEDIATE by hand, which is correct but silent: they took the write lock
    up front and then gave up on the first refusal past the 5 s busy timeout
    with nothing in the log to say a writer had lost a race. The policy lint now
    enforces the DEFERRED rule across nine storage modules and the
    helper-routing rule across five.
  • The dashboard event-channel capacity is defined once. The MCP binary and
    vestige-cli serve each hardcoded 4096 beside EVENT_CHANNEL_CAPACITY,
    so tuning the constant would have silently left two servers on the old
    value.

Added

  • The consolidation cycle ends with a PASSIVE WAL checkpoint, so long
    uptimes fold the .wal back into the main file between busy periods.
  • Migration regression test: applying every migration twice must apply nothing
    the second time and leave sqlite_master identical.
  • Device-loss recovery is covered by behaviour tests, not just its delay
    table: the loop reports a numbered recovering status per attempt, stops at
    the first successful re-acquire, keeps the original reason in the terminal
    error, refuses to re-acquire or paint an error once the engine is
    disposed, and runs one loop at a time when device.lost fires twice.

Fixed — MCP conformance

  • server/discover now answers with the DiscoverResult shape the 2026-07-28
    schema defines (supportedVersions, resultType, cacheScope, ttlMs,
    identity under _meta) instead of a home-grown protocolVersions object. A
    conforming client could not read our supported revisions from the old answer,
    concluded the server offered none, and tested the newest revision anyway,
    which is exactly what the reporter's harness did in #175. Vestige still does
    not advertise 2026-07-28. resources/templates/list returns an empty list
    instead of method-not-found, and an unrecognised pagination cursor on any list
    method is rejected with -32602 instead of being answered with page one.

Asset provenance. The macOS and Linux assets were built by the release workflow from the tagged commit (56265ea). The first Windows build failed on a dashboard test that had never run on Windows (stage-privacy.test.ts built paths with .pathname, see #231), so the Windows zip and checksum were built by the same workflow from fix/dashboard-tests-windows-paths at 715373d: the tagged commit plus that test fix and a CI change, with no Rust or dashboard source difference. The other six assets were rebuilt in the same run and replaced with identical files.