Repository navigation
Engram v0.24.0
Release candidate prepared for the v0.24.0 GitHub Release (v0.21.2 remains the latest published release until the signed v0.24.0 tag is pushed; registry publication to crates.io, PyPI, and npm is a separate decision).
Security
palace_visualizesaving is a filesystem write — a call withoutput_pathnow requiresscoped_writewithinENGRAM_EXPORT_BASE_DIR, otherwisemaintenance, likememory_export_markdown, andpermission_mode_statusexposesoutput_pathinwrite_flag_params. Paths outside the export base are refused before any directory is created,..components are rejected unconditionally, active SQLite artifacts are refused both before directory creation and after parent canonicalization across CLI and MCP callers, symlinks and multi-link files (nlink > 1) are rejected, and outputs are published via an owner-only (0600) sibling temp file and atomic rename.GET /v1/eventsis workspace-scoped for restricted principals — a keyless (anonymous loopback) stream opened withoutworkspacenow receivesdefaultevents only instead of every workspace's events and content previews.- Context assembly and graph expansion stay in scope —
memory_build_context,memory_get_injection_prompt, andmemory_prepare_contextnormalize and authorizeworkspace,workspaces, andfilters.workspaceconsistently with central MCP permission checks, keep restricted principals within their allowed workspace, advancedepth2–3 graph frontiers only through authorized in-scope memories, and filterinclude_graphcrossrefsso foreign-workspace edges are never exposed. - Keyless HTTP and CORS align on
HostandOrigin— without an API key,/mcp,/v1/mcpand/v1/eventsanswer403to requests whoseHostorOrigindoes not name this machine (DNS rebinding, cross-site requests).ENGRAM_HTTP_ALLOWED_HOSTSextends theHostallowlist; blankENGRAM_CORS_ORIGINSnormalizes to unset, empty comma tokens and emptyOriginheaders are rejected, and bothCorsLayerpreflight (OPTIONS) and keyless checks accept case-insensitive loopbackOriginURIs (localhost,127.0.0.1,[::1], any port) by default; see MCP_AUTH. - Multimodal path confinement and handler scope guards (
A-5,A-6,A-7,Q-A) —memory_search_by_imagerequiresENGRAM_MEDIA_BASE_DIRinread_only/scoped_writemodes (escalating tomaintenancewhen unset or blank), all media path validators refuse active SQLite artifacts before canonicalization,memory_digestauthorizes innercontext_build_bundlecalls,harness_statusrestrictsinclude_gitto unrestricted writers, andmemory_agent_statusscopestotal_memoriesto the target workspace. - Bundled SQLite 3.54.1 (RUSTSEC-2026-0459, CVE-2026-119375) —
rusqlite0.31 → 0.41.1 andlibsqlite3-sys0.28.0 → 0.39.1 in the root andfuzz/lockfiles. The previous bundled SQLite 3.45.0 could return uninitialized page-cache memory for a database opened with a-walof a different page size. No schema, MCP or SDK change;rusqlitekeeps the same feature surface (cacheandfallible_uintare now explicit opt-ins). - Markdown filesystem import (RISK-0011) — scoped writers require an administrator-configured root; unconfined reads require maintenance even during review. Bounded directory-relative reads reject links, special files and SQLite artifacts before opening regular files. Existing IDs are authorized by their stored workspace, cross-workspace ID overwrites against an explicit target
workspaceare rejected, and confirmed batches authorize/classify/apply within one transaction. Pre-commit failures roll back; completion uncertainty is explicit. See compatibility notes. - Markdown filesystem export —
memory_export_markdownis a write operation: scoped writers need an administrator-configuredENGRAM_EXPORT_BASE_DIR; unconfined destinations require maintenance privileges. Linux/macOS directory-relative writes reject unsafe components, symlinks, hard links, active SQLite artifacts and directory ACLs, and publish private temporary files by atomic rename. Other platforms fail closed. See compatibility notes. - Batch tools enforce per-item workspaces —
memory_create_batchandmemory_ingest_fact_batchauthorize every item'sworkspace(central check, gRPC pre-check and handlers), closing a cross-workspace write introduced by the search/auth hardening commite04d3307. Batch item arrays are derived from the catalog schema and fail closed on unreadable nestedworkspaceclaims. - Resource subscriptions are principal-aware —
resources/subscribeandresources/unsubscribeapply the same workspace policy asresources/read. memory_search_suggestis workspace-scoped — restricted principals get spelling corrections built only from their own workspace vocabulary.permission_denial_for_modefails closed for unknown tools.- One
tools/callunwrap point — HTTP and gRPC unwrap the envelope once and authorize the same bare arguments the dispatcher sees;name/argumentskeys inside real tool arguments are no longer treated as an envelope. A caller's per-call_permission_modeself-restriction is still applied by the dispatcher, so its denial stays in the tool result (HTTP 200). - Log redaction (RISK-0002) — sync/cloud, hooks, multimodal,
engram-serverHTTP/WebSocket transport startup errors, and the watcher (including its daemon) no longer log bucket/object keys, workspace and tool names, filesystem paths,engram_url, dry-run memory content, subprocess stderr or raw error text; errors are logged by class (ENGRAM_LOG_ERROR_DETAIL=1restores detail for debugging). - Cryptographic release-tag verification (
R3) — Addedscripts/verify-release-tag.shanddocs/releases/allowed-signers, wired into.github/workflows/sdk-release.ymlacrosspreflight,publish-python, andpublish-npmwith anti-self-authorization and anti-ref-shadowing (materializing both verifier script and trust store from canonicalrefs/remotes/origin/mainvia--end-of-options), and hardenedscripts/sign-candidate-manifest.shoption/PEM/leading-dash guards. - Lifecycle, policy, compression, and server hook authorization (
Unit 1,Unit 3) — Added 12 lifecycle/policy/compression by-ID tools (memory_set_lifecycle,memory_lifecycle_update,memory_score,memory_promote,memory_explain,memory_reconcile_conflict,memory_promote_to_permanent,memory_boost,memory_set_expiration,memory_compress,memory_decompress,memory_compress_for_context) toID_SCOPED_TOOLSwith in-transactionensure_memory_accesschecks and single-IDdecayscoping;engram-serverlifecycle hooks now enforce_permission_mode/ENGRAM_PERMISSION_MODE, requirePermission::WriteonResourceType::Memory, reject unauthorized fallback to"default", and skipPostToolUsereinforcement on error responses. - Dependency advisories — zip 9.0.5 and duckdb 1.10506.1 (RUSTSEC-2026-0367/0368/0369), tokio 1.53.3 (RUSTSEC-2026-0386), rustls-webpki 0.103.16, webpki-roots 1.0.11 and ctutils 0.4.6 (RUSTSEC-2026-0382/0381/0385), rustix 1.1.6 (RUSTSEC-2026-0486), quinn-proto 0.11.19 (RUSTSEC-2026-0523/0524), and quinn-udp 0.5.16 (RUSTSEC-2026-0538); crypto-bigint 0.5 removed from the graph (RUSTSEC-2026-0387) by not enabling
aws-sdk-s3/sigv4a.
Added
- RFC 0009, 0010, and 0011 Implemented — Promoted and wired MCP Contract Validation (
docs/rfcs/0009-mcp-contract-validation.md), Permission Modes (docs/rfcs/0010-permission-modes.md), and Deterministic Model Routing (docs/rfcs/0011-model-routing.md,governance/models.toml,scripts/select-model.py). - MCP Error Normalizer (
G-6,G-7,G-10) — Addedsrc/mcp/error_normalizer.rsand upgradedcontext/blocks.rs,compression.rs,dream.rs,memory_crud/read_update_delete.rs, andsearch.rsso wrong-typedid/limit/offsetparameters and domain errors emit RFC 0006 structuredToolErrorpayloads (invalid_params,memory_not_found,validation_error). - Zero-Drift Python & TypeScript SDKs (
v0.7.0,G-2,G-4,G-5) — Aligned all Python and TypeScript SDK resource methods with the MCP server tool registry, routedresources/*calls over JSON-RPC (_rpc_call/rpcCall), ratcheteddocs/quality/sdk-contract-drift-baseline.jsonfrom 161 entries down to0(known_drift: []), and added TypeScriptisError: truestructuredEngramError(message, code, details)plustimeout/connection_errorparity. - Server Lifecycle Hooks & Cloud Sync Worker (
#237,C6) — WiredSessionStartHandler,PostToolUseHandler, andSyncWorker::start_backgroundintoengram-serverstartup and shutdown lifecycle.
Changed
- Search (from
e04d3307): catalog defaultlimitis 20,offsetandinclude_archivedare accepted,SearchOptions.offsetis public, andmemory_search_by_identity/memory_session_searchare read-only. Python and TypeScript SDKs defaultsearchto limit 20 and acceptoffset(Python alsoinclude_archived). - HNSW is opt-in —
hnsw_enableddefaults tofalseuntil the benchmark required by RFC 0003 exists; enable it withENGRAM_HNSW_ENABLED/--hnsw-enabled. When off, no index is built at startup or maintained on writes. HNSW widening stops after two empty hydration rounds. - Snapshot loader reads archive entry names from the central directory and rejects undecodable names.
- DuckDB bundled engine 1.4.4 → 1.5.6;
sqlite-vecis a dev-dependency (benchmarks only). - CI pins cargo-audit 0.22.0 and cargo-outdated 0.19.0 (
--locked); the TypeScript live lane uses Node 24; the docs job runs the public-claims, advertised-contracts and release-version checks. bump-version.pyverifies lockfiles and__version__and adds a read-only--release-planmode.- Miri smoke uses 256 deterministic Unicode oracle cases; native tests retain all 20,000 cases and the explicit adversarial matrix runs in both modes.
- Breaking cache-clear response:
memory_cache_clearnow returns{success, entries_cleared}for global, explicit-workspace and principal-bound calls. Consumers readingcleared,scopeorworkspacemust migrate;search_cache_clearkeeps its existing response. See the migration guide. - Workspace-scoped
media_assetsschema migrationv49(#235) — Rebuiltmedia_assetswith compositePRIMARY KEY (workspace, file_hash)andidx_media_assets_workspace_memoryso identical media bytes ingested in different workspaces remain strictly isolated. - Bounded multimodal media reads (
#236) — EnforcedMAX_MEDIA_FILE_BYTES = 50 MiB(open_bounded_media_file+Read::take) before hashing or reading media files insrc/storage/media_ops.rsandsrc/mcp/handlers/multimodal.rs. - Cloud sync supply chain & atomic snapshot upload (
#239,#240,G1) — Upgradedaws-sdk-s3(1.129.0),aws-config(1.8.15), andrustls(0.23.46), removingRUSTSEC-2026-0253,RUSTSEC-2026-0346, andRUSTSEC-2026-0349from the default/cloud dependency graph; isolatedCloudStorage::uploadsnapshots into unique sibling tempfiles (Invariante #27). - SQLite read-path & entity extraction performance (
P1,G1) — Sampledsqlite3_total_changes64aroundStorage::with_connectionto skip redundant-wal/-shmlstatsyscalls when no writes occurred (memory_get/by_id15.18 µs), added concurrentwith_read_connectionCriterion benchmarks (readers_readonly_82.94 ms, 226× faster than write-serializedreaders_8), and optimizedEntityExtractor(entity_extractor_new1.82 ns,extract_mixed4.50 µs, 4.75× faster). - WAL replay/recovery hardening (C2) — Replay validates every frame before touching the target: a 64 GiB byte budget (
ReplayLimits, injectable via the new*_with_limitsvariants), page caps, the SQLite WAL checksum chain, salts and ordering. Recovery stages to an owner-only (0600) file beside the target and replaces the target atomically only after fsync andintegrity_check. Rust API: new public fieldsWalDelta::chain_seedandWalDeltaPack::chain(both#[serde(default)]) and a newWalReplicationError::ReplayBudgetExceededvariant. Struct literals and exhaustive matches need updating. Fail-closed: delta packs created before this change have no chain context and are refused at replay. Uncompressed packs are now subject to the 64 MiB per-pack decode cap. - Storage concurrency (C2) — Each migration step runs in its own
BEGIN IMMEDIATEtransaction with a version recheck, so concurrent opens no longer double-apply steps and a failed step is rolled back.Storage::with_transactionnow uses IMMEDIATE transactions, so a writer waits up tobusy_timeoutinstead of failing with "database is locked". A database whose schema version is newer than this build'sSCHEMA_VERSIONis still refused without changes (now pinned by a regression test). - Workspace authorization for restricted principals (C1) — Restricted principals (anonymous loopback HTTP/gRPC and namespaced tokens) are refused, with
permission_deniedanddetails.reason = "tool_not_workspace_scoped", on the 65 by-ID tools that take noworkspaceparameter (for examplememory_related,memory_traverse,graph_query,memory_get_full), which previously ran unscoped. Tool-catalog metadata tools such asdiscover_toolsstay allowed. stdio and keyed HTTP clients are unaffected. Memory IDs are authorized against the persisted row's workspace. - Unknown tools under a restricted permission mode (C1) — Calling an unknown tool name when the principal or environment permission mode is below
adminnow returnspermission_denied(admin required) instead oftool_not_found. - Bounded time inputs (C3, Q2F) —
session_index.ttl_daysmust be within0..=36500(negative values used to fall back to 24 h and now return an error). Second- and day-based TTL, age and lookback inputs (context_record_artifactttl_seconds/stale_after_seconds,memory_boostduration_seconds,memory_archive_oldmax_age_days,memory_get_working_memorysince_minutes,memory_set_expiration) are bounded to about 100 years and rejected with an error instead of panicking. A natural-language lookback out of range applies no date filter and setsignored_date_filter. - Byte-bounded truncation (C3) — Text truncation is char-boundary safe and bounded in bytes: session chunk truncation (
truncate_with_marker) now guaranteeslen() <= max_charsin bytes, so multibyte text keeps fewer characters than before. replication_recoveron a live database (G1) — When the source is the active database, recovery restores the latest state only (from aVACUUM INTOsnapshot); frame- or time-based point-in-time recovery from the live database is refused. Copy the database first if PITR is needed.memory_ingest_mediaretry dedup (C6) — Re-ingesting the same file bytes in the same (normalized) workspace returns the existing live memory withdeduplicated: trueand ignores the retry's content, tags and importance; deleted memories are not revived. With schemav49(#235),media_assetsis scoped by(workspace, file_hash).defer_embedding = trueenqueues (C7) — Every producer (MCPmemory_create, CLI create and interactive create, snapshot load, dream promotion) now enqueues a background embedding job; previously such memories were never embedded while health reported Healthy. Withfalse, the memory is embedded right after commit and no stale pending job is left behind.rebuild_derived_indexesrepairs memories flagged as pending without a job row.- Storage errors are surfaced (Q2F) —
delete_crossrefreturns statement errors instead ofOk, andlist_memories/memory_listfail with an error naming the undecodable row id (undecodable memory row (id N)) instead of silently omitting it. engram-cli mcp install(Q2F, final fixes) — Refuses to overwrite a client config that is not a strict-JSON object (or whosemcpServersis not an object) unless--forceis given, keeps every backup under a unique name (an identical backup is reused), writes atomically (temp file + rename), and exits non-zero if any target fails. The temp file and backups carry the config's permission bits (0600 for a new config), and a symlinked config is written through to its target so the link survives.- Breaking Markdown import safe review-mode default (
C1, RFC 0004) —engram-cli import markdown, RustImportOptions::default(), Pythonclient.vault.vault_import(), and TypeScriptclient.vault.import()now default to non-mutating review/dry-run mode (confirm=false), matching the MCPmemory_import_markdowncontract (which now also reports"dry_run": !confirm). Pass--confirm(CLI) orconfirm=True/{ confirm: true }(Rust/SDKs) to apply changes (--dry-runremains accepted on the CLI as a deprecated alias). - Retention policy validation & enforcement (PR review,
Unit 3) —retention_policy_setvalidatesmax_age_days,compress_after_daysandauto_delete_after_dayswithin1..=36500,max_memories > 0,compress_max_importancein0.0..=1.0,compress_min_access >= 0, and validMemoryTypenames inexclude_types, while preserving existingcompress_max_importance/compress_min_accesson partial updates.retention_policy_applyenforcesmax_age_days, scopescompress_after_daysstrictly to the policy's workspace, honorsexclude_typesacross all stages, and skips legacy out-of-range rows without blocking other workspaces. - AAAK
losslessround trip (PR review) —losslesscompression is verified on every call: when dictionary substitution would not decompress to the exact input (text that already contains shorthands such asw/ore.g., word forms the dictionary cannot restore, surrounding whitespace), the input is stored unchanged under a new[AAAK:v1:verbatim]header, whichdecompressreturns as-is. - Multimodal output capture (PR review) — A read error on
ffprobe/ffmpegoutput, or a reader that does not finish, now fails the call with<bin> <stream> capture failed: ...instead of returning empty or silently truncated output. - Observability (O1) — HTTP responses carry an
x-request-idheader (a caller-supplied value is echoed only if it is 1-64 chars of[A-Za-z0-9._-]).mcp_success_totalno longer counts JSON-RPC errors orisErrortool results returned with HTTP 200. Logs carry error classes instead of error text, paths or panic payloads unlessENGRAM_LOG_ERROR_DETAIL=1is set; the gRPC transport no longer returns the handler panic payload in itsInternalstatus message.
Fixed
memory_consolidate_batchno longer reports merges it never made — consolidation only proposes actions, butdry_run=false(and the auto-consolidation scheduler) returnedduplicate_merged/conflict_resolvedactions as if applied, recorded the run as non-dry, and emittedconsolidationenrichment events with statuscompleted. Every report is now a dry run, the response addsapplied: falseand anote, no enrichment events are emitted, and the scheduler logs proposals. Actually applying merges is not implemented yet.memory_embedding_migrateno longer fakes a migration — withoutdry_runit computed embeddings, discarded them, relabelledembedding_modeltotarget_model, swallowed errors and returnedsuccess: true. It now returns an error and changes nothing;dry_run=truestill counts affected memories and reports storage errors instead of0. Usememory_rebuild_embeddingsto queue re-embedding with the active model. Both Pythonembedding_migrate(dry_run=True, target_model=...)and TypeScriptadmin.embeddingMigrate({ dryRun: true, targetModel })expose the count mode and raise/throwEngramErrorwhendry_runis omitted orfalse; the TypeScript SDK declaresengines.node >= 22.12.0to match its test/type toolchain.- Markdown entity grouping safely encodes internal tags (
session:,category:,file:, etc.) and reserved names instead of rejecting the workspace. Export uses POSIX fsync with one final strong flush per device, avoiding repeated macOS device flushes; public Markdown wrappers propagate structured errors. - Markdown export excludes expired memories and ineligible references, reads data/relations in one snapshot, propagates decode/query errors, sorts ties deterministically and marks incoming relations explicitly. No success is reported for an incomplete batch; atomicity is per file.
memory_cache_clearreturns{success, entries_cleared}for every scope.permission_mode_statusreports write-flag escalation (write_flag_params,required_mode_with_write_flags,allowed_with_write_flags).- Harness:
check-live-state.shrequires the newest PASS review while the active task has none and matches review versions by exact slug;harness-risk-register.shallocates ids past RISK-0008 and validates under Psych 4. - The HNSW index is reconciled with SQLite (expiry, supersede, merge, GC, content rewrites) on drain/cleanup cycles and compacts tombstones live; warmup no longer replays stale vectors or checkpointed deleted ids.
- Deleted (single, cascade, batch) and content-edited memories are removed from the in-memory HNSW index, saving the index no longer persists removed vectors, and HNSW semantic search widens its bounded candidate fetch so
limitholds after filtering. dream_consolidation_statusanddream_insightswere documented as always available but requiredream-phase; tool feature gating now has a single source (catalog::required_features).CrossEncoderRerankerreturns an explicit "not implemented" error instead of fabricating scores, and model routing no longer reports the placeholder as operational.- SDK release workflow no longer hardcodes
v0.22.0/0.5.0: it derives the core tag and SDK versions from the manifests and re-checks them after environment approval;sdks/typescript/package-lock.jsonsynced to 0.7.0. - SDK artifact metadata maps the npm registry channel to TypeScript compatibility and excludes its own output on retries, so generation works for both channels and is repeatable.
- Quality-candidate scripts no longer use empty
exceptblocks; the Rust risk inventory parser returns one tuple shape. - Hierarchical scope trees now attach children to their direct parents, preserving sibling branches and orphan roots; previously every stored scope could be returned as a separate root.
- PDF worker coverage profile under
RLIMIT_FSIZE = 0— Integration tests now spawnengram-pdf-workerwithLLVM_PROFILE_FILE=/dev/nullso coverage-instrumented builds do not fail withSIGXFSZon exit. - macOS
SQLITE_READONLY(1544)-shmrace retry (#238) —Storage::with_connectionandStorage::with_read_connectionnow retry transientSQLITE_READONLY(1544) errors when concurrent readers open-shmin WAL mode on macOS. - SQLite artifact permission hardening dropped POSIX locks (G1) — Restricting permissions on the database, WAL and SHM files used open/close on the live files, which drops the process's POSIX locks and could lose WAL frames. Permissions are now set by path (
fchmodatwithout following symlinks), and caller-controlled paths that resolve to the active database are refused. - Unicode and hex panics & mutation-loop termination (
C3,R1) — Search compaction, context grouping, gardening, compression budget checks and attestation hex parsing no longer panic on multibyte text or short hex input;src/text_util.rsdelegates UTF-8 boundary search to stdlib methods and uses slice-head destructuring inLowercaseIndex::find_window, eliminating all fourcargo-mutantstimeout loops. - Offline consolidation, document ingest, storage, and sync hardening (
S1,Units 1–5) — FixedOfflineConsolidatorRFC3339 cutoff/timestamp formatting, added read-onlydry_runpreview tomemory_consolidate, wrapped per-group consolidation in SQLSAVEPOINTs, attached the'consolidated'tag on archive, and excludedtranscript_chunk/ expired / soft-deleted / empty-word-set rows. FixedDocumentIngestorcharacter-based word-boundary chunking with high overlap, Markdown block/GFM table extraction, intra-document chunk deduplication, and split-UTF-8BoundedPdfTextbuffering. ClampedStoragePool::new(":memory:", N)to 1 connection, prevented integer overflow inStorage::compactfree-space checks, bounded/deduplicatedSyncWorkerlast_erroracross crash loops, enforced active-DB inode guards on sync push/pull, addedTempDownloadGuardinCloudStorage, retriedEINTRinmultimodal/process.rs, wiredrealtime/meilisearchintoengram-server, and made harnessmerge_gate_ci.pyfail closed (CiRefusal) onGitError.
Removed
- S3 multi-region access point (SigV4a) signing.
- Turso/libSQL backend (breaking Rust/build change) — Completed the v0.24.0 sunset from ADR-STORAGE-20260816-1. Removed the
tursoCargo feature, optionallibsqldependency,storage::turso_backendmodule, andTursoBackend/TursoConfigexports. Builds requesting--features tursoand Rust imports of those types must change. SQLite WAL remains the canonical store, with optional S3/R2 backup sync through the existingcloudfeature. The legacy backend had a separate schema runner; existing Turso databases are not automatically converted or guaranteed to open as Engram SQLite databases. See the migration guide.
Docs
- Public claims corrected: measured latency (FTS5 p50 3.2 ms, hybrid p50 250 ms at 100K×384d) instead of "sub-millisecond"; nonexistent model "native adapters" and unsourced competitor tool counts removed; gRPC and ONNX marked optional; tool counts are 301 catalog / 254 default build.
- RFC, ADR, plan and spec statuses record their implementation evidence; ROADMAP tool names corrected and cross-host federation marked not implemented.
- Shared decision-triage guide trimmed for publication.
- Optional server variables documented (
ENGRAM_CLEANUP_INTERVAL,ENGRAM_GRPC_BIND_ADDRESS,ENGRAM_WS_*,MEILISEARCH_*); vector-search docs corrected (HNSW opt-in, sqlite-vec benchmarks only); Rust risk inventory regenerated without the removed Turso backend.
Full changelog: v0.23.0...v0.24.0