Skip to content

v0.24.0

Latest

Choose a tag to compare

@github-actions github-actions released this 14 Nov 17:05
· 16 commits to main since this release
f4be2c4

Engram v0.24.0

Release candidate prepared for the v0.24.0 GitHub Release (v0.21.2 remains the latest published release until the signed v0.24.0 tag is pushed; registry publication to crates.io, PyPI, and npm is a separate decision).

Security

  • palace_visualize saving is a filesystem write — a call with output_path now requires scoped_write within ENGRAM_EXPORT_BASE_DIR, otherwise maintenance, like memory_export_markdown, and permission_mode_status exposes output_path in write_flag_params. Paths outside the export base are refused before any directory is created, .. components are rejected unconditionally, active SQLite artifacts are refused both before directory creation and after parent canonicalization across CLI and MCP callers, symlinks and multi-link files (nlink > 1) are rejected, and outputs are published via an owner-only (0600) sibling temp file and atomic rename.
  • GET /v1/events is workspace-scoped for restricted principals — a keyless (anonymous loopback) stream opened without workspace now receives default events only instead of every workspace's events and content previews.
  • Context assembly and graph expansion stay in scope — memory_build_context, memory_get_injection_prompt, and memory_prepare_context normalize and authorize workspace, workspaces, and filters.workspace consistently with central MCP permission checks, keep restricted principals within their allowed workspace, advance depth 2–3 graph frontiers only through authorized in-scope memories, and filter include_graph crossrefs so foreign-workspace edges are never exposed.
  • Keyless HTTP and CORS align on Host and Origin — without an API key, /mcp, /v1/mcp and /v1/events answer 403 to requests whose Host or Origin does not name this machine (DNS rebinding, cross-site requests). ENGRAM_HTTP_ALLOWED_HOSTS extends the Host allowlist; blank ENGRAM_CORS_ORIGINS normalizes to unset, empty comma tokens and empty Origin headers are rejected, and both CorsLayer preflight (OPTIONS) and keyless checks accept case-insensitive loopback Origin URIs (localhost, 127.0.0.1, [::1], any port) by default; see MCP_AUTH.
  • Multimodal path confinement and handler scope guards (A-5, A-6, A-7, Q-A) — memory_search_by_image requires ENGRAM_MEDIA_BASE_DIR in read_only/scoped_write modes (escalating to maintenance when unset or blank), all media path validators refuse active SQLite artifacts before canonicalization, memory_digest authorizes inner context_build_bundle calls, harness_status restricts include_git to unrestricted writers, and memory_agent_status scopes total_memories to the target workspace.
  • Bundled SQLite 3.54.1 (RUSTSEC-2026-0459, CVE-2026-119375) — rusqlite 0.31 → 0.41.1 and libsqlite3-sys 0.28.0 → 0.39.1 in the root and fuzz/ lockfiles. The previous bundled SQLite 3.45.0 could return uninitialized page-cache memory for a database opened with a -wal of a different page size. No schema, MCP or SDK change; rusqlite keeps the same feature surface (cache and fallible_uint are now explicit opt-ins).
  • Markdown filesystem import (RISK-0011) — scoped writers require an administrator-configured root; unconfined reads require maintenance even during review. Bounded directory-relative reads reject links, special files and SQLite artifacts before opening regular files. Existing IDs are authorized by their stored workspace, cross-workspace ID overwrites against an explicit target workspace are rejected, and confirmed batches authorize/classify/apply within one transaction. Pre-commit failures roll back; completion uncertainty is explicit. See compatibility notes.
  • Markdown filesystem export — memory_export_markdown is a write operation: scoped writers need an administrator-configured ENGRAM_EXPORT_BASE_DIR; unconfined destinations require maintenance privileges. Linux/macOS directory-relative writes reject unsafe components, symlinks, hard links, active SQLite artifacts and directory ACLs, and publish private temporary files by atomic rename. Other platforms fail closed. See compatibility notes.
  • Batch tools enforce per-item workspaces — memory_create_batch and memory_ingest_fact_batch authorize every item's workspace (central check, gRPC pre-check and handlers), closing a cross-workspace write introduced by the search/auth hardening commit e04d3307. Batch item arrays are derived from the catalog schema and fail closed on unreadable nested workspace claims.
  • Resource subscriptions are principal-aware — resources/subscribe and resources/unsubscribe apply the same workspace policy as resources/read.
  • memory_search_suggest is workspace-scoped — restricted principals get spelling corrections built only from their own workspace vocabulary.
  • permission_denial_for_mode fails closed for unknown tools.
  • One tools/call unwrap point — HTTP and gRPC unwrap the envelope once and authorize the same bare arguments the dispatcher sees; name/arguments keys inside real tool arguments are no longer treated as an envelope. A caller's per-call _permission_mode self-restriction is still applied by the dispatcher, so its denial stays in the tool result (HTTP 200).
  • Log redaction (RISK-0002) — sync/cloud, hooks, multimodal, engram-server HTTP/WebSocket transport startup errors, and the watcher (including its daemon) no longer log bucket/object keys, workspace and tool names, filesystem paths, engram_url, dry-run memory content, subprocess stderr or raw error text; errors are logged by class (ENGRAM_LOG_ERROR_DETAIL=1 restores detail for debugging).
  • Cryptographic release-tag verification (R3) — Added scripts/verify-release-tag.sh and docs/releases/allowed-signers, wired into .github/workflows/sdk-release.yml across preflight, publish-python, and publish-npm with anti-self-authorization and anti-ref-shadowing (materializing both verifier script and trust store from canonical refs/remotes/origin/main via --end-of-options), and hardened scripts/sign-candidate-manifest.sh option/PEM/leading-dash guards.
  • Lifecycle, policy, compression, and server hook authorization (Unit 1, Unit 3) — Added 12 lifecycle/policy/compression by-ID tools (memory_set_lifecycle, memory_lifecycle_update, memory_score, memory_promote, memory_explain, memory_reconcile_conflict, memory_promote_to_permanent, memory_boost, memory_set_expiration, memory_compress, memory_decompress, memory_compress_for_context) to ID_SCOPED_TOOLS with in-transaction ensure_memory_access checks and single-ID decay scoping; engram-server lifecycle hooks now enforce _permission_mode / ENGRAM_PERMISSION_MODE, require Permission::Write on ResourceType::Memory, reject unauthorized fallback to "default", and skip PostToolUse reinforcement on error responses.
  • Dependency advisories — zip 9.0.5 and duckdb 1.10506.1 (RUSTSEC-2026-0367/0368/0369), tokio 1.53.3 (RUSTSEC-2026-0386), rustls-webpki 0.103.16, webpki-roots 1.0.11 and ctutils 0.4.6 (RUSTSEC-2026-0382/0381/0385), rustix 1.1.6 (RUSTSEC-2026-0486), quinn-proto 0.11.19 (RUSTSEC-2026-0523/0524), and quinn-udp 0.5.16 (RUSTSEC-2026-0538); crypto-bigint 0.5 removed from the graph (RUSTSEC-2026-0387) by not enabling aws-sdk-s3/sigv4a.

Added

  • RFC 0009, 0010, and 0011 Implemented — Promoted and wired MCP Contract Validation (docs/rfcs/0009-mcp-contract-validation.md), Permission Modes (docs/rfcs/0010-permission-modes.md), and Deterministic Model Routing (docs/rfcs/0011-model-routing.md, governance/models.toml, scripts/select-model.py).
  • MCP Error Normalizer (G-6, G-7, G-10) — Added src/mcp/error_normalizer.rs and upgraded context/blocks.rs, compression.rs, dream.rs, memory_crud/read_update_delete.rs, and search.rs so wrong-typed id/limit/offset parameters and domain errors emit RFC 0006 structured ToolError payloads (invalid_params, memory_not_found, validation_error).
  • Zero-Drift Python & TypeScript SDKs (v0.7.0, G-2, G-4, G-5) — Aligned all Python and TypeScript SDK resource methods with the MCP server tool registry, routed resources/* calls over JSON-RPC (_rpc_call / rpcCall), ratcheted docs/quality/sdk-contract-drift-baseline.json from 161 entries down to 0 (known_drift: []), and added TypeScript isError: true structured EngramError(message, code, details) plus timeout/connection_error parity.
  • Server Lifecycle Hooks & Cloud Sync Worker (#237, C6) — Wired SessionStartHandler, PostToolUseHandler, and SyncWorker::start_background into engram-server startup and shutdown lifecycle.

Changed

  • Search (from e04d3307): catalog default limit is 20, offset and include_archived are accepted, SearchOptions.offset is public, and memory_search_by_identity / memory_session_search are read-only. Python and TypeScript SDKs default search to limit 20 and accept offset (Python also include_archived).
  • HNSW is opt-in — hnsw_enabled defaults to false until the benchmark required by RFC 0003 exists; enable it with ENGRAM_HNSW_ENABLED / --hnsw-enabled. When off, no index is built at startup or maintained on writes. HNSW widening stops after two empty hydration rounds.
  • Snapshot loader reads archive entry names from the central directory and rejects undecodable names.
  • DuckDB bundled engine 1.4.4 → 1.5.6; sqlite-vec is a dev-dependency (benchmarks only).
  • CI pins cargo-audit 0.22.0 and cargo-outdated 0.19.0 (--locked); the TypeScript live lane uses Node 24; the docs job runs the public-claims, advertised-contracts and release-version checks.
  • bump-version.py verifies lockfiles and __version__ and adds a read-only --release-plan mode.
  • Miri smoke uses 256 deterministic Unicode oracle cases; native tests retain all 20,000 cases and the explicit adversarial matrix runs in both modes.
  • Breaking cache-clear response: memory_cache_clear now returns {success, entries_cleared} for global, explicit-workspace and principal-bound calls. Consumers reading cleared, scope or workspace must migrate; search_cache_clear keeps its existing response. See the migration guide.
  • Workspace-scoped media_assets schema migration v49 (#235) — Rebuilt media_assets with composite PRIMARY KEY (workspace, file_hash) and idx_media_assets_workspace_memory so identical media bytes ingested in different workspaces remain strictly isolated.
  • Bounded multimodal media reads (#236) — Enforced MAX_MEDIA_FILE_BYTES = 50 MiB (open_bounded_media_file + Read::take) before hashing or reading media files in src/storage/media_ops.rs and src/mcp/handlers/multimodal.rs.
  • Cloud sync supply chain & atomic snapshot upload (#239, #240, G1) — Upgraded aws-sdk-s3 (1.129.0), aws-config (1.8.15), and rustls (0.23.46), removing RUSTSEC-2026-0253, RUSTSEC-2026-0346, and RUSTSEC-2026-0349 from the default/cloud dependency graph; isolated CloudStorage::upload snapshots into unique sibling tempfiles (Invariante #27).
  • SQLite read-path & entity extraction performance (P1, G1) — Sampled sqlite3_total_changes64 around Storage::with_connection to skip redundant -wal/-shm lstat syscalls when no writes occurred (memory_get/by_id 15.18 µs), added concurrent with_read_connection Criterion benchmarks (readers_readonly_8 2.94 ms, 226× faster than write-serialized readers_8), and optimized EntityExtractor (entity_extractor_new 1.82 ns, extract_mixed 4.50 µs, 4.75× faster).
  • WAL replay/recovery hardening (C2) — Replay validates every frame before touching the target: a 64 GiB byte budget (ReplayLimits, injectable via the new *_with_limits variants), page caps, the SQLite WAL checksum chain, salts and ordering. Recovery stages to an owner-only (0600) file beside the target and replaces the target atomically only after fsync and integrity_check. Rust API: new public fields WalDelta::chain_seed and WalDeltaPack::chain (both #[serde(default)]) and a new WalReplicationError::ReplayBudgetExceeded variant. Struct literals and exhaustive matches need updating. Fail-closed: delta packs created before this change have no chain context and are refused at replay. Uncompressed packs are now subject to the 64 MiB per-pack decode cap.
  • Storage concurrency (C2) — Each migration step runs in its own BEGIN IMMEDIATE transaction with a version recheck, so concurrent opens no longer double-apply steps and a failed step is rolled back. Storage::with_transaction now uses IMMEDIATE transactions, so a writer waits up to busy_timeout instead of failing with "database is locked". A database whose schema version is newer than this build's SCHEMA_VERSION is still refused without changes (now pinned by a regression test).
  • Workspace authorization for restricted principals (C1) — Restricted principals (anonymous loopback HTTP/gRPC and namespaced tokens) are refused, with permission_denied and details.reason = "tool_not_workspace_scoped", on the 65 by-ID tools that take no workspace parameter (for example memory_related, memory_traverse, graph_query, memory_get_full), which previously ran unscoped. Tool-catalog metadata tools such as discover_tools stay allowed. stdio and keyed HTTP clients are unaffected. Memory IDs are authorized against the persisted row's workspace.
  • Unknown tools under a restricted permission mode (C1) — Calling an unknown tool name when the principal or environment permission mode is below admin now returns permission_denied (admin required) instead of tool_not_found.
  • Bounded time inputs (C3, Q2F) — session_index.ttl_days must be within 0..=36500 (negative values used to fall back to 24 h and now return an error). Second- and day-based TTL, age and lookback inputs (context_record_artifact ttl_seconds/stale_after_seconds, memory_boost duration_seconds, memory_archive_old max_age_days, memory_get_working_memory since_minutes, memory_set_expiration) are bounded to about 100 years and rejected with an error instead of panicking. A natural-language lookback out of range applies no date filter and sets ignored_date_filter.
  • Byte-bounded truncation (C3) — Text truncation is char-boundary safe and bounded in bytes: session chunk truncation (truncate_with_marker) now guarantees len() <= max_chars in bytes, so multibyte text keeps fewer characters than before.
  • replication_recover on a live database (G1) — When the source is the active database, recovery restores the latest state only (from a VACUUM INTO snapshot); frame- or time-based point-in-time recovery from the live database is refused. Copy the database first if PITR is needed.
  • memory_ingest_media retry dedup (C6) — Re-ingesting the same file bytes in the same (normalized) workspace returns the existing live memory with deduplicated: true and ignores the retry's content, tags and importance; deleted memories are not revived. With schema v49 (#235), media_assets is scoped by (workspace, file_hash).
  • defer_embedding = true enqueues (C7) — Every producer (MCP memory_create, CLI create and interactive create, snapshot load, dream promotion) now enqueues a background embedding job; previously such memories were never embedded while health reported Healthy. With false, the memory is embedded right after commit and no stale pending job is left behind. rebuild_derived_indexes repairs memories flagged as pending without a job row.
  • Storage errors are surfaced (Q2F) — delete_crossref returns statement errors instead of Ok, and list_memories / memory_list fail with an error naming the undecodable row id (undecodable memory row (id N)) instead of silently omitting it.
  • engram-cli mcp install (Q2F, final fixes) — Refuses to overwrite a client config that is not a strict-JSON object (or whose mcpServers is not an object) unless --force is given, keeps every backup under a unique name (an identical backup is reused), writes atomically (temp file + rename), and exits non-zero if any target fails. The temp file and backups carry the config's permission bits (0600 for a new config), and a symlinked config is written through to its target so the link survives.
  • Breaking Markdown import safe review-mode default (C1, RFC 0004) — engram-cli import markdown, Rust ImportOptions::default(), Python client.vault.vault_import(), and TypeScript client.vault.import() now default to non-mutating review/dry-run mode (confirm=false), matching the MCP memory_import_markdown contract (which now also reports "dry_run": !confirm). Pass --confirm (CLI) or confirm=True / { confirm: true } (Rust/SDKs) to apply changes (--dry-run remains accepted on the CLI as a deprecated alias).
  • Retention policy validation & enforcement (PR review, Unit 3) — retention_policy_set validates max_age_days, compress_after_days and auto_delete_after_days within 1..=36500, max_memories > 0, compress_max_importance in 0.0..=1.0, compress_min_access >= 0, and valid MemoryType names in exclude_types, while preserving existing compress_max_importance / compress_min_access on partial updates. retention_policy_apply enforces max_age_days, scopes compress_after_days strictly to the policy's workspace, honors exclude_types across all stages, and skips legacy out-of-range rows without blocking other workspaces.
  • AAAK lossless round trip (PR review) — lossless compression is verified on every call: when dictionary substitution would not decompress to the exact input (text that already contains shorthands such as w/ or e.g., word forms the dictionary cannot restore, surrounding whitespace), the input is stored unchanged under a new [AAAK:v1:verbatim] header, which decompress returns as-is.
  • Multimodal output capture (PR review) — A read error on ffprobe/ffmpeg output, or a reader that does not finish, now fails the call with <bin> <stream> capture failed: ... instead of returning empty or silently truncated output.
  • Observability (O1) — HTTP responses carry an x-request-id header (a caller-supplied value is echoed only if it is 1-64 chars of [A-Za-z0-9._-]). mcp_success_total no longer counts JSON-RPC errors or isError tool results returned with HTTP 200. Logs carry error classes instead of error text, paths or panic payloads unless ENGRAM_LOG_ERROR_DETAIL=1 is set; the gRPC transport no longer returns the handler panic payload in its Internal status message.

Fixed

  • memory_consolidate_batch no longer reports merges it never made — consolidation only proposes actions, but dry_run=false (and the auto-consolidation scheduler) returned duplicate_merged/conflict_resolved actions as if applied, recorded the run as non-dry, and emitted consolidation enrichment events with status completed. Every report is now a dry run, the response adds applied: false and a note, no enrichment events are emitted, and the scheduler logs proposals. Actually applying merges is not implemented yet.
  • memory_embedding_migrate no longer fakes a migration — without dry_run it computed embeddings, discarded them, relabelled embedding_model to target_model, swallowed errors and returned success: true. It now returns an error and changes nothing; dry_run=true still counts affected memories and reports storage errors instead of 0. Use memory_rebuild_embeddings to queue re-embedding with the active model. Both Python embedding_migrate(dry_run=True, target_model=...) and TypeScript admin.embeddingMigrate({ dryRun: true, targetModel }) expose the count mode and raise/throw EngramError when dry_run is omitted or false; the TypeScript SDK declares engines.node >= 22.12.0 to match its test/type toolchain.
  • Markdown entity grouping safely encodes internal tags (session:, category:, file:, etc.) and reserved names instead of rejecting the workspace. Export uses POSIX fsync with one final strong flush per device, avoiding repeated macOS device flushes; public Markdown wrappers propagate structured errors.
  • Markdown export excludes expired memories and ineligible references, reads data/relations in one snapshot, propagates decode/query errors, sorts ties deterministically and marks incoming relations explicitly. No success is reported for an incomplete batch; atomicity is per file.
  • memory_cache_clear returns {success, entries_cleared} for every scope.
  • permission_mode_status reports write-flag escalation (write_flag_params, required_mode_with_write_flags, allowed_with_write_flags).
  • Harness: check-live-state.sh requires the newest PASS review while the active task has none and matches review versions by exact slug; harness-risk-register.sh allocates ids past RISK-0008 and validates under Psych 4.
  • The HNSW index is reconciled with SQLite (expiry, supersede, merge, GC, content rewrites) on drain/cleanup cycles and compacts tombstones live; warmup no longer replays stale vectors or checkpointed deleted ids.
  • Deleted (single, cascade, batch) and content-edited memories are removed from the in-memory HNSW index, saving the index no longer persists removed vectors, and HNSW semantic search widens its bounded candidate fetch so limit holds after filtering.
  • dream_consolidation_status and dream_insights were documented as always available but require dream-phase; tool feature gating now has a single source (catalog::required_features).
  • CrossEncoderReranker returns an explicit "not implemented" error instead of fabricating scores, and model routing no longer reports the placeholder as operational.
  • SDK release workflow no longer hardcodes v0.22.0 / 0.5.0: it derives the core tag and SDK versions from the manifests and re-checks them after environment approval; sdks/typescript/package-lock.json synced to 0.7.0.
  • SDK artifact metadata maps the npm registry channel to TypeScript compatibility and excludes its own output on retries, so generation works for both channels and is repeatable.
  • Quality-candidate scripts no longer use empty except blocks; the Rust risk inventory parser returns one tuple shape.
  • Hierarchical scope trees now attach children to their direct parents, preserving sibling branches and orphan roots; previously every stored scope could be returned as a separate root.
  • PDF worker coverage profile under RLIMIT_FSIZE = 0 — Integration tests now spawn engram-pdf-worker with LLVM_PROFILE_FILE=/dev/null so coverage-instrumented builds do not fail with SIGXFSZ on exit.
  • macOS SQLITE_READONLY (1544) -shm race retry (#238) — Storage::with_connection and Storage::with_read_connection now retry transient SQLITE_READONLY (1544) errors when concurrent readers open -shm in WAL mode on macOS.
  • SQLite artifact permission hardening dropped POSIX locks (G1) — Restricting permissions on the database, WAL and SHM files used open/close on the live files, which drops the process's POSIX locks and could lose WAL frames. Permissions are now set by path (fchmodat without following symlinks), and caller-controlled paths that resolve to the active database are refused.
  • Unicode and hex panics & mutation-loop termination (C3, R1) — Search compaction, context grouping, gardening, compression budget checks and attestation hex parsing no longer panic on multibyte text or short hex input; src/text_util.rs delegates UTF-8 boundary search to stdlib methods and uses slice-head destructuring in LowercaseIndex::find_window, eliminating all four cargo-mutants timeout loops.
  • Offline consolidation, document ingest, storage, and sync hardening (S1, Units 1–5) — Fixed OfflineConsolidator RFC3339 cutoff/timestamp formatting, added read-only dry_run preview to memory_consolidate, wrapped per-group consolidation in SQL SAVEPOINTs, attached the 'consolidated' tag on archive, and excluded transcript_chunk / expired / soft-deleted / empty-word-set rows. Fixed DocumentIngestor character-based word-boundary chunking with high overlap, Markdown block/GFM table extraction, intra-document chunk deduplication, and split-UTF-8 BoundedPdfText buffering. Clamped StoragePool::new(":memory:", N) to 1 connection, prevented integer overflow in Storage::compact free-space checks, bounded/deduplicated SyncWorker last_error across crash loops, enforced active-DB inode guards on sync push/pull, added TempDownloadGuard in CloudStorage, retried EINTR in multimodal/process.rs, wired realtime/meilisearch into engram-server, and made harness merge_gate_ci.py fail closed (CiRefusal) on GitError.

Removed

  • S3 multi-region access point (SigV4a) signing.
  • Turso/libSQL backend (breaking Rust/build change) — Completed the v0.24.0 sunset from ADR-STORAGE-20260816-1. Removed the turso Cargo feature, optional libsql dependency, storage::turso_backend module, and TursoBackend / TursoConfig exports. Builds requesting --features turso and Rust imports of those types must change. SQLite WAL remains the canonical store, with optional S3/R2 backup sync through the existing cloud feature. The legacy backend had a separate schema runner; existing Turso databases are not automatically converted or guaranteed to open as Engram SQLite databases. See the migration guide.

Docs

  • Public claims corrected: measured latency (FTS5 p50 3.2 ms, hybrid p50 250 ms at 100K×384d) instead of "sub-millisecond"; nonexistent model "native adapters" and unsourced competitor tool counts removed; gRPC and ONNX marked optional; tool counts are 301 catalog / 254 default build.
  • RFC, ADR, plan and spec statuses record their implementation evidence; ROADMAP tool names corrected and cross-host federation marked not implemented.
  • Shared decision-triage guide trimmed for publication.
  • Optional server variables documented (ENGRAM_CLEANUP_INTERVAL, ENGRAM_GRPC_BIND_ADDRESS, ENGRAM_WS_*, MEILISEARCH_*); vector-search docs corrected (HNSW opt-in, sqlite-vec benchmarks only); Rust risk inventory regenerated without the removed Turso backend.

Full changelog: v0.23.0...v0.24.0