Skip to content

v3.7.0 — agent logstream, integrity spine, and pluggable embeddings

Latest
Latest

Choose a tag to compare

View all tags
@igorls igorls released this 11 Aug 12:35
v3.7.0
7db7e0b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v3.7.0 — agent logstream, integrity spine, and pluggable embeddings

The headline of this release is agent logstream coordination: an append-only, local-first event layer so agents on different machines can delegate work, wait for replies, exchange patches, and acknowledge handoffs through the MemPalace hub — without a human relaying messages. Everything else in 3.7.0 hardens long-lived palaces under multi-session load (single-writer ownership, safer repair/re-mine, MCP lifecycle) and expands embed/search options (OpenAI-compatible embeddings, search date windows) without changing the local-first default.

Features

  • Agent logstream coordination (RFC 003). Append-only event layer for multi-agent work: durable task packets, wait/ack handoffs, patch and file artifacts, and live tailing over the MCP HTTP hub (GET /logstream/stream SSE). MCP tools include mempalace_event_append / list / wait / ack, artifact put/get, and patch submit; CLI mempalace logstream mirrors the core verbs. Events and artifacts stay local, verbatim, and separate from the drawer palace (logstream.sqlite3). Shared-brain / multi-machine agent fleets no longer need a human to relay status between hosts. (#2162, phases 1–5)

  • Logstream multi-master sync foundation (RFC 004 step 0). Estate-level logstream replication hooks so coordinated agents can share the coordination layer across palace replicas — the storage step for a replicated shared brain. (#2162, logsync)

  • OpenAI-compatible embeddings. Opt-in embedding_model: "openai-compat" talks to any /v1/embeddings endpoint (LM Studio, llama.cpp, vLLM, Ollama's OpenAI shim, or a self-hosted server) over stdlib urllib — no new dependency. Config and env vars set URL, model, and key; the model id is part of the embedder name so a model change forces a reindex. Default MiniLM/ONNX path is unchanged. (#1671, #1559)

  • Search date window. mempalace_search and mempalace search accept since / before ([since, before) on filed_at), shared with list_drawers via mempalace.date_window. Undated drawers are excluded while a bound is active; the vector candidate pool widens under a filter and reports truncation when the pool is full. (#2000, #463)

  • Hermes memory provider (core). In-package Hermes MemoryProvider files live turns through file_conversation_exchange() so metadata matches convo mining, with a background worker so the agent loop never blocks. Install/backfill/docs remain a stacked follow-up. (#1915, #2215)

  • RFC 002 source adapters on mine. mempalace mine <source> --source <adapter> resolves registered adapters, holds the palace writer lease for the full ingest, and keeps dry-runs inert. Legacy --mode paths are unchanged. (#2068, #2062)

  • Hook write-routing through the daemon. Background hook saves and mines can honor the shared write-routing policy so multi-session setups serialize mutations through one local owner. (#2030, #1963)

Performance

  • EmbeddingGemma groups documents by size before sub-batching, cutting padded-token work on long sweeps without changing vector meaning. Applies only to embedding_model: embeddinggemma. (#2104)

  • HNSW capacity probes are cached and invalidated by palace file signature, so repeated status/taxonomy paths no longer re-scan native segments every call. (#2051, #1471)

  • chunk_text line numbering is O(N), fixing multi-second hangs on large sources. (#2054, #2055)

Bug Fixes

  • Mining works again on the default Chroma backend. Explicit-embedding writes no longer hand Chroma np.float32 scalars that normalize_embeddings rejects. (#2187)

  • ChatGPT data exports are parsed as conversations, not stored as raw JSON arrays. (#2160)

  • Local backends enforce process-lifetime single-writer ownership. File-backed palaces require one writer owner for the process lifetime; read-only MCP may coexist; remote backends remain multi-process. (#2079, #2045)

  • MCP refuses writes when the served library drifts (mempalace, and chromadb when that backend is active) after an upgrade without restart. Opt out with MEMPALACE_MCP_ALLOW_STALE_LIBRARY. (#2081, #899)

  • Chroma HNSW write defaults match chromadb (batch_size=100 / sync_threshold=1000), retiring the old 2/2 bloat guard. (#2107, #2106)

  • Repair and recovery are safer under contention. Mine-lock before archive, preserve temp collections on failed swap, fail loud on truncated pagination; dry-run is a true preview in both legacy and from-sqlite modes; backups skip sockets/pipes/device nodes so a live palace socket no longer aborts repair/migrate. (#2109, #2086, #2087, #2133, #2144, #2207, #2212)

  • rebuild_index holds the palace writer lease for the full snapshot→rebuild/swap cycle. (#2195)

  • Orphaned per-source mine locks are reaped (age + nonblocking flock), throttled to once per 15 minutes; palace-level locks are untouched. (#2200)

  • HNSW divergence is preflighted before remaining count() crash sites across mine, dedup, migrate, repair, and palace helpers. (#2093)

  • Re-mine and conversation ingest no longer lose or duplicate drawers. Content-hash dedup, sweeper purge scope, round-trippable drawer_id on search; Claude Code subagents/ skipped by default (--include-subagents to opt in). (#2050, #2125, #2090, #1330, #1217)

  • MCP and daemon lifecycle harden multi-agent use. Read-only refuses config/checkpoint-ack host mutations; stdio exits on EOF; daemon defers lock-refused jobs; Windows stdout capture falls back or fails closed if the protocol stream cannot be restored. (#2126, #2072, #2029, #2211, #2210)

  • Encoding and Windows locale hardening. Pin UTF-8 on config/dialect opens; mojibake repair no longer destroys clean Portuguese/Vietnamese/Turkish prose; repair-encoding CLI reconfigures stdio; non-ASCII CLI symbols replaced for GBK consoles. (#2098, #2208, #2194, #1104, #1034, #2193)

  • Small correctness fixes. Entity-candidate ReDoS guard; reject pathological chunk_overlap; worktree cwd maps to the project wing; markdown emphasis is not emotion; service entrypoints restore MEMPALACE_PALACE_PATH; systemd Restart=always with idle watchdog docs; valid docker-compose.yml environment key. (#2127, #2056, #2206, #2199, #2192, #2203, #2204, #2188)

Documentation

  • Agent logstream concept page, coordination protocol, shared-brain fleet guide, and RFC 003/004. (#2162)
  • Operator write-routing / single-writer recovery notes. (#2079)
  • Remote-server idle watchdog and read-only semantics. (#2126, #2204)
  • README Docker section leads with the published image and real mount/permission pitfalls. (#2196)
  • MX3 public-shim example and CONTRIBUTING Discussions cleanup. (#597, #555)

Internal

  • Docker publish is gated on a real smoke script (Compose parse, mine, MCP handshake). (#2189)
  • Embedding empty-batch / plain-sequence regression tests; HNSW defaults assertions. (#2191, #2159)

A big thank-you to everyone who contributed to this release. Contributors and first-time contributors are credited below.

What's Changed

  • fix(miner): reject chunk_overlap above half the chunk size to stop chunk_text hang (#2056) by @mvalentsev in #2058
  • perf(miner): O(N) incremental line-number tally in chunk_text (#2054) by @mvalentsev in #2055
  • feat(hooks): apply shared daemon write-routing policy by @fatkobra in #2030
  • perf(mcp): cache the HNSW capacity probe, invalidated by file signature (#1471) by @mvalentsev in #2051
  • fix(storage): enforce local single-writer ownership by @igorls in #2079
  • fix(repair): take the mine-lock before archiving in repair --mode from-sqlite by @joshuafontany in #2109
  • fix(chroma): adopt chromadb's own HNSW write defaults by @joshuafontany in #2107
  • fix: preflight HNSW divergence before every col.count() call site the audit found by @KeilerHirsch in #2093
  • fix(repair): stop dropping drawers with zero embedding_metadata rows by @KeilerHirsch in #2087
  • fix(repair): preserve verified temp copy on failed rebuild swap; fail loud on truncated pagination by @KeilerHirsch in #2086
  • fix: prevent duplicate drawers when re-mining LLM conversations by @rohanprichard in #2050
  • fix(search): return round-trippable drawer IDs by @fatkobra in #2090
  • fix(convo_miner): stop sweeper drawers being purged (#2089) by @igorls in #2125
  • fix(mcp): self-exit on stdin EOF/broken pipe so orphaned stdio sessions release locks by @xcarbo in #2072
  • fix(daemon): defer jobs refused the palace lock, don't fail them (#2014) by @mvalentsev in #2029
  • fix(mcp): refuse config and ack writes in read-only mode (#2103) by @igorls in #2126
  • fix(mcp): refuse config and ack writes in read-only mode (#2101) by @mvalentsev in #2103
  • fix(entity): defuse entity-candidate ReDoS on long ASCII runs (#2065) by @igorls in #2127
  • chore(release): 3.7.0 by @igorls in #2128
  • fix: release 3.7.0 polish — WAL reopen, search errors, hash stamp by @igorls in #2130
  • fix(repair): honor --dry-run for repair --mode from-sqlite (#2133) by @igorls in #2138
  • fix(convos): honor mined state during dry runs by @fatkobra in #2134
  • fix(embedding): remap unsupported EmbeddingGemma token IDs by @fatkobra in #2136
  • fix(hooks): ingest only the active transcript (#2137) by @igorls in #2140
  • fix(hooks): ingest only the active transcript by @fatkobra in #2137
  • feat(searcher): wire i18n stop words into BM25 tokenizer (#973) by @mvalentsev in #977
  • fix(windows): safely repair legacy mojibake with backups by @fatkobra in #2148
  • fix(mcp): refuse vector writes into a diverged HNSW index instead of hanging forever by @Evgen197310 in #2158
  • fix(repair): honor --dry-run in the default mode (#2144) by @mvalentsev in #2145
  • fix(normalize): parse ChatGPT export arrays, not just single conversations (#2160) by @mvalentsev in #2166
  • perf(embedding): group documents by size before sub-batching (#2104) by @mvalentsev in #2149
  • feat: agent logstream — coordination for a multi-machine agent fleet (RFC 003) by @igorls in #2162
  • chore(deps-dev): bump ruff from 0.15.20 to 0.16.1 by @dependabot[bot] in #2181
  • fix(backends): convert embedding vectors to Python floats before upsert by @igorls in #2187
  • fix(docker): drop the null environment: block that invalidated compose by @igorls in #2188
  • fix(mcp): resolve chunked diary entries by their entry_id (#2185) by @igorls in #2186
  • ci(docker): run the image before publishing it by @igorls in #2189
  • docs(readme): lead the Docker section with the published image by @igorls in #2196
  • fix(extractor): ignore markdown emphasis as emotion by @KeilerHirsch in #2199
  • fix(hooks): derive wing from project, not the git-worktree dir by @bethaniamh in #2206
  • fix(deploy): survive the MCP idle watchdog exit in the systemd unit by @rubicon in #2203
  • fix(codex): make marketplace plugin installable by @miky-mfw in #2178
  • fix(service): restore MEMPALACE_PALACE_PATH after the call that stamped it by @joshuafontany in #2192
  • feat(pgvector): implement facet_counts server-side aggregation (#1868 contract) by @messelink in #2038
  • fix(sweep): collapse repeated message UUIDs before upsert by @fatkobra in #2135
  • fix(repair): hold writer lease across rebuild_index by @KeilerHirsch in #2195
  • fix(encoding): pin encoding=utf-8 on dialect.py + config.py text opens by @KeilerHirsch in #2098
  • fix(repair-encoding-cli): reconfigure stdio to UTF-8 like the other entry points by @arnoldwender in #2194
  • fix(encoding): require local evidence before repairing the ambiguous [ÂÃ] window (#2193) by @arnoldwender in #2208
  • fix(encoding): replace non-ASCII symbols in CLI output (#1034) by @arnoldwender in #1104
  • fix(convo-miner): skip Claude Code subagent transcripts by default (#1217) by @mvalentsev in #1330
  • fix(backups): stop a socket in the palace from aborting repair (#2207) by @mvalentsev in #2212
  • feat(embedding): add OpenAI-compatible /v1/embeddings backend by @maximilize in #1671
  • docs: RFC 001 — storage backend plugin specification by @igorls in #743
  • fix: handle unavailable MCP stdout redirection on Windows by @MohabMohie in #2211
  • fix(mcp): refuse writes when the served library is no longer installed (#899) by @mvalentsev in #2081
  • feat(mine): route explicit source adapters through the RFC 002 registry (#2062) by @ggettert in #2068
  • feat(integrations): Hermes memory provider core by @raman325 in #1915
  • fix(palace): reap orphaned per-source-file mine locks by @fxinfo24 in #2200
  • feat(search): add since/before date window to mempalace_search and CLI search (#463) by @mvalentsev in #2000
  • fix(hermes): stop second Chroma client racing the filing worker by @igorls in #2215
  • fix(tests): isolate lock-reap home on Windows via USERPROFILE by @igorls in #2216
  • docs: remove GitHub Discussions reference from contributing guide by @grtninja in #555
  • docs: document MEMPALACE_MCP_IDLE_HOURS by @rubicon in #2204
  • test(backends): cover the _embed_texts fallback and empty-batch guard by @mbeacom in #2191
  • fix: crash in --llm-rerank output for model names without a dash by @ATKabli in #2163
  • docs: add MX3 public shim examples by @grtninja in #597
  • test(chroma): assert the HNSW defaults rather than a threshold arithmetic by @joshuafontany in #2159
  • docs(changelog): fold wave 1–3 notes into 3.7.0 for release by @igorls in #2217
  • chore: merge main into develop before 3.7.0 promotion by @igorls in #2218
  • Release v3.7.0 — promote develop to main by @igorls in #2129

New Contributors

Full Changelog: v3.6.0...v3.7.0

Install: pip install -U mempalace==3.7.0

Docs: Agent logstream · RFC 003

Contributors

rubicon, arnoldwender, and 20 other contributors
Assets 2
Loading