v3.12.0 — consolidation that can finish the job

v3.12.0 — consolidation that can finish the job

This release turns smem consolidate from a long-running, restart-prone maintenance command into a durable workflow. Its 20 strategies can make bounded progress, pause at a safe checkpoint, and continue on the next invocation. The largest production run of this work resumed through semantic-link discovery and application and completed all 20 strategies, creating 4,000 semantic synapses without reported strategy failures.

It also substantially reduces the cost of prune and deep semantic scans, strengthens concurrent-run and source-change safeguards, and makes the reranker usable against an OpenAI-compatible LiteLLM proxy. The work spans #255–#281 in this independent project.

What changes for users

• Resume instead of restart. Consolidation records committed work in SurrealDB and reports when it finds progress to resume. A time budget can yield a paused run with a phase and checkpoint rather than discarding completed work. --strategy all continues the unfinished cycle before starting another; --dry-run does not advance its checkpoint. The same engine serves CLI, MCP, and scheduled execution. A genuinely over-budget individual operation still needs attention: checkpointing does not make every database query instantaneous. #256, #263, #266
• A faster, bounded prune. Projected, timestamp-indexed keyset pages replace increasingly expensive offset scans; retention and neuron-scan work are split into resumable substeps. Eligibility rules are unchanged. A measured SurrealDB 3.2 snapshot with roughly 252,000 synapses showed a deep keyset seek reading 5,000 index entries for 2,000 results; this is a query-plan measurement, not a universal speedup claim. #256, #258, #264, #268
• Bounded work across the strategy set. Deduplication, dreaming, schema, interference, lifecycle, maturation, inference, merge, summarize, and other group/record workflows now persist stable units of work and can resume after interruption. Compression retains recoverable backups and fiber work can continue safely. #257, #259, #260, #266
• Tool-event processing is safer to replay. It consumes committed event batches and acknowledges only the processed prefix of the local buffer, preserving concurrent hook appends and avoiding repeat reinforcement after a committed marker. #256
• Semantic-link discovery survives real-world restarts. Durable staged sources, a paged fiber census, source-change fencing and replay-safe application protect the long discovery/similarity/apply pipeline. Recovery covers legacy checkpoints, unchanged changefeed metadata, repeated source drift, and schema migration to v12 without silently abandoning compatible progress. Deep neuron/synapse scans use bounded ID windows, and apply checkpoints cover small candidate batches instead of repeatedly rewriting a large manifest for each edge. #261, #262, #265, #269–#281
• More robust model routing. The reranker splits requests that overflow a context window without losing candidate scores. OpenAI-compatible embeddings use their configured credential independently of an ambient OPENAI_API_KEY, so proxy routing is predictable. #255, #279

Persistence and compatibility

The SurrealDB schema advances through durable consolidation checkpoints and source-change safeguards to v12. Back up the database before upgrading. Run the application and local clients at a compatible version; an older client attempting to downgrade a v12 schema fails closed. Do not reset a pending checkpoint or bulk-import data into a database with pending semantic discovery: SurrealDB import does not emit the source changefeed events on which the fence relies. An incomplete run has priority over a new cycle, and records written after its reference time belong to a later cycle. Operators should watch the reported phase, counters, lease and source-fence errors rather than treating a pause as a failure.

There is no intended change to prune deletion eligibility. External LLM work may be invoked again after interruption, while the database application path is guarded against duplicate effects; this is not a promise of exactly-once billing or inference calls.

Validation and deployment evidence

• A clean local non-stress gate for the final semantic-apply change passed 7,867 tests, with 50 skipped, 32 deselected and 1 expected failure. The v3.12.0 preparation PR #282 then passed all nine final-head CI jobs, including Python 3.11/3.12, real SurrealDB integration, docs freshness and package build. The eventual tag has its own release pipeline, which this draft does not claim has run.
• A production consolidate --strategy all resumed its existing run and finished 20/20 strategies with semantic_synapses_created=4000, failures=0. The last strategy finished in about 320 seconds; ordinary recall afterward returned a nonempty result.
• A read-only post-run graph check scanned 61,356 neurons and 251,002 synapses. It identified two older edges with missing endpoints; neither was among the newly created semantic edges. This release does not claim to repair those historical edges.
• Production ran the exact #281 source image under Helm revision 39; app, SurrealDB 3.2.4 and WebUI were Ready with zero app restarts at validation. /health and /ui returned HTTP 200. A pre-rollout SurrealQL backup was completed and verified by checksum.
• The deployed LiteLLM configuration was checked with a 1,024-dimensional bge-m3 embedding, finite scores from bge-reranker-v2-m3, and the glm-5.3-flash LLM route. These are environment checks, not a claim that every user deploy uses those models.

Upgrade notes

1. Back up SurrealDB and check available disk space before replacing images. Keep the previous image for rollback until the new deployment is stable.
2. Deploy compatible server and client packages, let schema migration finish, and verify /health, /ui, recall and any configured embedding/reranker paths.
3. If a run is paused, inspect its checkpoint and lease, then retry with the compatible engine and the same parameters. Expect an explicit resume message and monitor the phase/counters; do not reset the state or start a conflicting run.