Skip to content

v4.2.0

Choose a tag to compare

@Korrnals Korrnals released this 13 Sep 16:04
· 529 commits to main since this release
2316cd8

Added

  • Round-3 embedder weights + fingerprint-aware re-embed migration (ADR-0021) (src/mnemos/models/mnema-embed-v1/, src/mnemos/manager.py, src/mnemos/embeddings/__init__.py, src/mnemos/storage/sqlite_store.py, src/mnemos/cli/doctor.py, benchmarks/baselines/{s1.json,BASELINE.md}, tests/{test_embeddings_mnema,test_pipeline_state,test_doctor_vector_vintage}.py, docs EN+RU) — the bundled mnema-embed-v1 artifact is swapped to the round-3 export (epoch-3 checkpoint, teacher Qwen/Qwen3-Embedding-0.6B, weights_sha256 3b752e06…, int8 29.6 MB, MRL dims 64/128/256/384, opset 15, dataset fingerprint b27f8cf0…; corpus ~100k text→teacher pairs incl. 8086 real store entries, RU 41.9%; manifest schema-merged — the training-side export uses embed_dim and carries no name, the bundle contract keeps name/dimensions and gains the full provenance tail). P0 migration gap closed: _vector_metadata previously stamped only content_hash, so after a weights swap the unchanged content made old-geometry vectors look fresh forever — mixed embedding spaces with silent vector-search degradation. Every upsert now also stamps model_fingerprint; heal_stale_embeddings re-embeds any vector whose fingerprint is missing (pre-swap vintage) or differs from the current embedder, through the single write point upsert_embedding, keeping the batch/limit budget — and the sweeper now PAGES through the whole refined set (previously it re-read the same created_at DESC head window every cycle, so a uniform-vintage migration could never drain past the first limit rows). Provider identity contract: EmbeddingProvider.fingerprint (concrete default = class name) with nano:sha256:<weights> for the bundled model, ollama:<model> / onnxhub:<model>@<revision> / st:<model> coarse switch-detection identities for the external providers (documented limitation: no weights hash, so silent upstream weight refreshes under the same name are not detected), plus a doctor-side twin config_fingerprint(cfg) that computes the identity WITHOUT loading a provider session and a test pinning twin==instance. mnemos doctor's Vector store check now counts vintage-mismatched rows (WARN + the mnemos reindex / background-heal recommendation; diagnostics only). S1 re-baselined in the same PR per ADR-0021: deterministic corridors unchanged (recall@5 0.8637), S1m honestly re-recorded on the round-3 weights — recall@5 0.87452 → 0.863002 (−0.0115, inside CI95 ±0.0439), with the eval-jig student recall@5 gain 0.428 → 0.4485 (191 judged queries, teacher const 0.6071) as the adoption evidence. Docs: EN/RU architecture overview carries the round-3 lineage + vintage-tracking note; the migrate runbook (EN/RU) explains the gradual background re-embed on upgrade. Tests: +12 (fingerprint pin/twin/legacy-degradation/coarse formats, metadata stamp, heal on mismatch/missing/no-op, limit-drain, quarantine skip, doctor vintage verdicts incl. corrupt metadata + JSON surface).
  • mnemos doctor: pending-refinement queue diagnostics (ADR-0019 Phase D) (src/mnemos/cli/doctor.py, tests/test_doctor_pending_refine.py, README.md) — new "Pending refine" health check: counts rows with pipeline_state='pending' (the B2a refine intake, which the B1 backfill filled with bypass-era heritage) and WARNs with the mnemos processor start recommendation when the queue is non-empty — CLI-only deployments have no background daemon, so the queue never drains on its own. Diagnostics only: the doctor deliberately does not run the processor (it is a server-side service). A missing pipeline_state column (pre-ADR-0019 schema) or a missing DB is a PASS with an explanatory detail. The README's Auto-pipeline feature row now documents the pending state and the processor command. Tests: 7 new cases (missing DB, zero-pending, 1/3 pending WARN + recommendation, pre-ADR-0019 schema, JSON output wiring).

Changed

  • Hermes bypass removed — publish_on_write neutralized (ADR-0019 Phase D) (src/mnemos/adapters/hermes.py, tests/test_hermes_adapter.py) — the adapter no longer publishes on its own: _maybe_publish (the publish_on_write → publish(skip_quality_check=True) path) is deleted and every write verb (add_memory / sync_turn / mirror_memory_write / session_end / save_checkpoint) now goes out WITHOUT an explicit status, so the server's mnemos.visibility policy owns the initial visibility through the fail-closed ingest gate (ADR-0019 §2 B2b): immediate (default) publishes clean content at once with pipeline_state=pending (refused content is stored RAW, zero-loss); curated holds the row RAW + pending until the refine cycle gates the refined projection. publish_on_write=False therefore no longer means "raw forever": it is a no-op compatibility knob (still accepted because the Hermes shim passes it; an INFO line says so when flipped) — under immediate every entry is visible regardless of the knob, under curated the row is RAW + pipeline_state=pending and the refine daemon completes its visibility. Per-adapter visibility flags were explicitly rejected by the ADR-0019 committee (server-level default); the first-class publish surface and the REST POST /publish/{id}?skip_quality_check=true endpoint are unchanged — only the adapter's automatic use of the bypass is gone. Tests: the old bypass tests are replaced by the new-semantics pins (immediate + knob off → visible; curated + knob off/on → RAW+pending, visible after refine_pending; injection write → refused at path=ingest, RAW + pipeline_state=None, audited) plus a removal-contract guard (_maybe_publish absent, no skip_quality_check/manager.publish in the adapter source).

Fixed

  • Security: fresh pip-audit advisories closed, make verify gate green again (#267) (pyproject.toml, uv.lock) — newly published PYSEC advisories against four pinned packages made every make security / CI pip-audit run red. Floors raised to the minimal fixed versions: aiohttp>=3.14.1,<4.0 → >=3.14.3,<4.0 (PYSEC-2026-3546/3547 fixed in 3.14.2, PYSEC-2026-3545 in 3.14.3 — one floor covers all three) and cryptography>=48.0.1 → >=50.0.0 (PYSEC-2026-3552; the major bump is verified compatible — the codebase only uses stable primitives: Fernet, AESGCM, PBKDF2HMAC, x509/rsa/hashes). The other two advisories hit venv tooling no direct pin controlled inside the audited dev environment: pip>=26.2 (PYSEC-2026-3721; enters transitively via pip-audit → pip-api) and setuptools>=83.0.0 (PYSEC-2026-3447; via grpcio-tools' runtime dependency) are now floored in the [dev] extra — the exact profile make security and CI audit. uv.lock freshly regenerated on a dedicated branch (no parallel-track drift carried in): the four fixes, the previously missing types-PyYAML dev pin added, and the setuptools-linked training-extra subtree re-resolved to latest (torch 2.12.1→2.14.0, triton, cuda-toolkit, nvidia-cudnn/nccl — lock-only collateral; training stays outside the runtime per ADR-0021 and is installed by no gate). No new --ignore-vuln crutches: the pre-existing documented CVE-2026-45829 ignore (chromadb legacy) is untouched.
  • Federation/import rows now pass the Phase A danger gate (#166) (src/mnemos/cli/import_.py, src/mnemos/cli/sync.py, tests/test_federation_import_gate.py) — run_sync_import (compact federation payload) and _import_json (JSON export merge/restore, including --overwrite) wrote peer-status rows directly through sqlite.save(), so a PUBLISHED record arriving from peering never met the ADR-0019 publication gate. Both paths now route every imported row through the new gate_imported_memory helper — the SAME single gate point the server uses (MemoryManager._publish_gate_detection, i.e. danger_detectors.detect over the served projection and the title, fail-closed, audited as publish gate: … path=federation-import): a positive danger signal or scanner error stores the row RAW + pipeline_state=NULL (zero-loss, invisible, no embed — and outside the refine intake so danger-class content cannot auto-refine back into visibility); a clean row keeps the peer's status and a NULL pipeline_state is stamped pending so the local refine queue picks it up (non-NULL peer states — refined, quarantined — are never clobbered; a peer's quarantine verdict survives import). An --overwrite refusal also evicts the formerly-published row's stale vector embed (N1 demotion hygiene, mirroring MemoryManager.update), and each refusal appends an operator-facing warning to the import result. Tests: 9 new cases (federated PUBLISHED with secret/injection-in-title/scanner-error → RAW+NULL+invisible+un-embedded+audited; clean → PUBLISHED+pending+embedded; JSON-import twin; overwrite demotion with embed eviction; quarantine preservation; clean RAW row joins the queue) — the mutation "remove the gate call" fails them first.