Conversation
- --whole flag for `smriti ingest file`: stores .md as single message (no paragraph splitting); warns without flag - `smriti projects <id>`: rich inspection report — sessions, messages, agents, tags, decisions, recent sessions - `smriti tags`: global or --project-scoped tag usage counts; --available mirrors category tree - `smriti status --project <id>`: scopes all stats (agents, categories) to a single project - 29 new tests in test/recall.test.ts covering all retrieval paths (full-doc, tags, project reports, multi-filter)
QMD submodule is now a clean upstream fork (d58fedf, v2.1.0+) — no Smriti-specific code lives there. Future upstream syncs are conflict-free. - src/memory.ts: moved from qmd/src/memory.ts; imports updated to ../qmd/src/store.js and ../qmd/src/llm.js; uses QMD's Database type - src/ollama.ts: moved from qmd/src/ollama.ts; self-contained, no changes - src/qmd.ts: re-exports now come from ./memory and ./ollama - qmd submodule: bumped to d58fedf (upstream v2.1.0+34 commits of fixes) Upstream picks up: security dep bumps, db-transaction-type fix, embedding overflow hardening, sqlite-vec actionable errors, GGUF magic error fix, Windows home fallback, status device probe opt-in, and more.
Adds `smriti consolidate` and `smriti learnings`, applying Progressive Summarization: cheap Stage-1 segmentation runs broadly over dense sessions into a new smriti_knowledge_units table, and expensive Stage-2 polish (existing segmentSession/generateDocument pipeline) only runs once a unit proves reuse via recall or scored high relevance at extraction time. - src/db.ts: smriti_knowledge_units table + CRUD helpers (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits, incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits) - src/search/recall.ts: track retrieval_count on every recall() path - src/learn/consolidate.ts: segment + promote phases, reusing the existing 3-stage segmentation pipeline from src/team/segment.ts and document.ts - src/index.ts, src/format.ts: CLI wiring for `consolidate` and `learnings` CLI-only, not wired into the daemon — consolidation runs two LLM stages, which the daemon's flush path deliberately avoids (see the enrichOnIngest comment in src/daemon/index.ts). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
Extends the Continuous Knowledge Consolidation Layer with canonical entities and typed subject-predicate-object relationship triples, inspired by RDF's data model (adapted pragmatically: no URIs/SPARQL, just typed (subject_type, subject_id) pairs in SQLite). - src/db.ts: smriti_entities + smriti_relationships tables; new `minEntityReach` promotion criterion in findPromotableUnits (a unit becomes promotable once its entity is independently mentioned by K other units, even at 0 retrievals); seeds a "team" agent (fixes a latent FK bug in syncTeamKnowledge's existing agent fallback) - src/learn/entities.ts: resolveEntity (exact-normalize canonicalization via slugify), insertRelationship/getRelationships (triple store), findRelatedCandidates, findEntity, getUnitsForEntity - src/learn/consolidate.ts: segment phase turns Stage-1 entities into "mentions" edges for free; promote phase adds one bounded LLM call to infer relatesTo/supersedes/contradicts edges against entity-sharing candidates, persisting what ollamaCheckConflicts previously only computed ephemerally - src/team/config.ts, share.ts, sync.ts: entities propagate team/org-wide through the same .smriti/config.json round-trip custom categories already use (exportEntities/mergeEntities mirror exportCustomCategories/mergeCategories); unit-to-unit relationship edges propagate via frontmatter directly, needing no canonicalization since unit ids are already portable UUIDs. Also fixes sync.ts treating "consolidated" pipeline docs as raw conversation transcripts (only "segmented" was previously recognized as single-message). - src/index.ts, src/format.ts: `smriti graph <entity>` command; `--min-entity-reach` flag on `smriti consolidate` Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
When two units sharing an entity are promoted in the same consolidate run, each independently asks the LLM "do I supersede/contradict the other" — found via a live demo that this can produce both directions asserted simultaneously (A supersedes B and B supersedes A), which is incoherent for a directional predicate. Skip inserting the reverse edge if the candidate already asserted it in the other direction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
The prompt asks for "RELATION [i]: predicate" but Ollama reliably gets the index/predicate right while dropping the literal brackets (observed: "RELATION 0: supersedes"). The regex required them exactly, so correct answers were silently discarded — inferRelationships swallows all errors by design (enrichment, not a promotion precondition), so this failure mode had zero visibility: promotion succeeded, the edge just never appeared. Verified against real Ollama output: a hand-fed prompt produced a correct "supersedes" verdict in 16s that the old regex dropped entirely; after the fix, an end-to-end consolidate run against seeded sessions produced a real relatesTo edge, confirmed via `smriti graph`. Also logs when a non-empty response yields zero parsed lines, since that almost always means format drift rather than every candidate genuinely being unrelated.
Replace regex-parsed "RELATION [i]: predicate" free-text prompting with a native tool call (record_relationships) for promote-time relationship inference — the model returns structured JSON directly, so there's nothing to drift from or fail to parse. classifyRelationshipsTextFormat is kept only as the "before" baseline for the eval comparison in test/eval/relation-inference.eval.ts. Also requires QMD_MEMORY_MODEL to be explicitly set (config.ts's new requireOllamaModel()) instead of every Ollama call site silently falling back to a hardcoded default model.
Live comparison of classifyRelationshipsTextFormat (regex-parsed, "before") vs classifyRelationshipsToolCall (native tool call, "after") against 9 hand-labeled scenarios, run against the real configured Ollama model. Manual-only (.eval.ts suffix, excluded from `bun test`) — run via `bun run eval:relations`.
… eval harness Closes the three gaps found when mapping Smriti against a standard 6-layer AI memory architecture (working memory, episodic store, semantic/facts, store/search/update/forget tools, a consolidation job, and an eval harness) — forget and prune existed only as dead code or not at all, and the only eval was a narrow classifier A/B test. - forget: `smriti forget <session-id>` (soft by default, --hard --yes for real deletion) and `forget --all [filters]` for bulk. Re-exports deleteSession/clearAllSessions from src/qmd.ts, adds forgetSession() orchestration (sidecar cleanup, unpromoted knowledge units + their relationship edges, orphaned vector embeddings) in src/db.ts. Canonical (promoted) units/docs are kept unless --purge-shared. - prune: `smriti consolidate --prune` expires stale never-promoted knowledge units (0 retrievals, low relevance, 30+ days old) and soft-archives canonical units superseded by a newer one (tier 'archived', doc kept with a deprecation banner). Dry-run by default; --yes/--apply to mutate. Pure DB logic, no LLM call. - eval harness: test/eval/fixtures/ (multi-session scenarios covering cross-session recall-over-time, project-filter isolation, density- score tie-breaking, and a semantic-only match). Tier 1 (test/recall-quality.test.ts) is BM25-only and runs in `bun test`; Tier 2 (test/eval/recall-quality.eval.ts, `bun run eval:recall`) adds embedding-dependent scenarios and asserts vector search actually fired rather than trusting a silent BM25 fallback.
mockOllamaFetch's `relation` handler still simulated the old
/api/generate free-text response ("RELATION [i]: predicate"), but
inferRelationships was switched to classifyRelationshipsToolCall, which
calls ollamaChat -> /api/chat with a `messages` body (no `prompt` field)
and expects a native record_relationships tool call back. The mismatch
made the mocked fetch throw (reading .includes on the now-undefined
body.prompt), silently swallowed by inferRelationships' best-effort
try/catch, so both tests asserting on the inferred edges failed with
0 edges instead of 1.
Distinguish /api/generate (stage1/stage2, has body.prompt) from
/api/chat (relation inference, no body.prompt) and return a
tool_calls-shaped response for the latter; relation handlers now return
structured {index, predicate} guesses instead of a free-text string.
Resolves conflicts against dev's: - 9fd16f6 feat: recall quality & project inspection (issue #56) — `smriti projects <id>`, `smriti tags`, `--whole` ingest flag, project-scoped `status`. Ordinary content conflicts in db.ts (auto- merged), format.ts, index.ts (kept both sides' additions). - dbb2eeb refactor: move memory.ts + ollama.ts out of QMD submodule. Add/add conflict on src/memory.ts and src/ollama.ts — kept "ours" in both cases after diffing: this branch's versions are a strict superset (getMemoryLlm via the SDK store, recallMemories's fast/ intent/expandQuery/rerank/density-blending pipeline, cleanupOrphanedMemoryVectors, tool-call support, ollamaAsk/ ollamaDrift/ollamaCheckConflicts) — dev's copies were the bare just-extracted-from-submodule versions with none of that. qmd submodule pointer: kept ours (da67604) — confirmed dev's target (d58fedf) is an ancestor of it, so nothing from dev's bump is lost. - 91effef fix(ci): picomatch@4 root dep — already present via package.json's clean auto-merge. bun.lock: kept ours, verified consistent via `bun install` (no changes made). Verified with `bun test --cwd ./test`: 404 pass, 4 fail — same pre-existing failures as before this merge (learn-entities.test.ts x3, team-segmented.test.ts x1; a known full-suite-only global-fetch-mock isolation issue, unrelated to this merge). Manual smoke test: both `smriti forget`/`smriti consolidate --prune` and dev's `smriti projects`/`smriti tags` work correctly in the merged CLI.
recallMemories' density-blending step queries smriti_session_meta directly, but src/memory.ts moved out of the qmd submodule specifically to stay usable as a clean, Smriti-agnostic layer (see the dev merge) — scripts/bench-qmd.ts runs it against a bare QMD store with no Smriti tables at all, which crashed with "no such table: smriti_session_meta". Wrap it in the same try/catch pattern already used for the vector-search fallback a few lines up: no smriti_session_meta means no density signal, so skip the blend and keep the RRF/rerank-only ordering.
feat(learn): consolidation layer + RDF-inspired entities/relationships graph
Contributor
Benchmark Scorecard (ci-small)Bench Scorecard (ci-small)threshold: 20.00%
Summary: WARN (4 metrics) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release Summary
devmaindevtomainChanges Included
Validation
devNotes
PR / Testsand cross-platformPush / Testsshow 4 known pre-existing failures (test/learn-entities.test.ts x3, test/team-segmented.test.ts x1) — a globalfetch-mock isolation issue between test files under full-suite load, confirmed present before this release's changes and consciously deferred (not a regression).Bench / ci-smallpasses after fixing a real bug this release surfaced (recallMemories crashing against a bare QMD store without smriti_session_meta).