feat(learn): consolidation layer + RDF-inspired entities/relationships graph

[ashu17706]

Summary

Started as the Continuous Knowledge Consolidation Layer and grew a second piece: an RDF-inspired entities/relationships graph on top of it, after discussing how RDF's triple model (subject-predicate-object, canonical resource identity, typed vocabularies) could strengthen the self-learning loop.

Part 1 — Continuous Knowledge Consolidation (Progressive Summarization): cheap Stage-1 extraction (segmentSession) runs broadly over dense sessions into a new smriti_knowledge_units table; expensive Stage-2 polish (generateDocument) only runs once a unit proves it's reused (recalled repeatedly) or scored high relevance at extraction time.

• src/db.ts — smriti_knowledge_units table + CRUD helpers (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits, incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits)
• src/search/recall.ts — tracks retrieval_count on every recall() exit path
• src/learn/consolidate.ts — segment phase + promote phase
• src/index.ts, src/format.ts — smriti consolidate and smriti learnings

Part 2 — Entities & relationships graph (RDF-inspired, not literal RDF — no URIs/SPARQL, just typed (subject_type, subject_id) pairs in SQLite): today's entities: string[] field is write-only — never queried or matched, so "JWT" in one session and "JSON Web Token" in another look unrelated. This closes that gap and also persists what ollamaCheckConflicts previously only computed ephemerally.

• src/db.ts — smriti_entities (canonical registry) + smriti_relationships (triple store) tables; new minEntityReach promotion criterion — a unit becomes promotable once its entity is independently mentioned by K other units, even at 0 retrievals
• src/learn/entities.ts — resolveEntity (exact-normalize canonicalization via slugify), insertRelationship/getRelationships (triple store), findRelatedCandidates, findEntity, getUnitsForEntity
• src/learn/consolidate.ts — segment phase turns Stage-1 entities into mentions edges for free (no extra LLM calls); promote phase adds one bounded LLM call (only when entity-sharing candidates exist) to infer relatesTo/supersedes/contradicts edges
• Team/org propagation — an entity's canonical id is only meaningful if every teammate's machine agrees on it. src/team/config.ts/share.ts/sync.ts route entities through the exact same .smriti/config.json git round-trip custom categories already use (exportEntities/mergeEntities mirror exportCustomCategories/mergeCategories). Unit-to-unit relationship edges propagate via frontmatter directly — no canonicalization needed there, since unit ids are already portable UUIDs once shared.
• src/index.ts, src/format.ts — smriti graph <entity> command, --min-entity-reach flag on consolidate

Two pre-existing bugs found and fixed while building the propagation tests (both predate this PR, exposed because nothing previously tested a real share→sync round trip on disk):

• syncTeamKnowledge's agent fallback used "team" as an agent id that was never seeded into smriti_agents, so importing any shared file without an explicit agent field (true for the entire pre-existing segmented pipeline) hit an FK violation. Fixed by seeding a team agent.
• syncTeamKnowledge only recognized pipeline === "segmented" as a single-message document; pipeline === "consolidated" docs were misparsed as raw transcripts and silently skipped on import.

Both remain CLI-only, never wired into the daemon — same reasoning as enrich: LLM work per-flush is unsafe (see src/daemon/index.ts's enrichOnIngest comment).

Test plan

• test/learn-consolidate.test.ts — segment-phase dedup, promotion threshold, graceful degradation, retrieval tracking
• test/learn-entities.test.ts — entity canonicalization (exact-normalize merge, no false-merge on real synonyms), relationship triple insert/dedup/pattern-lookup, mentions edges from segment phase (zero extra LLM calls), minEntityReach promotion, LLM-inferred relationship edges (+ best-effort failure handling), findRelatedCandidates
• test/learn-entities-sync.test.ts — two independent in-memory DBs standing in for two teammates' machines, connected only through a shared tmp .smriti/ directory: confirms an entity converges to the same canonical id via share→sync (not an independently re-slugified duplicate), and that supersedes edges between shared units survive the round-trip
• bun test --cwd ./test — 391 pass, 0 fail across 33 files, no regressions
• Manual smoke test: smriti consolidate, smriti learnings, smriti graph <entity> run cleanly against a fresh DB

Note: the tool-call switch (classifyRelationshipsToolCall now wired into inferRelationships, mirrored by the "Also in this PR" commits below) broke the mock format in 2 tests under test/learn-entities.test.ts (promote phase persists LLM-inferred relatesTo/supersedes/contradicts edges, promote phase never asserts a directional predicate in both directions for the same pair — they still mock the old free-text RELATION [i]: predicate response). Confirmed pre-existing as of this push, not caused by the memory-stack work below. Worth a follow-up fix before merging.

Also in this PR: close the working-memory-stack gaps

Mapped Smriti against a standard 6-layer AI memory architecture (working
memory, episodic store, semantic/facts, store/search/update/forget
tools, a consolidation job with summarize/promote/prune, and an
eval harness) and closed the three gaps found:

• smriti forget <session-id> — soft delete by default (active=0,
  already understood by list --all); --hard --yes for real deletion
  (messages, all sidecar tables, unpromoted knowledge units + their
  relationship edges, orphaned vector embeddings). forget --all [--project/--category/--agent] for bulk. Canonical (promoted) units
  and their .smriti/knowledge/ docs are kept unless --purge-shared.
  deleteSession/clearAllSessions existed in src/memory.ts but were
  dead code — never re-exported from src/qmd.ts, no CLI command.

• smriti consolidate --prune — expires stale never-promoted units
  (0 retrievals, relevance below the promotion bar, 30+ days old — hard
  delete, safe since nothing external references them) and soft-archives
  canonical units superseded by a newer one (tier='archived', doc file
  kept in place with a deprecation banner prepended, not deleted/moved).
  Dry-run by default, prints a candidate table; --yes/--apply to
  mutate. Pure DB logic (age/retrieval_count/supersedes-edges already
  in SQLite) — no LLM call.

• Recall-quality eval harness — the only prior eval
  (test/eval/relation-inference.eval.ts, added earlier in this PR) was
  a narrow single-function A/B test, not a "what should the system
  remember" check. test/eval/fixtures/ now has multi-session scenarios
  (cross-session recall-over-time + a same-vocabulary distractor,
  project-filter isolation, density-score tie-breaking on a BM25 tie, and
  a semantic-only paraphrase match). Tier 1
  (test/recall-quality.test.ts) is BM25-only and deterministic, runs in
  bun test/CI. Tier 2 (test/eval/recall-quality.eval.ts, bun run eval:recall) adds the embedding-dependent scenario and explicitly
  asserts vector search fired rather than trusting a silent BM25
  fallback.

Files: src/qmd.ts, src/memory.ts, src/db.ts, src/index.ts,
src/format.ts, src/learn/consolidate.ts, src/learn/entities.ts,
package.json, test/forget.test.ts, test/recall-quality.test.ts,
test/eval/fixtures/*, test/eval/recall-quality.eval.ts.

Test plan (forget/prune/eval)

• test/forget.test.ts — soft delete visibility, hard delete
  cascade (messages/sidecars/relationship edges), canonical units kept
  unless --purge-shared, bulk forget
• test/learn-consolidate.test.ts (prune additions) — dry-run
  reporting, apply deletes stale units + edges, high-relevance units
  never pruned, superseded units archived with banner appended, archived
  units' relationship edges left untouched
• test/recall-quality.test.ts — all CI-safe scenarios pass
• bun test --cwd ./test — 404 pass, 4 fail (all 4 pre-existing, see
  note above — not caused by this work; confirmed by isolating these
  files with the forget/prune/eval commits reverted)
• Manual CLI smoke test against a scratch DB: smriti forget <id> (soft, then --hard --yes), smriti consolidate --prune (dry-run,
  then --yes) — all behaved as expected end-to-end

[github-actions]

Benchmark Scorecard (ci-small)

Bench Scorecard (ci-small)

threshold: 20.00%

metric	baseline	current (median)	delta	status
ingest_throughput_msgs_per_sec	1735.800	419.350	-75.84%	WARN
ingest_p95_ms_per_session	6.960	27.218	+291.06%	WARN
fts_p95_ms	0.410	0.815	+98.78%	WARN
recall_p95_ms	0.436	1.093	+150.69%	WARN

Summary: WARN (4 metrics)