feat: v0.16.0 — Memory & Persistence

CHANGELOG.md

@@ -5,6 +5,62 @@ All notable changes to selectools will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.16.0] - 2026-03-13
+
+### Added — Memory & Persistence
+
+#### Persistent Conversation Sessions (new `sessions.py` module)
+
+- **`SessionStore` protocol**: Pluggable backends for saving/loading `ConversationMemory` state. Three methods: `save()`, `load()`, `list()`, `delete()`.
+- **`JsonFileSessionStore`**: File-based backend, one JSON file per session.
+- **`SQLiteSessionStore`**: Single-database backend with JSON column.
+- **`RedisSessionStore`**: Distributed backend with server-side TTL.
+- **Agent integration**: `AgentConfig(session_store=store, session_id="user-123")` — auto-loads on init, auto-saves after each `run()` / `arun()`.
+- **TTL-based expiry**: All backends support configurable `default_ttl`.
+
+#### Summarize-on-Trim (enhanced `memory.py`)
+
+- **LLM-generated summaries**: When `ConversationMemory` trims messages, it generates a 2-3 sentence summary of dropped messages using a configurable provider/model.
+- **Context preservation**: Summary injected as system-level context message.
+- **Configuration**: `AgentConfig(summarize_on_trim=True, summarize_provider=provider)`.
+
+#### Entity Memory (new `entity_memory.py` module)
+
+- **`EntityMemory`**: LLM-based entity extraction after each turn.
+- **Entity types**: person, organization, project, location, date, custom.
+- **Deduplication**: Case-insensitive matching with attribute merging.
+- **LRU pruning**: Configurable `max_entities` limit.
+- **System prompt injection**: `[Known Entities]` context for subsequent turns.
+
+#### Knowledge Graph Memory (new `knowledge_graph.py` module)
+
+- **`KnowledgeGraphMemory`**: Extracts (subject, relation, object) triples from conversation.
+- **`TripleStore` protocol**: `InMemoryTripleStore` and `SQLiteTripleStore` backends.
+- **Keyword-based query**: `query_relevant(query)` for relevant triple retrieval.
+- **System prompt injection**: `[Known Relationships]` context.
+
+#### Cross-Session Knowledge Memory (new `knowledge.py` module)
+
+- **`KnowledgeMemory`**: Daily log files + persistent `MEMORY.md` for long-term facts.
+- **Auto-registered `remember` tool** for explicit knowledge storage.
+- **System prompt injection**: `[Long-term Memory]` + `[Recent Memory]` context.
+
+### Changed
+
+- **`AgentConfig`**: New fields: `session_store`, `session_id`, `summarize_on_trim`, `summarize_provider`, `summarize_model`, `entity_memory`, `knowledge_graph`, `knowledge_memory`.
+- **`AgentObserver`**: 4 new events (total: 19): `on_session_load`, `on_session_save`, `on_memory_summarize`, `on_entity_extraction`.
+- **`StepType`**: 5 new trace step types: `session_load`, `session_save`, `memory_summarize`, `entity_extraction`, `kg_extraction`.
+- **`ConversationMemory`**: New `summarize_on_trim` parameter and `summary` property.
+
+### Documentation
+
+- **4 new module docs**: `SESSIONS.md`, `ENTITY_MEMORY.md`, `KNOWLEDGE_GRAPH.md`, `KNOWLEDGE.md`
+- **Updated**: `ARCHITECTURE.md`, `QUICKSTART.md` (Steps 12-15), `docs/README.md`, `docs/index.md`
+- **5 new examples**: `33_persistent_sessions.py` through `37_knowledge_memory.py`
+- **Updated notebook**: sections 14-16 for sessions, entity memory, knowledge
+
+---
+
 ## [0.15.0] - 2026-03-12
 
 ### Added — Enterprise Reliability

README.md

@@ -7,18 +7,26 @@
 
 **Production-ready AI agents with tool calling, RAG, and hybrid search.** Connect LLMs to your Python functions, embed and search your documents with vector + keyword fusion, stream responses in real time, and dynamically manage tools at runtime. Works with OpenAI, Anthropic, Gemini, and Ollama. Tracks costs automatically.
 
-## What's New in v0.15.0
+## What's New in v0.16.0
 
-**Enterprise Reliability** — Four new security and compliance features:
+**Memory & Persistence** — Five new features for durable conversation state:
 
-- **Guardrails Engine** — Pluggable input/output validation pipeline with 5 built-in guardrails (`TopicGuardrail`, `PIIGuardrail`, `ToxicityGuardrail`, `FormatGuardrail`, `LengthGuardrail`). Supports block, rewrite, and warn actions.
-- **Audit Logging** — JSONL append-only audit trail via `AuditLogger` with 4 privacy levels (full, keys-only, hashed, none) and daily file rotation. Implements `AgentObserver` for zero-config integration.
-- **Tool Output Screening** — Prompt injection detection with 15 built-in patterns. Per-tool opt-in via `@tool(screen_output=True)` or global via `AgentConfig(screen_tool_output=True)`.
-- **Coherence Checking** — LLM-based intent verification that catches tool calls diverging from the user's original request. Enable with `AgentConfig(coherence_check=True)`.
-- **83 new tests** (total: 1183)
+- **Persistent Sessions** — `SessionStore` protocol with 3 backends (JSON file, SQLite, Redis). Auto-save/load via `AgentConfig(session_store=store, session_id="user-123")`.
+- **Summarize-on-Trim** — LLM-generated summaries of trimmed messages, injected as system context. No more silent context loss.
+- **Entity Memory** — Auto-extract named entities (person, org, project) across turns with LRU-pruned registry and system prompt injection.
+- **Knowledge Graph** — Relationship triple extraction with in-memory and SQLite storage. Query-relevant triples auto-injected into prompts.
+- **Cross-Session Knowledge** — Daily logs + persistent facts with auto-registered `remember` tool. Give your agent durable memory across conversations.
+- **182 new tests** (total: 1365)
 
 > Full changelog: [CHANGELOG.md](https://github.com/johnnichev/selectools/blob/main/CHANGELOG.md)
 
+<details>
+<summary><strong>v0.15.x highlights</strong></summary>
+
+- **v0.15.0**: Enterprise Reliability — Guardrails engine (5 built-in), audit logging (4 privacy levels), tool output screening (15 patterns), coherence checking
+
+</details>
+
 <details>
 <summary><strong>v0.14.x highlights</strong></summary>
 
@@ -48,7 +56,11 @@
 | **Audit Logging** | JSONL audit trail with privacy controls (redact, hash, omit) and daily rotation. |
 | **Tool Output Screening** | Prompt injection detection with 15 built-in patterns. Per-tool or global. |
 | **Coherence Checking** | LLM-based verification that tool calls match user intent — catches injection-driven tool misuse. |
-| **AgentObserver Protocol** | 15-event lifecycle observer with `run_id`/`call_id` correlation. Built-in `LoggingObserver` for structured JSON logs. |
+| **Persistent Sessions** | `SessionStore` with JSON file, SQLite, and Redis backends. Auto-save/load with TTL expiry. |
+| **Entity Memory** | LLM-based entity extraction with deduplication, LRU pruning, and system prompt injection. |
+| **Knowledge Graph** | Relationship triple extraction with in-memory and SQLite storage and keyword-based querying. |
+| **Cross-Session Knowledge** | Daily logs + persistent facts with auto-registered `remember` tool. |
+| **AgentObserver Protocol** | 19-event lifecycle observer with `run_id`/`call_id` correlation. Built-in `LoggingObserver` for structured JSON logs. |
 | **Production Hardened** | Retries with backoff, per-tool timeouts, iteration caps, cost warnings, observability hooks + observers. |
 | **Library-First** | Not a framework. No magic globals, no hidden state. Use as much or as little as you need. |
 
@@ -68,9 +80,13 @@
 - **Response Caching**: InMemoryCache and RedisCache with stats tracking
 - **146 Model Registry**: Type-safe constants with pricing and metadata
 - **Pre-built Toolbox**: 24 tools for files, data, text, datetime, web
-- **32 Examples**: RAG, hybrid search, streaming, structured output, traces, batch, policy, observer, guardrails, audit, and more
-- **AgentObserver Protocol**: 15 lifecycle events with `run_id` correlation, `LoggingObserver`, OTel export
-- **1183 Tests**: Unit, integration, regression, and E2E with real API calls
+- **Persistent Sessions**: 3 backends (JSON file, SQLite, Redis) with TTL
+- **Entity Memory**: LLM-based named entity extraction and tracking
+- **Knowledge Graph**: Triple extraction with in-memory and SQLite storage
+- **Cross-Session Knowledge**: Daily logs + persistent memory with `remember` tool
+- **37 Examples**: RAG, hybrid search, streaming, structured output, traces, batch, policy, observer, guardrails, audit, sessions, entity memory, knowledge graph, and more
+- **AgentObserver Protocol**: 19 lifecycle events with `run_id` correlation, `LoggingObserver`, OTel export
+- **1365 Tests**: Unit, integration, regression, and E2E with real API calls
 
 ## Install
 

ROADMAP.md

@@ -162,11 +162,11 @@ agent = Agent(
 
 | Feature                              | Priority  | Impact | Effort |
 | ------------------------------------ | --------- | ------ | ------ |
-| **Persistent Conversation Sessions** | 🟡 High   | High   | Medium |
-| **Summarize-on-Trim**                | 🟡 Medium | Medium | Small  |
-| **Cross-Session Knowledge Memory**   | 🟡 Medium | Medium | Medium |
-| **Entity Memory**                    | 🟡 Medium | High   | Medium |
-| **Knowledge Graph Memory**           | 🟡 Low    | High   | Large  |
+| **Persistent Conversation Sessions** | ✅ Done   | High   | Medium |
+| **Summarize-on-Trim**                | ✅ Done   | Medium | Small  |
+| **Cross-Session Knowledge Memory**   | ✅ Done   | Medium | Medium |
+| **Entity Memory**                    | ✅ Done   | High   | Medium |
+| **Knowledge Graph Memory**           | ✅ Done   | High   | Large  |
 
 ---
 
@@ -187,7 +187,7 @@ v0.14.1  ✅ Streaming & Provider Fixes (Complete)
 v0.15.0  ✅ Enterprise Reliability (Complete)
          Guardrails engine → Audit logging → Tool output screening → Coherence checking
 
-v0.16.0  🟡 Memory & Persistence
+v0.16.0  ✅ Memory & Persistence (Complete)
          Sessions → Summarize-on-trim → Knowledge memory → Entity memory → KG memory
 
 v0.17.0  🟡 Multi-Agent Orchestration
@@ -763,6 +763,18 @@ Focus: Niche integrations, community sharing, and developer experience polish.
 
 ## Release History
 
+### v0.16.0 - Memory & Persistence
+
+- ✅ **Persistent Sessions**: `SessionStore` protocol with 3 backends (JSON file, SQLite, Redis), TTL expiry, auto-save/load via `AgentConfig`
+- ✅ **Summarize-on-Trim**: LLM-generated summaries of trimmed messages, injected as system context; configurable provider/model
+- ✅ **Entity Memory**: LLM-based entity extraction (person, org, project, location, date, custom), LRU-pruned registry, system prompt injection
+- ✅ **Knowledge Graph Memory**: Triple extraction (subject, relation, object), in-memory and SQLite storage, keyword-based querying
+- ✅ **Cross-Session Knowledge**: Daily log files + persistent `MEMORY.md`, auto-registered `remember` tool, system prompt injection
+- ✅ **Memory Tools**: Built-in `remember` tool auto-registered when `knowledge_memory` is configured
+- ✅ **4 new observer events**: `on_session_load`, `on_session_save`, `on_memory_summarize`, `on_entity_extraction` (total: 19)
+- ✅ **5 new trace step types**: `session_load`, `session_save`, `memory_summarize`, `entity_extraction`, `kg_extraction`
+- ✅ **182 new tests** (total: 1365)
+
 ### v0.15.0 - Enterprise Reliability
 
 - ✅ **Guardrails Engine**: `GuardrailsPipeline` with 5 built-in guardrails (Topic, PII, Toxicity, Format, Length) and block/rewrite/warn actions

docs/ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # Selectools Architecture
 
-**Version:** 0.15.0
+**Version:** 0.16.0
 **Last Updated:** March 2026
 
 ## Table of Contents
@@ -41,6 +41,11 @@ Selectools is a production-ready Python framework for building AI agents with to
 - **Audit Logging**: JSONL audit trail with privacy controls (full/keys-only/hashed/none)
 - **Tool Output Screening**: Pattern-based prompt injection detection (15 built-in patterns)
 - **Coherence Checking**: LLM-based intent verification for tool calls
+- **Persistent Sessions**: SessionStore protocol with JSON file, SQLite, and Redis backends
+- **Summarize-on-Trim**: LLM-generated summaries of trimmed messages
+- **Entity Memory**: Auto-extract named entities with LRU-pruned registry
+- **Knowledge Graph**: Relationship triple extraction and keyword-based querying
+- **Cross-Session Knowledge**: Daily logs + persistent facts with auto-registered `remember` tool
 
 ---
 
@@ -70,6 +75,8 @@ Selectools is a production-ready Python framework for building AI agents with to
 │  │  • Tool output screening (security.py)                           │  │
 │  │  • Coherence checking (coherence.py)                             │  │
 │  │  • Audit logging (audit.py)                                      │  │
+│  │  • Session persistence (sessions.py)                             │  │
+│  │  • Memory context injection (entity, KG, knowledge)              │  │
 │  └─────────┬────────────────────────┬──────────────────┬────────────┘  │
 │            │                        │                  │               │
 │            ▼                        ▼                  ▼               │
@@ -238,8 +245,45 @@ Each implements the `Provider` protocol with `complete()`, `stream()`, `acomplet
 - Sliding window with configurable limits (message count, token count)
 - Automatic pruning when limits exceeded
 - Tool-pair-aware trimming: never orphans a tool_use without its tool_result
+- Summarize-on-trim: LLM-generated summaries of trimmed messages
 - Integrates seamlessly with Agent
 
+### 6a. Persistent Sessions (`sessions.py`)
+
+**SessionStore** protocol with three backends for saving/loading `ConversationMemory`:
+
+- `JsonFileSessionStore` — file-based, one JSON file per session
+- `SQLiteSessionStore` — single database, JSON column
+- `RedisSessionStore` — distributed, server-side TTL
+- Auto-save after each run, auto-load on init via `AgentConfig`
+
+### 6b. Entity Memory (`entity_memory.py`)
+
+**EntityMemory** auto-extracts named entities from conversation using an LLM:
+
+- Tracks name, type, attributes, mention count, timestamps
+- Deduplication by name (case-insensitive) with attribute merging
+- LRU pruning when over `max_entities`
+- Injects `[Known Entities]` context into system prompt
+
+### 6c. Knowledge Graph Memory (`knowledge_graph.py`)
+
+**KnowledgeGraphMemory** extracts relationship triples from conversation:
+
+- `Triple` dataclass: subject, relation, object, confidence
+- `TripleStore` protocol with in-memory and SQLite backends
+- Keyword-based query for relevant triples
+- Injects `[Known Relationships]` context into system prompt
+
+### 6d. Cross-Session Knowledge (`knowledge.py`)
+
+**KnowledgeMemory** provides durable cross-session memory:
+
+- Daily log files (`YYYY-MM-DD.log`) for recent entries
+- Persistent `MEMORY.md` for long-term facts
+- Auto-registered `remember` tool for explicit knowledge storage
+- Injects `[Long-term Memory]` + `[Recent Memory]` into system prompt
+
 ### 7. RAG System (`rag/`)
 
 The **RAG module** provides end-to-end document search:
@@ -284,7 +328,7 @@ Enforces typed responses from LLMs:
 
 Structured timeline of every agent execution:
 
-- `TraceStep` types: `llm_call`, `tool_selection`, `tool_execution`, `cache_hit`, `error`, `structured_retry`
+- `TraceStep` types: `llm_call`, `tool_selection`, `tool_execution`, `cache_hit`, `error`, `structured_retry`, `session_load`, `session_save`, `memory_summarize`, `entity_extraction`, `kg_extraction`
 - Captures timestamps, durations, input/output summaries, token usage
 - `AgentTrace` container with `.to_dict()`, `.to_json()`, `.timeline()`, `.filter()`
 - Always populated on `result.trace` — zero cost when not accessed
@@ -311,7 +355,7 @@ Resilient provider orchestration:
 
 Class-based lifecycle observability:
 
-- 15 event methods with `run_id` correlation for concurrent requests
+- 19 event methods with `run_id` correlation for concurrent requests
 - `call_id` for matching parallel tool start/end pairs
 - Built-in `LoggingObserver` for structured JSON log output
 - OpenTelemetry span export via `AgentTrace.to_otel_spans()`
@@ -453,7 +497,11 @@ Single source of truth for 146 models:
          │    ├─→ usage.py (AgentUsage, UsageStats)
          │    ├─→ analytics.py (AgentAnalytics)
          │    ├─→ observer.py (AgentObserver, LoggingObserver)
-         │    └─→ cache.py (Cache, InMemoryCache, CacheKeyBuilder)
+         │    ├─→ cache.py (Cache, InMemoryCache, CacheKeyBuilder)
+         │    ├─→ sessions.py (SessionStore, JsonFile/SQLite/Redis)
+         │    ├─→ entity_memory.py (EntityMemory)
+         │    ├─→ knowledge_graph.py (KnowledgeGraphMemory)
+         │    └─→ knowledge.py (KnowledgeMemory)
          │
          ├─→ cache.py (core caching)
          │    └─→ types.py, tools.py, usage.py