Memory System

🧠 Memory System

GPT Home uses LangMem for persistent memory storage, enabling the assistant to remember user preferences, past interactions, and important context across conversations.

Overview

┌────────────────────────────────────────────────────────────────────┐
│                        Memory System                               │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                     MemoryManager                           │   │
│  │                     (Facade Pattern)                        │   │
│  │                                                             │   │
│  │  ┌─────────────────┐        ┌─────────────────┐             │   │
│  │  │    Semantic     │        │    Episodic     │             │   │
│  │  │    Manager      │        │    Manager      │             │   │
│  │  │                 │        │                 │             │   │
│  │  │ User prefs,     │        │ Conversation    │             │   │
│  │  │ facts, topics   │        │ summaries,      │             │   │
│  │  │                 │        │ interactions    │             │   │
│  │  └────────┬────────┘        └────────┬────────┘             │   │
│  │           │                          │                      │   │
│  │           └──────────┬───────────────┘                      │   │
│  │                      ▼                                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                         │                                          │
│                         ▼                                          │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │              AsyncPostgresStore (LangGraph)                 │   │
│  │                                                             │   │
│  │   ┌──────────────────────────────────────────────────────┐  │   │
│  │   │  PostgreSQL + pgvector                               │  │   │
│  │   │                                                      │  │   │
│  │   │  • Vector embeddings (1536 dims)                     │  │   │
│  │   │  • Semantic similarity search                        │  │   │
│  │   │  • Namespaced storage per user                       │  │   │
│  │   └──────────────────────────────────────────────────────┘  │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

---

Memory Types

GPT Home implements three types of memories following LangMem's conceptual model:

1. Semantic Memories

Facts and knowledge about the user

class MemoryType(Enum):
    SEMANTIC = auto()    # Facts and knowledge

Examples:

• User preferences: "User prefers Celsius for temperature"
• Personal facts: "User's name is John"
• Recurring interests: "User often asks about stocks"

2. Episodic Memories

Past experiences and interactions

class MemoryType(Enum):
    EPISODIC = auto()    # Past experiences

Examples:

• "User asked about weather in Paris last week"
• "User scheduled a meeting with Bob on Tuesday"
• "Successful alarm set for 7 AM worked well"

3. Procedural Memories

Learned system behaviors (future enhancement)

class MemoryType(Enum):
    PROCEDURAL = auto()  # System behavior

Examples:

• Optimized prompts for specific tasks
• Learned response patterns

---

MemoryManager Class

The MemoryManager in src/memory/manager.py provides a unified interface:

class MemoryManager:
    """Manages agent memories using the Facade pattern.
    
    Provides a unified interface for:
    - Semantic memories (user preferences, facts)
    - Episodic memories (conversation history)
    - Procedural memories (learned behaviors)
    """
    
    def __init__(
        self,
        store: BaseStore,
        model: str = "gpt-4o-mini",
        user_id: str = "default"
    ):
        self.store = store
        self.model = model
        self.user_id = user_id
        
        # Semantic memory manager
        self._semantic_manager = create_memory_store_manager(
            model,
            store=store,
            namespace=("memories", "{user_id}", "semantic"),
            instructions="""Extract and manage semantic memories about the user.
Focus on:
- Personal preferences (display mode, communication style, etc.)
- Important facts shared by the user
- Recurring topics of interest
Consolidate related memories to avoid redundancy."""
        )
        
        # Episodic memory manager
        self._episodic_manager = create_memory_store_manager(
            model,
            store=store,
            namespace=("memories", "{user_id}", "episodic"),
            instructions="""Extract episodic memories from conversations.
Focus on:
- Successful interaction patterns
- Notable events or requests
- Context that might be relevant for future interactions"""
        )

Key Methods

Method	Description
add_semantic_memory(content, user_id)	Store a fact about the user
add_episodic_memory(content, user_id)	Store an interaction record
search_memories(query, user_id, limit)	Semantic search for relevant memories
get_all_memories(user_id)	Retrieve all memories for a user

---

Memory Storage Architecture

PostgreSQL Schema

-- Memory Store Table (LangGraph AsyncPostgresStore)
CREATE TABLE store (
    prefix TEXT,              -- Namespace: ("memories", "user_id", "type")
    key TEXT,                 -- Unique memory ID
    value JSONB,              -- Memory content
    created_at TIMESTAMPTZ,
    updated_at TIMESTAMPTZ,
    embedding VECTOR(1536),   -- pgvector embedding
    PRIMARY KEY (prefix, key)
);

-- Vector similarity index for semantic search
CREATE INDEX store_embedding_idx 
ON store USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

Namespace Structure

memories/
├── user_1/
│   ├── semantic/
│   │   ├── pref_temperature_celsius
│   │   ├── fact_name_john
│   │   └── interest_stocks
│   └── episodic/
│       ├── interaction_20240115_weather
│       └── interaction_20240116_meeting
├── user_2/
│   ├── semantic/
│   └── episodic/
└── default/
    ├── semantic/
    └── episodic/

---

Memory Flow

Storing Memories (Background Processing)

User: "I prefer temperatures in Celsius"
              │
              ▼
        ┌───────────┐
        │  Agent    │──────▶ Response: "Got it! I'll use Celsius."
        │  Response │
        └─────┬─────┘
              │
              ▼
        ┌───────────────────────────────────────────────┐
        │    Background Memory Processing               │
        │    (_background_memory_processing)            │
        │                                               │
        │    1. Analyze conversation                    │
        │    2. Extract memory-worthy content           │
        │    3. Classify as semantic/episodic           │
        │    4. Generate embedding via LiteLLM          │
        │    5. Store in PostgreSQL                     │
        └───────────────────────────────────────────────┘

Retrieving Memories (Agent Invocation)

User: "What's the weather?"
              │
              ▼
        ┌───────────────────────────────────────────────┐
        │    Agent Initialization                       │
        │                                               │
        │    1. Build system prompt                     │
        │    2. Search memories for "weather"           │
        │       • Vector similarity search              │
        │       • Returns: "User prefers Celsius"       │
        │    3. Include memories in prompt context      │
        └───────────────────────────────────────────────┘
              │
              ▼
        ┌───────────────────────────────────────────────┐
        │    System Prompt with Memory                  │
        │                                               │
        │    ## User Memories                           │
        │    <memories>                                 │
        │    - User prefers Celsius for temperature     │
        │    </memories>                                │
        │                                               │
        └───────────────────────────────────────────────┘
              │
              ▼
        Agent responds with temperature in Celsius

---

Memory Tools (LangMem)

The agent has access to memory tools for explicit memory management:

manage_memory_tool

Created via LangMem:

from langmem import create_manage_memory_tool

manage_memory = create_manage_memory_tool(
    namespace=("memories", "{user_id}")
)

The agent can invoke this to:

• Save new memories explicitly
• Update existing memories
• Delete outdated memories

search_memory_tool

from langmem import create_search_memory_tool

search_memory = create_search_memory_tool(
    namespace=("memories", "{user_id}")
)

The agent can invoke this to:

• Search for relevant context
• Find specific memories
• Retrieve user preferences

---

Store Factory

The create_memory_store factory in src/memory/store.py handles store creation:

async def create_memory_store(
    database_url: Optional[str] = None,
    embedding_model: str = "openai:text-embedding-3-small",
    embedding_dims: int = 1536,
) -> BaseStore:
    """Factory function to create appropriate memory store.
    
    Uses Strategy pattern to select between PostgreSQL and in-memory storage.
    """
    index_config = {
        "dims": embedding_dims,
        "embed": embedding_model,
        "fields": ["content", "summary", "$"],
    }
    
    if database_url:
        try:
            store = AsyncPostgresStore.from_conn_string(
                database_url,
                index=index_config
            )
            await store.setup()
            return store
        except Exception as e:
            print(f"Warning: PostgreSQL failed, using in-memory: {e}")
    
    # Fallback to in-memory store
    return InMemoryStore(index=index_config)

---

Embedding Configuration

Embedding Model Options

Model	Provider	Dimensions	Notes
text-embedding-3-small	OpenAI	1536	Default, fast
text-embedding-3-large	OpenAI	3072	Higher quality
cohere/embed-english-v3.0	Cohere	1024	Alternative
voyage/voyage-large-2	Voyage	1024	Specialized

Configuration

# .env
EMBEDDING_MODEL=openai:text-embedding-3-small
EMBEDDING_DIMS=1536

---

Memory Retrieval in System Prompt

The agent's system prompt includes relevant memories:

def _build_system_prompt(self, state: AgentState) -> list:
    """Build system prompt with memory context."""
    memories_text = ""
    
    if self.store and state.get("user_id"):
        # Semantic search based on current message
        memories = store.search(
            ("memories", state["user_id"]),
            query=state["messages"][-1].content,
            limit=5
        )
        if memories:
            memories_text = "\n".join([
                f"- {m.value.get('content', m.value)}" 
                for m in memories
            ])
    
    system_content = f"""...
    
## User Memories
<memories>
{memories_text if memories_text else "No memories stored yet."}
</memories>

When users share preferences or important information, 
use the memory tools to save them."""

    return [{"role": "system", "content": system_content}, *state["messages"]]

---

Background Memory Processing

After each interaction, memories are processed asynchronously:

async def _background_memory_processing(
    user_input: str,
    response: str,
    user_id: str
):
    """Background task to extract and store memories from conversation."""
    try:
        memory_manager = await _get_memory_manager()
        
        # Combine user input and response for analysis
        conversation = f"User: {user_input}\nAssistant: {response}"
        
        # Extract and store semantic memories (preferences, facts)
        await memory_manager.extract_and_store_semantic(
            conversation, 
            user_id
        )
        
        # Extract and store episodic memories (interactions)
        await memory_manager.extract_and_store_episodic(
            conversation,
            user_id
        )
        
    except Exception as e:
        logger.error(f"Background memory processing failed: {e}")

---

Memory Consolidation

LangMem automatically consolidates similar memories to avoid redundancy:

Before Consolidation:
├── "User likes jazz music"
├── "User prefers jazz over rock"
├── "User mentioned liking jazz"

After Consolidation:
├── "User prefers jazz music over other genres"

This is controlled by the instructions parameter:

create_memory_store_manager(
    model,
    store=store,
    namespace=("memories", "{user_id}", "semantic"),
    instructions="""...
Consolidate related memories to avoid redundancy."""
)

---

Persistence Fallback

When PostgreSQL is unavailable, the system falls back to in-memory storage:

┌───────────────────────────────────────────────────────────┐
│                  Persistence Strategy                     │
├───────────────────────────────────────────────────────────┤
│                                                           │
│   DATABASE_URL set?                                       │
│        │                                                  │
│        ├── Yes ──▶ Try PostgreSQL                        │
│        │              │                                   │
│        │              ├── Success ──▶ AsyncPostgresStore |
│        │              │                                   │
│        │              └── Fail ──▶ InMemoryStore         │
│        │                          (with warning)          │
│        │                                                  │
│        └── No ───▶ InMemoryStore                         │
│                                                           │
└───────────────────────────────────────────────────────────┘

Warning: In-memory storage is lost on restart. Always configure PostgreSQL for production.

---

Debugging Memories

View Stored Memories

Connect to PostgreSQL:

docker compose exec db psql -U gpt_home -d gpt_home

Query memories:

-- List all memories for a user
SELECT prefix, key, value, created_at 
FROM store 
WHERE prefix LIKE 'memories,default%'
ORDER BY created_at DESC;

-- Search by content
SELECT * FROM store 
WHERE value->>'content' ILIKE '%weather%';

-- Check embedding dimensions
SELECT key, array_length(embedding, 1) as dims 
FROM store LIMIT 5;

Clear Memories

Via Web Interface (Recommended):

Navigate to Settings → Quick Actions → Clear Memories button. This will:

• Clear all conversation history (checkpoints)
• Clear all stored memories (semantic & episodic)
• Require confirmation before proceeding

Via API:

curl -X POST http://gpt-home.local/clearMemory

Response:

{
  "success": true,
  "message": "Memory cleared successfully"
}

Via SQL (Manual):

-- Delete all memories for a user
DELETE FROM store WHERE prefix LIKE 'memories,user_id%';

-- Clear all memories and conversation history
TRUNCATE TABLE checkpoints CASCADE;
TRUNCATE TABLE checkpoint_blobs CASCADE;
TRUNCATE TABLE checkpoint_writes CASCADE;
TRUNCATE TABLE store CASCADE;

⚠️ Warning: Clearing memory is irreversible. The assistant will lose all learned preferences and conversation history.

---

Configuration Reference

Variable	Default	Description
DATABASE_URL	-	PostgreSQL connection string
EMBEDDING_MODEL	openai:text-embedding-3-small	Model for embeddings (format: provider:model)
EMBEDDING_DIMS	1536	Embedding vector dimensions

---

Next Steps

• See Agent System for how memories integrate with the agent
• Check Configuration for database setup
• Explore API Reference for memory-related endpoints