Vector HashMap Memory System v2 (CRUD-Focused)

A persistent vector storage system focused on CRUD operations with Redis caching and graph-based memory relationships. Search functionality has been moved to the mcp__omni__ service for better cross-service coordination.

🚀 Current Status: Refactored for Service Separation

Latest Update: January 2025

✅ Simplified API - CRUD operations only, search moved to omni service
✅ Clean Architecture - Clear separation of concerns between services
✅ Testing Verified - 90% API success rate, comprehensive test coverage
✅ Performance Validated - Sub-second response times, efficient caching
✅ Infrastructure Ready - Docker deployment, Redis health monitoring

Architecture Change: Search and query operations now handled by mcp__omni__ service

Features

• Vector HashMap: Fast hash-based lookups with vector embeddings
• Graph Structure: Content nodes connected by contextual edges
• Semantic Analysis: Similarity detection and deduplication
• Persistence: Redis caching + ChromaDB vector storage
• Queue Management: Async request processing with buffering
• Edge Weighting: Automatic weight normalization and resolve tracking
• Inference Engine: Query-based memory retrieval with context

Architecture

┌─────────────────┐
│   FastAPI App   │
├─────────────────┤
│   Middleware    │
│  - Cache        │
│  - RateLimit    │
│  - Queue        │
├─────────────────┤
│ Vector HashMap  │
│  - Embeddings   │
│  - Graph Store  │
├─────────────────┤
│   Storage       │
│  - Redis        │
│  - ChromaDB     │
└─────────────────┘

Project Structure

├── src/                     # Source code
│   ├── api/                 # FastAPI REST endpoints
│   │   └── main.py          # Application entry point
│   ├── core/                # Core business logic
│   │   ├── vector_hashmap.py # Main memory system
│   │   └── middleware.py    # Request processing middleware
│   └── models/              # Data models and schemas
│       └── models.py        # Pydantic models
├── tests/                   # Test suite
│   ├── unit/               # Unit tests (future)
│   ├── integration/        # Integration tests (future)
│   ├── reports/           # Test result reports
│   └── *.py               # Current test files
├── static/                 # Web interface
│   ├── index.html         # Main web UI
│   ├── js/app.js         # Frontend JavaScript
│   └── css/styles.css    # Styling
├── docs/                  # Documentation
│   └── README.md         # Documentation index
└── README.md             # This file

Installation

pip install -r requirements.txt

Quick Start

Using Docker Compose

docker-compose up -d

Manual Setup

1. Start Redis:

redis-server

2. Run the application:

uvicorn src.api.main:app --reload

API Changes (Important)

Simplified CRUD API

The memory service now focuses exclusively on CRUD operations. All search functionality has been moved to the mcp__omni__ service.

New API Functions:

• mcp__memory__remember - Store new memories (Create)
• mcp__memory__update - Update existing memories (Update)
• mcp__memory__forget - Delete memories (Delete)
• mcp__memory__list - List memories with pagination (Read)
• mcp__memory__get - Get specific memory by ID (Read)
• mcp__memory__stats - Get statistics
• mcp__memory__health - Health check

Removed Functions (use mcp__omni__ instead):

• memory/search → Use mcp__omni__search
• memory/query → Use mcp__omni__search
• memory/related → Use mcp__omni__get_related

See MEMORY_SERVICE_API.md for complete API documentation.

API Examples

Store Memory (Create)

mcp__memory__remember
{
  "content": ["word1", "word2", "word3"],
  "context": "brief context"
}

Update Memory

mcp__memory__update
{
  "node_id": "node-123",
  "content": "updated content",
  "context": "new context"
}

Delete Memory

mcp__memory__forget
{
  "node_id": "node-123"
}

List Memories

mcp__memory__list
{
  "offset": 0,
  "limit": 100
}

Inference

POST /memory/infer?query=your+question&context_limit=5

Statistics

GET /memory/statistics

Data Model

Nodes

• Content Nodes: Individual concepts/words with embeddings
• Context Nodes: Contextual relationships between content

Edges

• Connect content nodes
• Weighted by distance and frequency
• resolve counter increases with repetition
• Context embedding for semantic relationships

Testing

pytest test_memory_system.py -v

Configuration

Parameter	Default	Description
redis_url	redis://localhost:6379	Redis connection URL
chroma_persist_dir	./chroma_db	ChromaDB storage directory
embedding_model	all-MiniLM-L6-v2	Sentence transformer model
cache_ttl	3600	Cache TTL in seconds
max_edges_per_node	100	Maximum edges per node

Example Usage

from models import ContentInput
import httpx

# Store memory
input_data = {
    "content": ["machine", "learning", "algorithm"],
    "context": "AI basics"
}

response = httpx.post("http://localhost:8000/memory/store", json=input_data)

# Query memory
query_data = {
    "query": "explain machine learning",
    "top_k": 10,
    "threshold": 0.7
}

response = httpx.post("http://localhost:8000/memory/query", json=query_data)
print(response.json())

Performance Considerations

• Embeddings are cached in Redis with configurable TTL
• Batch processing for large inputs
• Rate limiting prevents overload
• Request queue manages concurrent operations
• Edge weight normalization prevents unbounded growth

Future Enhancements (Phase 2)

• Additional abstraction layers
• Advanced inference algorithms
• Multi-modal embeddings
• Distributed processing
• Real-time synchronization