🧠 SmartMemory - Intelligent Memory System

An intelligent data processing system that creates a knowledge graph of data relationships using vector embeddings and AI analysis.

🎯 What It Does (example: email)

SmartMemory automatically processes emails to:

• Generate vector embeddings using Google Gemini AI
• Store emails in Supabase PostgreSQL with pgvector
• Find semantically similar emails using vector similarity search
• Analyze relationships between emails using LLM
• Build knowledge graphs in Neo4j showing email connections

🏗️ System Architecture

📧 Email Input
    ↓
🤖 Gemini Embedding (1536-dim vectors)
    ↓
🗄️  Supabase Storage (PostgreSQL + pgvector)
    ↓
🔍 Vector Similarity Search
    ↓
🧠 LLM Relationship Analysis
    ↓
🕸️  Neo4j Knowledge Graph

🛠️ Tech Stack

Component	Technology	Purpose
AI/ML	Google Gemini API	Vector embeddings & relationship analysis
Database	Supabase PostgreSQL	Email storage with pgvector extension
Graph DB	Neo4j AuraDB	Knowledge graph for email relationships
Backend	Python 3.11+	Main processing logic
Environment	python-dotenv	Configuration management

📁 Project Structure

SmartMemory/
├── .env                           # Environment variables (API keys, DB credentials)
├── .gitignore                     # Git ignore rules
├── config.py                      # Configuration management
├── email_processor.py             # Main email processing pipeline
├── gemini_service.py              # Gemini AI embedding & analysis
├── supabase_client.py             # Supabase database operations
├── neo4j_client.py                # Neo4j graph database operations
├── main.py                        # Application entry point
├── test_connections.py            # Connection testing utility
├── setup_supabase_function.sql    # SQL function for vector similarity
└── README.md                      # This file

🚀 Quick Start

1. Prerequisites

• Python 3.11+
• Google Gemini API key
• Supabase account with PostgreSQL + pgvector
• Neo4j AuraDB instance

2. Installation

git clone https://github.com/santhosh-005/Smart_Memory.git
cd Smart_Memory

# Create virtual environment
python -m venv .venv
.venv\Scripts\activate  # Windows
# source .venv/bin/activate  # macOS/Linux

# Install dependencies
pip install google-generativeai supabase neo4j python-dotenv

3. Configuration

Create .env file with your credentials:

# Supabase Configuration
NEXT_PUBLIC_SUPABASE_URL=your_supabase_url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_supabase_anon_key

# Gemini API Configuration  
GEMINI_API_KEY=your_gemini_api_key

# Neo4j Configuration
NEO4J_URI=your_neo4j_uri
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=your_neo4j_password

4. Database Setup

Supabase Setup:

1. Create emails table:

CREATE TABLE IF NOT EXISTS emails (
  id BIGSERIAL PRIMARY KEY,
  title TEXT NOT NULL,
  body TEXT NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now(),
  embedding vector(1536)
);

2. Run the SQL from setup_supabase_function.sql in your Supabase SQL Editor

Neo4j Setup:

• No additional setup required - the system creates nodes and relationships automatically

5. Run the System

# Test connections
python test_connections.py

# Process sample emails
python main.py

🔄 System Workflow

Email Processing Pipeline:

1. 📥 Email Input: Title and body text
2. 🎯 Content Preparation: Combine title + body for comprehensive context
3. 🤖 Generate Embedding: Use Gemini with RETRIEVAL_DOCUMENT task type
4. 💾 Store in Supabase: Save email with 1536-dimension vector
5. 📊 Create Neo4j Node: Add email node to knowledge graph
6. 🔍 Find Similar Emails: Use pgvector similarity search via RPC function
7. 🧠 LLM Analysis: Analyze relationships using Gemini Pro
8. 🔗 Update Graph: Create relationship edges in Neo4j

Relationship Types Detected:

• REPLY_TO - Direct email replies
• FOLLOW_UP_ON - Follow-up discussions
• REFERENCES_SAME_TOPIC - Related topic discussions
• RELATED_TO - General relationships

📊 Example Usage

from email_processor import process_email

# Process an email
process_email(
    title="Project Update - Q4 Status", 
    body="Hi team, here's the latest update on our Q4 project..."
)

Output:

--- Processing new email: 'Project Update - Q4 Status' ---
Storing email in Supabase...
Stored successfully. New email ID: 123
Created initial node in Neo4j for email 123
Searching for similar emails...
Found 2 similar emails.
Querying LLM for relationship analysis...
LLM Response: {'relationship_found': True, 'target_email_id': 45, 'relationship_type': 'FOLLOW_UP_ON'}
Relationship found! Creating edge in Neo4j: (123)-[FOLLOW_UP_ON]->(45)
Graph updated.

🔧 Configuration Options

Key settings in config.py:

EMBEDDING_DIMENSIONS = 1536      # Gemini embedding size
SIMILARITY_THRESHOLD = 0.7       # Minimum similarity for relationships  
MATCH_COUNT = 5                  # Max similar emails to analyze

🧪 Testing

# Test all connections
python test_connections.py

# Process sample emails
python main.py

📈 Performance Considerations

• Embedding Generation: ~1-2 seconds per email
• Vector Search: Milliseconds with pgvector indexing
• LLM Analysis: ~2-3 seconds for relationship detection
• Graph Updates: Near-instantaneous

🚨 Common Issues

Gemini API Quota Exceeded

• Solution: Wait for quota reset or upgrade to paid plan
• Check: https://ai.dev/usage?tab=rate-limit

Supabase Connection Issues

• Verify: URL and API key in .env
• Check: pgvector extension is enabled

Neo4j Connection Issues

• Verify: URI and credentials in .env
• Check: AuraDB instance is running

🛣️ Roadmap

• Real-time Processing: WebSocket integration for live email processing
• Advanced Relationships: Thread detection and conversation mapping
• Search Interface: Query interface for finding related emails
• Analytics Dashboard: Visualization of email networks
• Batch Processing: Bulk email import functionality

📝 License

This project is licensed under the MIT License.

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📞 Support

For issues and questions:

• Create an issue on GitHub
• Check the troubleshooting section above
• Review API documentation for Gemini, Supabase, and Neo4j

---

Built with ❤️ for intelligent email management