Add user memory feature to SurfSense

[manojag115]

This PR adds the ability to save and retreive user info from memory. Specifically:

1. Adds 2 MCP tools save_memory and retreive_memory
2. The memories are stored in a postgres table
3. the tools convert content aka memories to embeddings and stores it against user's userid
4. Prompts are added to retreive and save memories as needed.

Description

Motivation and Context

FIX #

Screenshots

API Changes

• This PR includes API changes

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactoring
• Documentation
• Dependency/Build system
• Breaking change
• Other (specify):

Testing Performed

• Tested locally
• Manual/QA verification

Checklist

• Follows project coding standards and conventions
• Documentation updated as needed
• Dependencies updated as needed
• No lint/build errors or new warnings
• All relevant tests are passing

High-level PR Summary

This PR implements a Claude-like memory feature that enables the AI assistant to remember user information across conversations. It adds two new MCP tools (save_memory and recall_memory) that store user facts, preferences, instructions, and context as embeddings in a PostgreSQL table with vector similarity search support. The feature includes database migrations, tool implementations with semantic search capabilities, UI components for displaying memory operations, and integration into the chat agent with comprehensive system prompts for guiding when and how to use the memory tools.

⏱️ Estimated Review Time: 1-3 hours

💡 Review Order Suggestion

Order	File Path
1	surfsense_backend/alembic/versions/73_add_user_memories_table.py
2	surfsense_backend/app/db.py
3	surfsense_backend/app/agents/new_chat/tools/user_memory.py
4	surfsense_backend/app/agents/new_chat/tools/__init__.py
5	surfsense_backend/app/agents/new_chat/tools/registry.py
6	surfsense_backend/app/agents/new_chat/system_prompt.py
7	surfsense_backend/app/agents/new_chat/chat_deepagent.py
8	surfsense_backend/app/tasks/chat/stream_new_chat.py
9	surfsense_backend/app/routes/new_chat_routes.py
10	surfsense_web/components/tool-ui/user-memory.tsx
11	surfsense_web/components/tool-ui/index.ts
12	surfsense_web/app/dashboard/[search_space_id]/new-chat/[[...chat_id]]/page.tsx

[vercel]

@manojag115 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[recurseml]

Review by RecurseML

🔍 Review performed on 6e331c3..e3fe4b8

✨ No bugs found, your code is sparkling clean

✅ Files analyzed, no issues (11)

• surfsense_backend/alembic/versions/73_add_user_memories_table.py
• surfsense_backend/app/agents/new_chat/chat_deepagent.py
• surfsense_backend/app/agents/new_chat/system_prompt.py
• surfsense_backend/app/agents/new_chat/tools/__init__.py
• surfsense_backend/app/agents/new_chat/tools/registry.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/new_chat_routes.py
• surfsense_backend/app/tasks/chat/stream_new_chat.py
• surfsense_web/app/dashboard/[search_space_id]/new-chat/[[...chat_id]]/page.tsx
• surfsense_web/components/tool-ui/index.ts
• surfsense_web/components/tool-ui/user-memory.tsx

[recurseml]

Review by RecurseML

🔍 Review performed on e3fe4b8..d640802

✨ No bugs found, your code is sparkling clean

✅ Files analyzed, no issues (3)

• surfsense_backend/alembic/versions/73_add_user_memories_table.py
• surfsense_backend/app/agents/new_chat/tools/user_memory.py
• surfsense_web/components/tool-ui/user-memory.tsx

[MODSetter]

@manojag115 Looks good to me.