[Performance] Brute-force vector search too slow; Task aggregation not utilized

[abakane1]

Issue: Memory Search Performance & Task-Level Retrieval Design

Problem Summary

The current memory search implementation has two major issues:

1. Brute-force vector search is too slow (10-15 seconds for 10k+ chunks)
2. Task aggregation is not utilized during search, defeating the purpose of task grouping

Current Implementation

From src/storage/vector.ts:

export function vectorSearch(...) {
  const all = store.getAllEmbeddings(ownerFilter);  // Loads ALL 10k+ embeddings
  const scored = all.map((row) => ({
    chunkId: row.chunkId,
    score: cosineSimilarity(queryVec, row.vector),  // O(n) calculation
  }));
  scored.sort((a, b) => b.score - a.score);
  return scored.slice(0, topK);
}

Issues

1. No Vector Index

• 10,824 chunks × 2048 dimensions = ~88MB loaded per search
• No HNSW, IVF, or any approximate nearest neighbor index
• Default vectorSearchMaxChunks: 0 means search ALL chunks

2. Task Aggregation Not Used for Search

The tasks table exists:

CREATE TABLE tasks (
  id TEXT PRIMARY KEY,
  title TEXT,
  summary TEXT,
  ...
);

But:

• No FTS index on tasks.title or tasks.summary
• RecallEngine.search() directly searches all 10k+ chunks
• Task is only used for:
  • Grouping chunks during ingestion
  • Generating skills
  • NOT for retrieval optimization

Expected Behavior

Two-level search for better performance and relevance:

Step 1: Search tasks (using FTS on title/summary)
         ↓
Step 2: Only search chunks within matching tasks
         ↓
Step 3: RRF fusion + MMR rerank

This would:

• Reduce search space from 10k chunks to ~100-500 chunks per task
• Improve relevance by focusing on coherent "conversations/tasks"
• Speed up queries by 10-20x

Workarounds Currently Used

• Setting vectorSearchMaxChunks: 2000 (sacrifices 80% of old memories)
• Using getRecentEmbeddings() instead of getAllEmbeddings()

Suggested Solutions

1. Add HNSW vector index for approximate nearest neighbor search
2. Add FTS index on tasks table for task-level search
3. Implement two-level retrieval: Task → Chunks
4. Add task embeddings for semantic task search

Environment

• memos-local version: latest
• Total chunks: ~11,000
• Embedding dimensions: 2048
• Search time: 10-15 seconds (unacceptable)

---

Labels: performance, enhancement, search