Claude Context 实现原理

[Pines-Cheng]

1. Merkle trees 增量索引
2. 支持 MCP
3. 支持 CLI 模式和 WebIDE 模式（本地代码的场景：如无代码仓库、代码未推送等）

高级混合代码搜索是怎么实现的？

根据代码库上下文，这指的是 Claude Context 系统中的混合搜索功能，它结合了密集向量搜索（语义相似性）和 BM25 稀疏向量搜索（文本匹配）。

混合搜索架构

混合搜索通过 HYBRID_MODE 环境变量控制，默认启用。 1 系统根据此设置决定创建混合集合还是常规集合。 2

混合集合创建

混合集合包含特殊的 BM25 函数和稀疏向量字段： 3

集合模式包含：

• vector 字段：密集向量（语义嵌入）
• sparse_vector 字段：稀疏向量（BM25）
• content_bm25_emb 函数：自动从内容生成 BM25 嵌入 4

数据插入过程

在混合模式下，文档插入时只需提供内容和密集向量，BM25 稀疏向量由数据库自动生成： 5

混合搜索执行

搜索时，系统准备两个搜索请求：

1. 密集向量搜索：使用语义嵌入 6

2. 稀疏向量搜索：使用 BM25 文本匹配 7

RRF 重排序

两种搜索结果通过 Reciprocal Rank Fusion (RRF) 算法合并和重排序： 8

在 REST API 实现中，重排序策略配置为： 9

搜索参数配置

• 密集搜索：使用 COSINE 相似度，nprobe: 10
• 稀疏搜索：使用 BM25，drop_ratio_search: 0.2
• RRF 参数：k: 100 10

这种混合方法结合了语义理解（密集向量）和精确文本匹配（BM25），提供更准确和全面的代码搜索结果。

Notes

混合搜索功能在两个 Milvus 实现中都可用：gRPC 客户端 (MilvusVectorDatabase) 和 REST API 客户端 (MilvusRestfulVectorDatabase)，它们使用相同的核心逻辑但通过不同的传输协议与 Milvus 通信。

Wiki pages you might want to explore:

• Core Indexing Engine (zilliztech/claude-context)
• Vector Database Layer (zilliztech/claude-context)
• Supported AI Assistants (zilliztech/claude-context)

Citations

File: packages/core/src/context.ts (L223-229)

    private getIsHybrid(): boolean {
        const isHybridEnv = envManager.get('HYBRID_MODE');
        if (isHybridEnv === undefined || isHybridEnv === null) {
            return true; // Default to true
        }
        return isHybridEnv.toLowerCase() === 'true';
    }

File: packages/core/src/context.ts (L234-240)

    public getCollectionName(codebasePath: string): string {
        const isHybrid = this.getIsHybrid();
        const normalizedPath = path.resolve(codebasePath);
        const hash = crypto.createHash('md5').update(normalizedPath).digest('hex');
        const prefix = isHybrid === true ? 'hybrid_code_chunks' : 'code_chunks';
        return `${prefix}_${hash.substring(0, 8)}`;
    }

File: packages/core/src/context.ts (L440-446)

            const searchRequests: HybridSearchRequest[] = [
                {
                    data: queryEmbedding.vector,
                    anns_field: "vector",
                    param: { "nprobe": 10 },
                    limit: topK
                },

File: packages/core/src/context.ts (L447-453)

                {
                    data: query,
                    anns_field: "sparse_vector",
                    param: { "drop_ratio_search": 0.2 },
                    limit: topK
                }
            ];

File: packages/core/src/context.ts (L464-467)

                    rerank: {
                        strategy: 'rrf',
                        params: { k: 100 }
                    },

File: packages/core/src/context.ts (L818-847)

        if (isHybrid === true) {
            // Create hybrid vector documents
            const documents: VectorDocument[] = chunks.map((chunk, index) => {
                if (!chunk.metadata.filePath) {
                    throw new Error(`Missing filePath in chunk metadata at index ${index}`);
                }

                const relativePath = path.relative(codebasePath, chunk.metadata.filePath);
                const fileExtension = path.extname(chunk.metadata.filePath);
                const { filePath, startLine, endLine, ...restMetadata } = chunk.metadata;

                return {
                    id: this.generateId(relativePath, chunk.metadata.startLine || 0, chunk.metadata.endLine || 0, chunk.content),
                    content: chunk.content, // Full text content for BM25 and storage
                    vector: embeddings[index].vector, // Dense vector
                    relativePath,
                    startLine: chunk.metadata.startLine || 0,
                    endLine: chunk.metadata.endLine || 0,
                    fileExtension,
                    metadata: {
                        ...restMetadata,
                        codebasePath,
                        language: chunk.metadata.language || 'unknown',
                        chunkIndex: index
                    }
                };
            });

            // Store to vector database
            await this.vectorDatabase.insertHybrid(this.getCollectionName(codebasePath), documents);

File: packages/core/src/vectordb/milvus-restful-vectordb.ts (L527-536)

                    functions: [
                        {
                            name: "content_bm25_emb",
                            description: "content bm25 function",
                            type: "BM25",
                            inputFieldNames: ["content"],
                            outputFieldNames: ["sparse_vector"],
                            params: {},
                        },
                    ],

File: packages/core/src/vectordb/milvus-restful-vectordb.ts (L555-564)

                            fieldName: "vector",
                            dataType: "FloatVector",
                            elementTypeParams: {
                                dim: dimension
                            }
                        },
                        {
                            fieldName: "sparse_vector",
                            dataType: "SparseFloatVector"
                        },

File: packages/core/src/vectordb/milvus-restful-vectordb.ts (L705-721)

                searchParams: {
                    metricType: "COSINE",
                    params: searchRequests[0].param || { "nprobe": 10 }
                }
            };

            // For sparse vector search - data must be array of queries: ["query text"]
            const search_param_2: any = {
                data: Array.isArray(searchRequests[1].data) ? searchRequests[1].data : [searchRequests[1].data],
                annsField: searchRequests[1].anns_field, // "sparse_vector"
                limit: searchRequests[1].limit,
                outputFields: ["*"],
                searchParams: {
                    metricType: "BM25",
                    params: searchRequests[1].param || { "drop_ratio_search": 0.2 }
                }
            };

File: packages/core/src/vectordb/milvus-restful-vectordb.ts (L729-734)

            const rerank_strategy = {
                strategy: "rrf",
                params: {
                    k: 100
                }
            };

[Pines-Cheng]

Merkle Trees 存储位置

Merkle trees 存储在用户主目录下的 ~/.context/merkle/ 目录中。 1 每个代码库都有一个唯一的快照文件，文件名基于代码库绝对路径的 MD5 哈希值生成。 2

快照文件结构

快照文件以 JSON 格式存储，包含两个主要部分： 3

• fileHashes: 文件路径到 SHA-256 哈希值的映射
• merkleDAG: 序列化的 Merkle DAG 数据结构

文件同步机制

FileSynchronizer 类负责管理 Merkle trees 的创建、存储和加载： 4

当系统初始化时，会尝试从快照文件加载现有的 Merkle tree 状态。 5 如果快照文件不存在，系统会生成新的文件哈希并创建新的 Merkle tree。 6

变更检测过程

系统使用 Merkle trees 进行高效的变更检测： 7

1. 生成当前文件状态的新 Merkle DAG
2. 与存储的快照进行比较
3. 如果检测到变更，进行文件级别的详细比较
4. 更新快照文件

清理机制

系统还提供了删除特定代码库快照的静态方法： 8

Notes

这个 Merkle tree 存储机制是 Claude Context 增量索引功能的核心，它通过 SHA-256 文件哈希和 Merkle DAG 结构实现高效的文件变更检测，避免重新处理未更改的文件。 9 存储位置的设计确保了跨不同环境的一致性，同时为每个代码库维护独立的状态快照。

Wiki pages you might want to explore:

• Core Indexing Engine (zilliztech/claude-context)
• Supported AI Assistants (zilliztech/claude-context)

Citations

File: packages/core/src/sync/synchronizer.ts (L7-20)

export class FileSynchronizer {
    private fileHashes: Map<string, string>;
    private merkleDAG: MerkleDAG;
    private rootDir: string;
    private snapshotPath: string;
    private ignorePatterns: string[];

    constructor(rootDir: string, ignorePatterns: string[] = []) {
        this.rootDir = rootDir;
        this.snapshotPath = this.getSnapshotPath(rootDir);
        this.fileHashes = new Map();
        this.merkleDAG = new MerkleDAG();
        this.ignorePatterns = ignorePatterns;
    }

File: packages/core/src/sync/synchronizer.ts (L22-30)

    private getSnapshotPath(codebasePath: string): string {
        const homeDir = os.homedir();
        const merkleDir = path.join(homeDir, '.context', 'merkle');

        const normalizedPath = path.resolve(codebasePath);
        const hash = crypto.createHash('md5').update(normalizedPath).digest('hex');

        return path.join(merkleDir, `${hash}.json`);
    }

File: packages/core/src/sync/synchronizer.ts (L224-248)

    public async checkForChanges(): Promise<{ added: string[], removed: string[], modified: string[] }> {
        console.log('[Synchronizer] Checking for file changes...');

        const newFileHashes = await this.generateFileHashes(this.rootDir);
        const newMerkleDAG = this.buildMerkleDAG(newFileHashes);

        // Compare the DAGs
        const changes = MerkleDAG.compare(this.merkleDAG, newMerkleDAG);

        // If there are any changes in the DAG, we should also do a file-level comparison
        if (changes.added.length > 0 || changes.removed.length > 0 || changes.modified.length > 0) {
            console.log('[Synchronizer] Merkle DAG has changed. Comparing file states...');
            const fileChanges = this.compareStates(this.fileHashes, newFileHashes);

            this.fileHashes = newFileHashes;
            this.merkleDAG = newMerkleDAG;
            await this.saveSnapshot();

            console.log(`[Synchronizer] Found changes: ${fileChanges.added.length} added, ${fileChanges.removed.length} removed, ${fileChanges.modified.length} modified.`);
            return fileChanges;
        }

        console.log('[Synchronizer] No changes detected based on Merkle DAG comparison.');
        return { added: [], removed: [], modified: [] };
    }

File: packages/core/src/sync/synchronizer.ts (L291-295)

        const data = JSON.stringify({
            fileHashes: fileHashesArray,
            merkleDAG: this.merkleDAG.serialize()
        });
        await fs.writeFile(this.snapshotPath, data, 'utf-8');

File: packages/core/src/sync/synchronizer.ts (L299-324)

    private async loadSnapshot(): Promise<void> {
        try {
            const data = await fs.readFile(this.snapshotPath, 'utf-8');
            const obj = JSON.parse(data);

            // Reconstruct Map without using constructor with iterator
            this.fileHashes = new Map();
            for (const [key, value] of obj.fileHashes) {
                this.fileHashes.set(key, value);
            }

            if (obj.merkleDAG) {
                this.merkleDAG = MerkleDAG.deserialize(obj.merkleDAG);
            }
            console.log(`Loaded snapshot from ${this.snapshotPath}`);
        } catch (error: any) {
            if (error.code === 'ENOENT') {
                console.log(`Snapshot file not found at ${this.snapshotPath}. Generating new one.`);
                this.fileHashes = await this.generateFileHashes(this.rootDir);
                this.merkleDAG = this.buildMerkleDAG(this.fileHashes);
                await this.saveSnapshot();
            } else {
                throw error;
            }
        }
    }

File: packages/core/src/sync/synchronizer.ts (L329-347)

    static async deleteSnapshot(codebasePath: string): Promise<void> {
        const homeDir = os.homedir();
        const merkleDir = path.join(homeDir, '.context', 'merkle');
        const normalizedPath = path.resolve(codebasePath);
        const hash = crypto.createHash('md5').update(normalizedPath).digest('hex');
        const snapshotPath = path.join(merkleDir, `${hash}.json`);

        try {
            await fs.unlink(snapshotPath);
            console.log(`Deleted snapshot file: ${snapshotPath}`);
        } catch (error: any) {
            if (error.code === 'ENOENT') {
                console.log(`Snapshot file not found (already deleted): ${snapshotPath}`);
            } else {
                console.error(`[Synchronizer] Failed to delete snapshot file ${snapshotPath}:`, error.message);
                throw error; // Re-throw non-ENOENT errors
            }
        }
    }

File: packages/core/README.md (L231-269)

## File Synchronization Architecture

Claude Context implements an intelligent file synchronization system that efficiently tracks and processes only the files that have changed since the last indexing operation. This dramatically improves performance when working with large codebases.

![File Synchronization Architecture](../../assets/file_synchronizer.png)

### How It Works

The file synchronization system uses a **Merkle tree-based approach** combined with SHA-256 file hashing to detect changes:

#### 1. File Hashing
- Each file in the codebase is hashed using SHA-256
- File hashes are computed based on file content, not metadata
- Hashes are stored with relative file paths for consistency across different environments

#### 2. Merkle Tree Construction
- All file hashes are organized into a Merkle tree structure
- The tree provides a single root hash that represents the entire codebase state
- Any change to any file will cause the root hash to change

#### 3. Snapshot Management
- File synchronization state is persisted to `~/.context/merkle/` directory
- Each codebase gets a unique snapshot file based on its absolute path hash
- Snapshots contain both file hashes and serialized Merkle tree data

#### 4. Change Detection Process
1. **Quick Check**: Compare current Merkle root hash with stored snapshot
2. **Detailed Analysis**: If root hashes differ, perform file-by-file comparison
3. **Change Classification**: Categorize changes into three types:
   - **Added**: New files that didn't exist before
   - **Modified**: Existing files with changed content
   - **Removed**: Files that were deleted from the codebase

#### 5. Incremental Updates
- Only process files that have actually changed
- Update vector database entries only for modified chunks
- Remove entries for deleted files
- Add entries for new files

[Pines-Cheng]

相同仓库的不同分支怎么避免文件重复索引？

当前索引机制

Claude Context 系统使用基于代码库绝对路径的索引策略。 1 每个代码库路径都会生成一个唯一的 MD5 哈希值作为快照文件名，存储在 ~/.context/merkle/ 目录中。

分支索引问题

问题核心：如果您在同一个仓库的不同分支之间切换，由于使用的是相同的绝对路径，系统会将它们视为同一个代码库进行索引。

当您切换分支时，FileSynchronizer 会检测到文件变更： 2

系统会将分支间的文件差异识别为：

• 添加的文件：新分支中存在但旧分支中不存在的文件
• 删除的文件：旧分支中存在但新分支中不存在的文件
• 修改的文件：两个分支中都存在但内容不同的文件

增量重新索引机制

当检测到变更时，系统会执行增量重新索引： 3

这个过程包括：

1. 删除已移除文件的索引块
2. 删除已修改文件的旧索引块
3. 为新增和修改的文件创建新的索引块

实际影响

虽然系统会自动处理分支切换，但这意味着：

• 每次分支切换都会触发重新索引，可能耗时较长
• 无法同时保持多个分支的索引，只能索引当前检出的分支
• 频繁切换分支会导致重复的索引工作

当前限制

目前的设计没有分支感知机制，无法区分不同分支或同时维护多个分支的索引。 系统将整个工作目录视为单一的索引单元，而不考虑 Git 分支状态。

Notes

如果您需要为多个分支维护独立的索引，目前的解决方案是为每个分支使用不同的工作目录（例如通过 git worktree 创建多个工作树），这样每个目录会有独立的绝对路径和对应的索引。

Wiki pages you might want to explore:

• Core Indexing Engine (zilliztech/claude-context)
• MCP Server (zilliztech/claude-context)
• Supported AI Assistants (zilliztech/claude-context)

Citations

File: packages/core/src/sync/synchronizer.ts (L22-30)

    private getSnapshotPath(codebasePath: string): string {
        const homeDir = os.homedir();
        const merkleDir = path.join(homeDir, '.context', 'merkle');

        const normalizedPath = path.resolve(codebasePath);
        const hash = crypto.createHash('md5').update(normalizedPath).digest('hex');

        return path.join(merkleDir, `${hash}.json`);
    }

File: packages/core/src/sync/synchronizer.ts (L224-248)

    public async checkForChanges(): Promise<{ added: string[], removed: string[], modified: string[] }> {
        console.log('[Synchronizer] Checking for file changes...');

        const newFileHashes = await this.generateFileHashes(this.rootDir);
        const newMerkleDAG = this.buildMerkleDAG(newFileHashes);

        // Compare the DAGs
        const changes = MerkleDAG.compare(this.merkleDAG, newMerkleDAG);

        // If there are any changes in the DAG, we should also do a file-level comparison
        if (changes.added.length > 0 || changes.removed.length > 0 || changes.modified.length > 0) {
            console.log('[Synchronizer] Merkle DAG has changed. Comparing file states...');
            const fileChanges = this.compareStates(this.fileHashes, newFileHashes);

            this.fileHashes = newFileHashes;
            this.merkleDAG = newMerkleDAG;
            await this.saveSnapshot();

            console.log(`[Synchronizer] Found changes: ${fileChanges.added.length} added, ${fileChanges.removed.length} removed, ${fileChanges.modified.length} modified.`);
            return fileChanges;
        }

        console.log('[Synchronizer] No changes detected based on Merkle DAG comparison.');
        return { added: [], removed: [], modified: [] };
    }

File: packages/core/src/context.ts (L315-382)

    async reindexByChange(
        codebasePath: string,
        progressCallback?: (progress: { phase: string; current: number; total: number; percentage: number }) => void
    ): Promise<{ added: number, removed: number, modified: number }> {
        const collectionName = this.getCollectionName(codebasePath);
        const synchronizer = this.synchronizers.get(collectionName);

        if (!synchronizer) {
            // Load project-specific ignore patterns before creating FileSynchronizer
            await this.loadIgnorePatterns(codebasePath);

            // To be safe, let's initialize if it's not there.
            const newSynchronizer = new FileSynchronizer(codebasePath, this.ignorePatterns);
            await newSynchronizer.initialize();
            this.synchronizers.set(collectionName, newSynchronizer);
        }

        const currentSynchronizer = this.synchronizers.get(collectionName)!;

        progressCallback?.({ phase: 'Checking for file changes...', current: 0, total: 100, percentage: 0 });
        const { added, removed, modified } = await currentSynchronizer.checkForChanges();
        const totalChanges = added.length + removed.length + modified.length;

        if (totalChanges === 0) {
            progressCallback?.({ phase: 'No changes detected', current: 100, total: 100, percentage: 100 });
            console.log('[Context] ✅ No file changes detected.');
            return { added: 0, removed: 0, modified: 0 };
        }

        console.log(`[Context] 🔄 Found changes: ${added.length} added, ${removed.length} removed, ${modified.length} modified.`);

        let processedChanges = 0;
        const updateProgress = (phase: string) => {
            processedChanges++;
            const percentage = Math.round((processedChanges / (removed.length + modified.length + added.length)) * 100);
            progressCallback?.({ phase, current: processedChanges, total: totalChanges, percentage });
        };

        // Handle removed files
        for (const file of removed) {
            await this.deleteFileChunks(collectionName, file);
            updateProgress(`Removed ${file}`);
        }

        // Handle modified files
        for (const file of modified) {
            await this.deleteFileChunks(collectionName, file);
            updateProgress(`Deleted old chunks for ${file}`);
        }

        // Handle added and modified files
        const filesToIndex = [...added, ...modified].map(f => path.join(codebasePath, f));

        if (filesToIndex.length > 0) {
            await this.processFileList(
                filesToIndex,
                codebasePath,
                (filePath, fileIndex, totalFiles) => {
                    updateProgress(`Indexed ${filePath} (${fileIndex}/${totalFiles})`);
                }
            );
        }

        console.log(`[Context] ✅ Re-indexing complete. Added: ${added.length}, Removed: ${removed.length}, Modified: ${modified.length}`);
        progressCallback?.({ phase: 'Re-indexing complete!', current: totalChanges, total: totalChanges, percentage: 100 });

        return { added: added.length, removed: removed.length, modified: modified.length };
    }