feat: high-priority feature requests from user needs analysis

[RobertLD]

Feature Requests

High-value features identified through analysis of common RAG pipeline patterns and gaps in the current libscope API.

1. [HIGH] Cross-Encoder Re-ranking

Add optional re-ranking of search results using a cross-encoder model. Bi-encoder embeddings optimise for retrieval speed but cross-encoders score (query, document) pairs jointly for higher precision. Expose as rerank: true in SearchOptions.

2. [HIGH] Maximal Marginal Relevance (MMR) Diversity

Current maxChunksPerDocument limits per-document redundancy but not cross-document semantic redundancy. MMR penalises results too similar to already-selected ones, giving better coverage for RAG context. Add diversity: number (0–1 lambda) to SearchOptions.

3. [HIGH] Anthropic Claude LLM Provider

The library is positioned as an MCP/Claude Code tool but the ask-question feature only supports OpenAI and Ollama. Add a Claude provider using the Anthropic Messages API.

4. [HIGH] EPUB + PowerPoint (PPTX) Parsers

High-value enterprise document formats currently unsupported. EPUB covers technical books and manuals; PPTX covers architecture decks and presentations.

5. [MEDIUM] Batch Search API

POST /api/v1/search/batch accepting an array of queries, returning results keyed by query. Enables multi-query RAG decomposition, "related documents" features, and precomputed suggestions.

6. [MEDIUM] Document TTL / Auto-Expiry

Add optional expiresAt field on documents. Prune expired documents on a schedule or on next access. Useful for time-sensitive content like Slack messages or meeting notes.

7. [MEDIUM] Fluent TypeScript SDK

A LibScope class wrapping the functional core API for programmatic use without managing db/provider manually:

const scope = await LibScope.create({ workspace: 'my-app' });
const results = await scope.search('query');

Priority Order

1. Claude LLM provider (quick win, high user demand given MCP positioning)
2. MMR diversity (small implementation, big quality impact)
3. Batch search API (small, unlocks common use cases)
4. EPUB parser (small)
5. Re-ranking (medium, requires transformers.js integration)
6. PPTX parser (small)
7. TTL/expiry (small)
8. Fluent SDK (medium)