v0.4.0 — Streaming Search, Robots.txt & Indexing Controls
What's New
NDJSON Streaming Search
The search endpoint now supports streaming results via NDJSON. A smart merge utility combines partial results as they arrive, letting you show results progressively without waiting for the full response.
Robots.txt Compliance
The indexer now reads and respects `robots.txt` directives during crawling, so pages marked as noindex are automatically excluded.
Meta-Based Page Weighting
Extract weight hints directly from page meta tags to boost or suppress individual pages in search rankings without touching config files.
Glob Pattern Filtering
Include or exclude pages from indexing using glob patterns in your config — useful for keeping staging pages, admin routes, or draft content out of the index.
MCP Reranking
The MCP `search` tool now supports reranking, bringing the same result quality improvements available in the HTTP API to AI assistant integrations.
Force Reindex via Env Var
Set `SEARCHSOCKET_FORCE_REINDEX=1` (or `true`/`yes`) in your CI/CD environment to trigger a full reindex without changing CLI flags or config files.
Changed
- Upgraded default embedding model to `jina-embeddings-v5-text-small`
- Upgraded default reranker to `jina-reranker-v3`
- Result merge metric replaced: fraction-based logic swapped for a `maxDisplacement` check, which is more reliable across varying result set sizes