Improve semantic search retrieval and ranking

[user1303836]

Summary

• switch article search from single summary vectors to chunked raw-content indexing with SQLite chunk metadata and zvec chunk documents
• rerank retrieved article chunks with a cross-encoder, filter weak matches with a minimum relevance threshold, and clean lore embeddings before indexing/search health checks
• refresh the Discord /search output with clearer embeds, relevance labels, links, and best-match excerpts
• add an article search evaluation harness plus config/docs for tuning chunking and reranking

Verification

• uv run ruff check src/intelstream/config.py src/intelstream/database/models.py src/intelstream/database/repository.py src/intelstream/database/vector_store.py src/intelstream/discord/cogs/lore.py src/intelstream/discord/cogs/search.py src/intelstream/services/article_search.py src/intelstream/services/message_ingestion.py src/intelstream/services/pipeline.py src/intelstream/services/search_eval.py scripts/eval_article_search.py tests/test_database.py tests/test_discord/test_search.py tests/test_services/test_article_search.py tests/test_services/test_pipeline.py tests/test_services/test_search_eval.py tests/test_vector_store.py
• uv run mypy src/
• uv run pytest -q (780 passed)

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.