Implement dense-first retrieval, citation enforcement, and Redis caching

[sanjeevafk]

Retrieval Pipeline Enhancements:

• Implements a dense-first retrieval strategy: dense search is attempted first, and only if results are insufficient does the system fall back to hybrid (dense + lexical) retrieval. The new dense_search_v5 RPC is added for this purpose. [1] [2]
• Reranking logic is improved: reranking is now skipped for clear dense matches (above a configurable similarity threshold), reducing unnecessary latency. Depth 5 always reranks, while depths 3-4 only rerank for marginal hits.
• Retrieval confidence scoring is refactored for clarity and accuracy, using a new helper function to map scores to confidence bands based on retrieval method. [1] [2]

Query Result Caching and Quotas:

• Adds a Redis-backed cache for query results, keyed on query parameters, and returns cached results when available to reduce latency and load. [1] [2] [3]
• Implements per-API-key daily token quotas, with enforcement and accounting via Redis. If Redis is unavailable, the system fails open with warnings. [1] [2] [3]
• Both caching and quotas are configurable via new settings, with sensible defaults and the ability to disable enforcement.

Configuration and API Improvements:

• Adds new configuration options for dense/hybrid retrieval thresholds, reranking, cache TTL, and quota limits in Settings.
• Updates API request/response and metadata to reflect retrieval mode, rerank status, and confidence, improving observability and downstream usage. [1] [2]

Code Refactoring and Maintainability:

• Refactors query.py for clarity: introduces helper functions, improves error handling, and streamlines logic for determining retrieval/reranking modes and cache usage. [1] [2]
• Extracts cache and quota logic into a dedicated, well-documented module (api/services/cache.py).

Other Notable Changes:

• Ensures that citation enforcement and metadata handling are consistent for both standard and streaming query endpoints. [1] [2] [3] [4]