feat(app,backend): RAG pipeline improvements and token estimation

[comppaz]

Summary

This PR introduces improvements to the RAG pipeline and replaces API-provided token counts with client-side estimation for a more stable UI experience.

Frontend Changes

• Token Estimation: Replaced API-provided token counts with client-side estimation (~4 chars/token)
• Context Indicator: Shows approximate values with ~ prefix, more stable without flickering
• Model Selector: Added provider tracking to prevent unnecessary re-fetches during polling

Backend Changes

• Dynamic Memory Filtering: Threshold-based filtering adapts to match quality (tiered: excellent/good/marginal)
• Enhanced Search Pipeline: Added match type tracking (vector/keyword/hybrid) and RRF scores
• Embedding Safety: Model-specific context windows with intelligent text chunking
• Logging: Comprehensive logging throughout search and query processing
• Other: Blocked all-minilm model, removed conversation history limit

Files Changed

• app/src/components/ContextUsageIndicator.tsx
• app/src/components/ModelSelector.tsx
• app/src/contexts/ConversationContext.tsx
• app/src/pages/ChatPage.tsx
• backend/app/db/search.py
• backend/app/main.py
• backend/app/native_messaging.py
• backend/app/routes/chat.py
• backend/app/routes/settings.py
• backend/app/services/ai.py
• backend/app/services/embeddings.py
• backend/app/services/query_processing.py

Test Plan

• Verify context usage indicator shows approximate tokens with ~ prefix
• Verify model selector doesn't flicker during polling
• Test RAG retrieval with various query lengths
• Verify logging output in backend for search operations
• Test with long documents to verify embedding chunking works