v0.12.1

Fixed

• RAG Crawler www/non-www Redirects - Handle sites that redirect between www and apex domains
  • Sites like example.com → www.example.com were being blocked as "external redirects"
  • New _is_same_site() helper normalizes domains by stripping www. prefix before comparison
  • Fixes both fetch_page() redirect check and _recursive_crawl() external link filtering

v0.12.0

Changed

• RAG Content Extraction - Replace readability-lxml with trafilatura
  • Trafilatura is designed specifically for web content extraction
  • Better boilerplate removal (navigation, ads, subscription boxes, footers)
  • Improved extraction on diverse page layouts
  • Rescues pages where readability-lxml returned minimal content
  • Cleaner output for RAG indexing with less noise
  • Index version bumped to 1.2.0-trafilatura (forces re-index on upgrade)

v0.11.1

Fixed

• WebUI Discovery - Fix open-webui executable not found when using uv
  • Now checks both .venv/bin/open-webui (uv) and venv/bin/open-webui (pip)
  • Updated error message to suggest uv sync --extra webui

v0.11.0

Added

• Request Hook - Configurable callback for debugging/logging LLM requests
  • New REQUEST_HOOK config option: Callable[[str, dict], None] | None
  • Called with (backend_name, payload) before each request to Ollama or LM Studio
  • Enables request logging, debugging, and monitoring without monkey-patching

Example usage

def log_request(backend: str, payload: dict):
    print(f"[{backend}] {json.dumps(payload)}")

config.REQUEST_HOOK = log_request

v0.10.0 - Periodic RAG Index Updates

Periodic RAG Index Updates

This release adds background sitemap polling for automatic index updates in long-running applications.

Added

• Periodic RAG Index Updates - Background thread for automatic sitemap polling

  • Configurable poll interval via periodic_update_interval_hours
  • Automatic sitemap change detection comparing lastmod timestamps
  • Incremental FAISS updates - new chunks added without full rebuild
  • Tombstoning for removed/updated URLs
  • Configurable batch size via update_batch_size
  • Auto-rebuild when tombstone ratio exceeds threshold
  • Pause/resume support for user request priority
  • Status API via get_updater_status() and force_update_check()

• Sitemap Change Detection - Efficient change detection for incremental updates

• Incremental Index Updates - Apply sitemap changes without full rebuild

Fixed

• Brotli decompression errors (exclude from Accept-Encoding)
• Page cache hash consistency (MD5 -> SHA256)
• Periodic updater startup after crawl_and_index()

See CHANGELOG.md for full details.

v0.9.4

Added

• Sitemap Caching - Cache sub-sitemap URLs with lastmod-based invalidation
  • Sub-sitemaps are cached in sitemap_cache.json with their lastmod dates and URLs
  • On subsequent crawls, only sub-sitemaps with changed lastmod are refetched
  • Significantly speeds up crawling sites with many sub-sitemaps (e.g., 17 sitemaps → only changed ones fetched)
  • Progress bar shows "(cached)" or "(fetching)" status for each sub-sitemap
  • Cache is automatically updated when sub-sitemaps change

Full Changelog: v0.9.3...v0.9.4

v0.9.3

Fixed

• RAG Sitemap URL Ordering - Global sort by lastmod before applying max_pages limit
  • Previously, URLs were sorted within each sub-sitemap but not globally
  • When max_pages limit was applied, older content could be included instead of newest
  • Now all URLs are sorted by lastmod (newest first) before limiting
  • Ensures max_pages always returns the most recently modified content

Added

• HTTP Response Status Summary - New table showing crawl response distribution
  • Displays status code, description, count, and percentage for each response type
  • Tracks robots.txt blocks, network errors, and all HTTP status codes
  • Printed to stderr after fetch phase completes
  • Helps diagnose crawl issues (404s, rate limiting, etc.)

Changed

• Crawler fetch_page() Return Type - Now returns (url, html, status_code) tuple
  • Previously returned (url, html) or None
  • Now returns status code even on HTTP errors for tracking
  • Returns None only when blocked by robots.txt (no request made)
  • Network errors use status code 0

Full Changelog: v0.9.2...v0.9.3

v0.9.2

Fixed

• Malformed Tool Token Detection - Detect and retry when models output raw internal tokens
  • Some models (especially Hermes/ChatML-style) output raw tokens like <|start|>assistant<|channel|>... instead of proper responses
  • New _contains_malformed_tool_tokens() method detects these patterns
  • Final response generation now retries once with a stern instruction if malformed tokens detected
  • Falls back to error response if retry also fails
  • Prevents garbage output from reaching users

v0.9.0 - Contextual Retrieval & Configurable Embeddings

Added

• Contextual Retrieval - Anthropic's approach for ~40-50% better retrieval accuracy

  • LLM-generated context prepended to each chunk before embedding
  • Background processing mode - index usable immediately while contexts generate
  • Progress bar with tqdm for context generation
  • Pause/resume support - yields to user requests during background processing
  • Graceful shutdown with stop_background_contextualization()
  • Resumable - progress saved every 50 chunks, can interrupt and resume
  • Uses same backend (LM Studio/Ollama) as main server
  • Reference: https://www.anthropic.com/news/contextual-retrieval

• Configurable Embedding Models - Speed vs quality tradeoff

  • sentence-transformers/all-MiniLM-L6-v2: Fast (22M params), good quality (default)
  • BAAI/bge-base-en-v1.5: Medium (110M params), better quality
  • BAAI/bge-large-en-v1.5: Slow (335M params), best quality
  • Hot-swap without re-crawling - automatically detects model change and rebuilds embeddings only
  • New rebuild_embeddings() method for explicit embedding rebuild

• Progress Bars for RAG Operations

  • tqdm progress during embedding generation
  • Progress bar for contextual retrieval with failed count display
  • Crawling progress with tqdm (sitemap, recursive, fetching)

• LLMServer RAG Integration

  • New optional rag_index parameter for automatic pause/resume during requests
  • Background RAG processing pauses when user request starts, resumes when complete
  • Ensures user requests get priority access to LLM backend

Changed

• DocSearchIndex now accepts optional server_config parameter for contextual retrieval backend settings
• ChunkContextualizer supports both LM Studio (OpenAI API) and Ollama backends
• FAISS index building now shows progress bar during embedding generation

Fixed

• stop_background_contextualization() now silently exits if no background thread running

v0.8.3

Per-Request Model Override

Requests can now specify a different backend model via the model field in the request payload.

Added

• Per-Request Model Override - Requests can now specify a different backend model via the model field
  • If request model differs from server's model_name, temporarily overrides BACKEND_MODEL for that request
  • Enables passthrough to LM Studio/Ollama for model selection at request time
  • Original model is restored after request completes (even on errors)
  • Model name now logged in request_started event for debugging

Example

# Uses qwen2.5:7b for this request only, regardless of server's configured model
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:7b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Full Changelog: v0.8.2...v0.8.3