(feat/search-index) Separate service

[nickscamara]

Summary by cubic

Moved search indexing to a standalone service and integrated a new HTTP client. The API now queries the search service, and the scraper sends documents to it asynchronously.

• New Features

  • Added SearchIndexClient (HTTP) with search, index, stats, and health endpoints.
  • Updated v2 search controller to call the service and return its SearchResponse.
  • Scraper transformer now sends documents to the service via indexDocumentIfEnabled.
  • Added a health check for the search service in the indexing worker.

• Migration

  • Set SEARCH_SERVICE_URL and ENABLE_SEARCH_INDEX=true; optionally set SEARCH_SERVICE_API_SECRET.
  • Remove old SEARCH_INDEX_SUPABASE_* env reliance; worker no longer processes the search-index queue.
  • If not configured, the search endpoint returns 503 and indexing is skipped (non-blocking).

[nickscamara]

@cubic review

[cubic-dev-ai]

@cubic review

@nickscamara I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

No issues found across 4 files