Skip to content

Make /api/search honest: explain empty results + clear config errors - #16

Merged
stretchcloud merged 1 commit into
mainfrom
fix/search-provider
Jul 22, 2026
Merged

Make /api/search honest: explain empty results + clear config errors#16
stretchcloud merged 1 commit into
mainfrom
fix/search-provider

Conversation

@stretchcloud

Copy link
Copy Markdown
Owner

The real problem

/api/search returned {success:true, count:0, results:[]} for every query — with no indication of why. You couldn't tell "no hits" from "provider blocked" from "provider not configured". That silent empty is what wasted debugging time.

Root cause of the empty results in production: the default keyless provider (DuckDuckGo's HTML endpoint) anti-bot-challenges datacenter / server IPs — HTTP 202. I confirmed this from two independent datacenter IPs (the VPS and this sandbox), on both html. and lite. endpoints. It's a DuckDuckGo policy, not a parser bug — the parser is fine, DDG just refuses to serve hosted IPs. So it genuinely cannot be "fixed" into working keyless from a server.

What this PR does — make it transparent, not pretend

  • search.service: new runSearch() returns { results, provider, reason? }. Each provider fetcher now reports an actionable reason (DDG 202 challenge, request failure…). searchWeb() is kept as a results-only wrapper, so existing callers/tests are untouched. New emptyResultHint(provider) gives the provider-specific explanation and the real fix.
  • Route: the response now includes provider and, when empty, a note explaining why — e.g. "DuckDuckGo returned an anti-bot challenge (HTTP 202)… set SERPER_API_KEY (free at serper.dev) or SEARXNG_URL." Missing provider config now returns a 400 with the actionable message instead of an opaque 500.

Reliable search still needs config, but it's free and already supported: SERPER_API_KEY (Serper.dev free tier, auto-selected when present) or a self-hosted SEARXNG_URL.

Verification (Docker, end-to-end)

Case Result
default provider, no config 200 · provider:duckduckgo · count:0 · note explaining the 202 block + fix
provider:searxng, no SEARXNG_URL 400 with the actionable message (was an opaque 500)
SEARXNG_URL configured (mock) 200 · provider:searxng · count:2 real results flow through

tsc ✅ · eslint ✅ · 204 tests ✅ (+3 new) · openapi:check

Note: keyless DuckDuckGo is left as the best-effort default (it works from residential IPs), but the response now makes its server-side limitation explicit rather than silent.

🤖 Generated with Claude Code

/api/search returned {success:true, count:0, results:[]} with no indication
of why — you couldn't tell "no hits" from "keyless provider blocked" from
"provider not configured". The root cause of the empty results in production:
DuckDuckGo's keyless endpoint anti-bot-challenges datacenter/server IPs (HTTP
202), confirmed from two independent datacenter IPs. That's a DDG policy, not
a parser bug — it can't be fixed into working keyless from a hosted API.

So make the endpoint transparent rather than pretend:

- search.service: new `runSearch()` returns { results, provider, reason? }; the
  per-provider fetchers now report an actionable `reason` (DDG 202 challenge,
  request failure, …). `searchWeb()` kept as a results-only wrapper so existing
  callers/tests are unchanged. New `emptyResultHint(provider)` gives the
  provider-specific explanation + the real fix.
- search route: response now includes `provider` and, when empty, a `note`
  saying why (e.g. the DDG datacenter block, pointing at SERPER_API_KEY /
  SEARXNG_URL). Missing provider config now returns a 400 with the actionable
  message instead of an opaque 500.

Reliable search still requires config — but it's free: SERPER_API_KEY (free
tier at serper.dev, auto-selected when present) or a self-hosted SEARXNG_URL.
Both already worked; now the endpoint tells you so.

Docker end-to-end verified: default provider -> 200 + note explaining the 202
block; provider=searxng w/o URL -> 400 with the fix; SEARXNG_URL configured ->
200 with real results (provider:searxng, count:2). tsc, eslint, 204 tests
(+3), openapi:check all pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@stretchcloud
stretchcloud merged commit 7beef0b into main Jul 22, 2026
1 check passed
@stretchcloud
stretchcloud deleted the fix/search-provider branch July 22, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant