fix(api): engines for robots and scrape + reduced sitemap limit

[delong3]

Summary by cubic

Improve robots.txt and sitemap scraping reliability with smarter engine fallbacks and caching. Reduce sitemap crawl size to cut load, and add clearer logging when proxy inadequacy triggers stealth mode.

• Refactors
  • Force index engine when available; add final chrome-cdp (+ stealth) fallbacks for robots and sitemaps.
  • Cache robots.txt for 24h and increase timeout to 8s.
  • Reduce sitemap limit from 100 to 25; extend sitemap max age to 7 days.
  • Log status code, content length, and checks when adding stealthProxy.

^{Written for commit 2a46cf2. Summary will update automatically on new commits.}

[cubic-dev-ai]

No issues found across 4 files