Investigate domains with high/total failure rates

[simonsmallchua]

Summary

Several domains show consistently high failure rates across multiple independent job runs. These are distinct from crash-related failures (which affect all domains equally) — these domains fail repeatedly even under normal operating conditions.

Suspect domains

Near-total blocks (0 or ~0 completed)

Domain	Runs	Pattern
glossier.com	3 runs	0 completed each, 14K–54K tasks
kmart.com.au	2 runs	1 completed each, 127K tasks
clerk.com	1 run	2,717/2,719 failed
openai.com	1 run	18/1,088 completed, 60s adaptive delay

Consistently high failure rates

Domain	Runs	Pattern
smashingmagazine.com	3 runs	0/7785, 498/13825, ~505/36551
creativebloq.com	3 runs	Heavy failures each run
sitepoint.com	2 runs	~30–33% failure rate
changelog.com	1 run	9,326/22,733 failed (41%)
sidebar.io	1 run	2,696/7,899 failed (34%)
render.com	1 run	559/10,649 failed
fly.io	2 runs	1,919/8,236 and 53/1,990 failed

Likely causes to investigate

• Bot protection / Cloudflare challenge pages (glossier, kmart, smashingmagazine)
• JS-rendered content the crawler can't parse
• Aggressive rate limiting causing task timeouts
• Redirect loops or auth walls on key URL patterns
• Invalid/unexpected content types being enqueued

Out of scope

Domains with 0 completed that only failed once during the 2026-04-10 system incident are excluded — those are crash artifacts, not domain issues.

[simonsmallchua]

Scope update — partial mitigation since 2026-04-10, but root causes still open.

Mitigated:

• Non-blocking domain rate-limit acquire (033295d9)
• Scheduler fairness + per-domain rate-limit gating (Improve scheduler fairness #328, 5576174e, 501d770a)
• Redis-backed DomainPacer with adaptive delay state (55874820, f7d1ce57, eaaa2121)
• HTTP/2 disabled to avoid misbehaving WAF servers (15d146f3)
• Throughput / backpressure recovery (Fix crawler throughput under load #325)
• Crawler phase diagnostics (Fix crawler warm recursion and telemetry #331)

Still open:

• No Cloudflare / bot-challenge detection or bypass
• No JS rendering (no headless browser)
• Single user-agent (HoverBot/1.0); no rotation
• No per-domain allow/denylist

Domains in the original report (glossier.com, kmart.com.au, openai.com, smashingmagazine.com, etc.) will still fail today. Keeping open as the tracker for the unresolved root-cause work.

[simonsmallchua]

Investigation complete — superseded by #365 which carries the action plan.

Summary of what the probe surfaced:

• Original failing-domain set has largely self-cleared. smashingmagazine, changelog, fly.io, render.com, clerk.com all return HTTP 200 to HoverBot/1.0 today, likely thanks to the HTTP/2 disable (15d146f3) and DomainPacer changes.
• Remaining failure modes concentrate in five categories: WAF challenges (Cloudflare/Imperva/DataDome/Akamai), Shopify URL fan-out, UA-string blocks (the word "bot" tripping naive filters), missing failure-class granularity, and JS-rendered SPAs.
• Glossier-class slowness traced to Shopify platform throttling + locale fan-out, not bot protection.
• kmart.com.au reclassified — it returns 200 + 2.3 MB to HoverBot today; original "127K tasks, 1 completed" was likely a different failure mode (sitemap fan-out or per-task timeout), not UA blocking.

See #365 for the prioritised action plan. Closing.