Detect Cloudflare challenge on 429

[simonsmallchua]

Summary

• Widen DetectWAF cf-mitigated check to fire on any non-200 status (was: 403/202 only), so the 429 + Cf-Mitigated: challenge response shape Cloudflare uses for Super Bot Fight Mode managed challenges is now classified correctly as a WAF block instead of a generic "Too Many Requests".
• Add a statusLabel case for 429 so the recorded reason reads cf-mitigated header present on 429 rather than the verbose Too Many Requests.
• Add a regression test (cloudflare — cf-mitigated challenge on 429) covering the failing shape captured against hinu.co and www.milkcan.com.au today.

Why

Today four jobs against two Shopify storefronts (hinu.co, www.milkcan.com.au) failed after one task each. The captured request_diagnostics for the failed tasks show:

HTTP/2 429
Cf-Mitigated: challenge
Server: cloudflare
Content-Length: 9246          ← CF challenge HTML

That is Cloudflare serving a managed bot challenge, not a true rate limit. DetectWAF already has a cf-mitigated branch that would have set domains.waf_blocked = true and surfaced a clear error, but it was gated behind isBlockingStatus which only matched 403 and 202 — so the verdict silently no-op'd, the executor burnt three retries, and the user saw a misleading "Too Many Requests" failure.

The doc comment on DetectWAF already described the intent as "cf-mitigated header set on a non-200 response"; this PR aligns the implementation with the documented behaviour.

Test plan

• go test ./internal/crawler/... — all WAF cases pass including the new 429 case and the existing "must NOT trip on 200" negative case.
• go vet ./internal/crawler/...
• gofmt -l — clean.
• Trigger a fresh job against a CF-challenged Shopify storefront in staging and confirm:
  • the job moves to failed after the first task, not after 3 retries
  • domains.waf_blocked flips to true with waf_vendor = cloudflare
  • the surfaced error message names Cloudflare rather than "Too Many Requests"

---

^{Need help on this PR? Tag @codesmith with what you need.}

• Let Codesmith autofix CI failures and bot reviews

Summary by CodeRabbit

• Bug Fixes

  • Improved Cloudflare WAF detection to recognize managed challenges and 429 rate-limit responses as blocked.

• Tests

  • Added test coverage verifying detection of Cloudflare-managed 429 challenge responses.

• Documentation

  • Updated changelog to reflect the WAF detection fix and its impact on job classification and recording.

[supabase]

Updates to Preview Branch (fix-waf-detect-cf-429) ↗︎

Deployments	Status	Updated
Database	✅	Mon, 11 May 2026 11:50:51 UTC
Services	✅	Mon, 11 May 2026 11:50:51 UTC
APIs	✅	Mon, 11 May 2026 11:50:51 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks	Status	Updated
Configurations	✅	Mon, 11 May 2026 11:50:53 UTC
Migrations	✅	Mon, 11 May 2026 11:50:55 UTC
Seeding	✅	Mon, 11 May 2026 11:50:56 UTC
Edge Functions	✅	Mon, 11 May 2026 11:50:56 UTC

---

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 87871ad1-3f4f-43df-886d-e3403430f9f7

📥 Commits

Reviewing files that changed from the base of the PR and between a180f67 and b5efe52.

📒 Files selected for processing (1)

• CHANGELOG.md

✅ Files skipped from review due to trivial changes (1)

• CHANGELOG.md

---

📝 Walkthrough

Walkthrough

The PR extends Cloudflare WAF detection to recognize HTTP 429 (TooManyRequests) responses as managed challenges when they carry a Cf-Mitigated header. The detection logic is broadened, status labeling for 429 is added, documentation and changelog are updated, and a test case validates the new behavior.

Changes

Cloudflare WAF Detection for 429 Managed Challenges

Layer / File(s)	Summary
Cloudflare Detection Logic internal/crawler/waf.go	DetectWAF condition changed to trigger Cloudflare WAF verdict whenever Cf-Mitigated header is non-empty and response status is not 200 (previously required specific blocking statuses).
HTTP Status Labeling internal/crawler/waf.go	statusLabel function extended to return "429" for http.StatusTooManyRequests so Cloudflare reason strings can label 429 responses.
Documentation & Comments internal/crawler/waf.go	DetectWAF documentation updated to clarify that Cloudflare fingerprinting now includes 429 managed-challenge responses with Cf-Mitigated headers.
Test Coverage internal/crawler/waf_test.go	New TestDetectWAF table-driven test case covering Cloudflare 429 responses with Cf-Mitigated: challenge header and challenge body, asserting WAFVendorCloudflare detection and correct reason prefix.
Changelog CHANGELOG.md	Updated ## [Unreleased] to document the fix recognizing Cloudflare managed challenges served as HTTP 429 with Cf-Mitigated: challenge.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• Good-Native/hover#368: Also modifies internal/crawler/waf.go WAF detection logic (Akamai cookie detection), related by touching the same detection area.

Poem

A rabbit hops through logic clean,
Where 429 now stands as WAF seen,
Cloudflare's mitigated dance,
🐰 Gives challenges their rightful glance.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Detect Cloudflare challenge on 429' precisely captures the main change: broadening WAF detection to recognize Cloudflare managed challenges served as HTTP 429 responses, which is the core functionality improvement across all modified files.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-waf-detect-cf-429

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[github-actions]

Release Versions

App patch: v0.34.8 → v0.34.9

Changelog

Fixed

• WAF detection now recognises Cloudflare managed challenges served as HTTP 429
  with Cf-Mitigated: challenge. Previously the verdict was gated behind status
  403 or 202 only, so CF challenge responses (observed against CF-fronted
  Shopify storefronts with Super Bot Fight Mode enabled) were misclassified as
  plain "Too Many Requests" and jobs burnt three retries before failing with a
  misleading error. Jobs now fail fast and stamp domains.waf_blocked = true
  with waf_vendor = cloudflare.

[github-actions]

🐝 Review App Deployed

Homepage: https://hover-pr-382.fly.dev
Dashboard: https://hover-pr-382.fly.dev/dashboard