fix: invert WebFetch User-Agent strategy to honest-bot-first#303
fix: invert WebFetch User-Agent strategy to honest-bot-first#303anandgupta42 merged 1 commit intomainfrom
Conversation
The fake Chrome UA was being blocked by Cloudflare TLS fingerprint detection, Anubis, and other anti-bot systems (~30% of sites). Inverting to an honest bot UA first with browser UA fallback improves success rate from 70% to 100% (honest UA solo) / 85% to 95% (with retry strategy). Changes: - Default to `altimate-code/1.0` honest bot UA instead of spoofed Chrome UA - Retry with browser UA on 403/406 (WAF/bot-detection codes only) - Remove 404/429 from retry set (semantically incorrect for UA retry) - Cancel response body stream before retry to prevent resource leaks - Move `clearTimeout()` into `try/finally` after body consumption to prevent Slowloris-style hangs on slow-streaming responses - Add tests: 403 retry, 406 retry, no-retry on success, no-retry on 500, both-UAs-fail with call tracking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| let response = await fetch(params.url, { signal, headers: honestHeaders }) | ||
|
|
||
| clearTimeout() | ||
|
|
||
| if (!response.ok) { | ||
| throw new Error(`Request failed with status code: ${response.status}`) | ||
| // Retry with browser UA if the honest UA was rejected | ||
| if (!response.ok && RETRYABLE_STATUSES.has(response.status)) { | ||
| await response.body?.cancel().catch(() => {}) | ||
| response = await fetch(params.url, { signal, headers: browserHeaders }) |
There was a problem hiding this comment.
Bug: If an initial fetch() call throws an error, the finally block is not reached, and clearTimeout() is never called, leaving a timer running unnecessarily.
Severity: MEDIUM
Suggested Fix
The try/finally block should be expanded to wrap all the code that executes after the timer is created with abortAfterAny(), including the initial fetch() calls. This will ensure that clearTimeout() is always called, regardless of whether the fetch operations succeed or fail.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: packages/opencode/src/tool/webfetch.ts#L73-L78
Potential issue: The `try/finally` block that is supposed to guarantee the execution of
`clearTimeout()` only starts after the initial `fetch()` calls. If either of these
`fetch()` operations throws an error, for instance due to a network failure, the
exception will propagate before the `try` block is entered. Consequently, the `finally`
block is never executed, and `clearTimeout()` is not called. This results in the timeout
timer continuing to run for its entire duration (30-120 seconds) even after the
associated fetch operation has failed, causing unnecessary resource consumption.
Did we get this right? 👍 / 👎 to inform future reviews.
1 participant
What does this PR do?
Inverts the WebFetch User-Agent strategy from spoofed-browser-first to honest-bot-first, fixing ~30% fetch failure rate caused by TLS fingerprint mismatch detection.
Key changes:
altimate-code/1.0honest bot UA instead of spoofed Chrome UAclearTimeout()intotry/finallyto prevent Slowloris-style hangsBenchmark results:
Type of change
Issue for this PR
Closes #302
How did you verify your code works?
Checklist
🤖 Generated with Claude Code