feat(enrichment): harden Open Graph browser mode against anti-bot pages#2724
Merged
Conversation
Stops the screenshot pipeline from capturing Cloudflare/Akamai challenge pages and from caching HTTP 4xx/5xx bodies as if they were normal HTML. Changes: - Inject realistic UA, Accept-Language, and a single chromium stealth arg (--disable-blink-features=AutomationControlled) on every agent-browser invocation. - Replace the blind 1.5s wait with `wait --load networkidle` (10s cap). - Parse the main document request via `network requests --status 400-599` and throw before the row enters the cache. - Detect challenge pages by signature (title + first 8KB of html); retry once via reload + networkidle, then throw a typed ChallengeBlockedError so the failure surfaces distinctly from generic fetch errors. - Drop the log level for ChallengeBlockedError from warn to info; expected anti-bot signals should not flood on-call dashboards. See docs/superpowers/specs/2026-05-15-og-browser-hardening-design.md for the full design rationale and rollout notes.
SafeDep Report SummaryNo dependency changes detected. Nothing to scan. This report is generated by SafeDep Github App |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
agent-browserinvocation.wait 1500withwait --load networkidle(10s cap), then parse the main document request vianetwork requests --status 400-599so failures throw before the row enters the cache.ChallengeBlockedError. Log challenge-error atinfo; reservewarnfor unexpected fetch faults.Spec:
docs/superpowers/specs/2026-05-15-og-browser-hardening-design.mdTest plan