Improve scraper resilience and patch dependabot security alerts#16
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves scraping reliability in the operator by truncating oversized HTTP responses (instead of failing) and improves browser-scraper error signaling by distinguishing “browser unavailable” outages (503) from other render failures. It also updates dependencies (notably hono) and adds pnpm overrides to address Dependabot security alerts.
Changes:
- Truncate native fetch response bodies at 2MB and propagate a
truncatedflag throughscrapeUrl. - Add browser error classification to return HTTP 503 for likely Cloudflare browser-worker outages.
- Patch dependency alerts via
honobump andpnpm.overrides(plus lockfile updates).
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
apps/operator/src/services/scrape.ts |
Implements body-size truncation (2MB) and propagates truncated into scrapeUrl results. |
apps/operator/src/services/scrape.test.ts |
Updates tests to assert truncation behavior instead of failure on >2MB bodies. |
apps/browser-scraper/src/services/playwright.ts |
Adds classifyBrowserError to separate “browser unavailable” from internal failures. |
apps/browser-scraper/src/services/playwright.test.ts |
Adds unit tests for classifyBrowserError. |
apps/browser-scraper/src/index.ts |
Returns 503 for “browser unavailable” errors; keeps 500 for other unexpected failures. |
package.json |
Adds pnpm.overrides for vulnerable transitive dependencies. |
apps/operator/package.json |
Bumps hono from 4.12.12 to 4.12.18. |
pnpm-lock.yaml |
Captures dependency bumps and override-resolved versions. |
.claude/settings.json |
Adjusts the gh api allowlist pattern syntax. |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1f55b8f328
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
What
How to test
pnpm typecheck
pnpm lint
pnpm test
Security review
Closes 15 Dependabot alerts (hono, esbuild, postcss, brace-expansion); 3 vite alerts intentionally unresolved.