Refuse the OVH VPS fleet and charge spoofed browsers (x402-gateway 0.2.2) - #547
Merged
Conversation
…2.2) The lists only catch crawlers that name themselves. Two on this site do not: an OVH VPS fleet (vps-*.vps.ovh.net, measured 2026-08-28 on rssamplifier) wearing "Chrome/148", and rotations cycling a few Chrome strings across many addresses. 0.2.2 adds the two edge controls for them, switched on here: - denyCidrs: eight OVH /16s answered with a tiny 403 before anything else, read from the edge's own last hop of x-forwarded-for (or x-real-ip), so a client seeding the header cannot step around it. - chargeSpoofedBrowsers: a "Chrome/..." UA without Sec-Fetch-Mode, a header every Chromium sends and no script can strip, is charged like GPTBot. Googlebot's evergreen Chrome string, Bingbot and anything "compatible;" are judged by the lists, never by this. - exempt: a request carrying a Supabase session cookie (sb-<ref>-auth-token, chunked .0/.1; not the pre-sign-in code-verifier) is never charged, whatever the client looks like. Tests cover each: OVH last hop 403, seeded first hop ignored, Chrome without sec-fetch-mode 402, with it passes, session cookie passes, Googlebot evergreen passes, and the cookie matcher's edges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC
ThreatCrush Security Scan45 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 8 | LOW: 36
Snippets are redacted; ThreatCrush never prints matched credential material. |
Contributor
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
@profullstack/x402-gateway0.1.0 → 0.2.2 and its three edge controls switched on insrc/lib/crawl-gateway.ts. The lists from #545 only catch crawlers that name themselves; this handles the ones that do not.51.38/16,54.38/16,141.94/16,145.239/16,149.202/16,151.80/16,57.129/16,213.32/16)Not available from this network.before anything else runs, from the edge's last hop ofx-forwarded-for(orx-real-ip), so a client seeding the header cannot step around itChrome/…user agent with noSec-Fetch-Modeheader (every Chromium since 76 sends it; it is a forbidden header no page script or extension can remove)… compatible; Googlebot/2.1 … Chrome/145 …), Bingbot, anything declaring itselfsb-<project-ref>-auth-token, chunked.0/.1)-auth-token-code-verifiercookie does not count/crawl,robots.txtThe OVH ranges are the VPS fleet (
vps-*.vps.ovh.net) measured 2026-08-28 on rssamplifier, crawling with a spoofedChrome/148string; nobody reads ugig from an OVH VPS. Order inside the gateway: CIDR deny →/crawl→ exempt (session) → lists or spoof check → open paths → pass header → 402.Cookie matching
The app's Supabase middleware (
@profullstack/stack/supabase→@supabase/ssrcreateServerClient) passes nocookieOptions.name, so the session cookie is the library defaultsb-<ref>-auth-token, chunked when large.hasSupabaseSessionmatchessb-*-auth-tokenoptionally followed by.N, then=.Tests (
src/lib/crawl-gateway.test.ts, 18 tests)New: OVH last-hop
x-forwarded-for→ 403; OVH seeded as the first hop → passes (denylist reads the edge's hop); Chrome UA withoutsec-fetch-modeon/gigs/x→ 402; withsec-fetch-mode: navigate→ passes; Chrome withoutsec-fetchbut withsb-xxx-auth-token=(and the chunked form) → passes; the cookie matcher rejects-code-verifierand unrelated cookies; Googlebot evergreen UA → passes. The existing Chrome pass-through and proxy composition cases now sendsec-fetch-mode, since a bare Chrome string is charged on purpose.Checks
pnpm type-check: cleanpnpm lint: 0 errors (40 pre-existing warnings, none in the changed files)pnpm test:run: 215 files, 2051 tests passedpnpm@9 install --frozen-lockfilefrom a wipednode_modules: passes; lockfile diff is the five lines for the version bump (0.2.2 has no dependencies)Worth knowing
Sec-Fetch-*from a real Chrome user would see a 402 on public pages while signed out; signing in exempts them.Sec-Fetch-Modewas chosen oversec-ch-uafor exactly this reason (it survives more proxies), but it is not zero.COINPAY_X402_KEYandCRAWL_PAY_TOstill gate whether the 402 carries a price.🤖 Generated with Claude Code
https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC