Charge AI training crawlers for access (x402 gateway) - #545
Merged
Conversation
65% of last week's hits were training crawlers, mostly Meta and Applebot. They copy pages into a corpus and send nobody back, so from now on they pay: GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider, Applebot-Extended and friends get 402 Payment Required with an x402 offer ($1 buys a day, USDC settled by CoinPay) or the sales page at /crawl, and a paid pass in x-crawl-pass lets them through. People, Googlebot and the retrieval crawlers behind AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot...) are untouched. - src/lib/crawl-gateway.ts builds the gateway from getAppUrl(), COINPAY_X402_KEY and CRAWL_PAY_TO; proxy.ts runs it before the polling throttle and the Supabase session refresh. API routes stay covered on purpose. The matcher now also skips fonts, scripts, css and media. - robots.txt is a route handler generated from the gateway's lists (training crawlers Disallow: / with Allow: /crawl, retrieval crawlers named as welcome, the old private paths kept in every group); the static public/robots.txt is deleted because it would shadow the route. - public/ai.txt now says Training: disallow and points at /crawl. - Tests cover the robots output, the gate's 402/pass-through decisions and the proxy composition order. Without the two env vars the gateway still answers training crawlers with 402, just with an empty offer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC
ThreatCrush Security Scan45 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 8 | LOW: 36
Snippets are redacted; ThreatCrush never prints matched credential material. |
Contributor
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
ralyodio
marked this pull request as ready for review
September 5, 2026 16:31
ralyodio
added a commit
that referenced
this pull request
Sep 5, 2026
With the app under src/app, Next 16 only looks for src/proxy.ts (or src/middleware.ts); a proxy.ts at the repository root is silently ignored and `next build` lists no "ƒ Proxy (Middleware)" entry. That is why #545 deployed and changed nothing: GPTBot still got 200 on / and /crawl was a 404 while the robots.txt route, a plain route handler, went live. It also means the TRACE block, the polling throttle, the Supabase session refresh with its /dashboard login redirect, and the ref cookie in that file have never run in production. git mv, the one relative import in the test switched to @/proxy, nothing else changed. The build now lists "ƒ Proxy (Middleware)", and against a local `next start`: GPTBot on / is 402 with the x402 offer, /crawl is 402 (JSON for the crawler, the HTML sales page for a browser), Chrome on / is 200, and /dashboard without a session is a 307 to /login. Claude-Session: https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes, for whom
402 Payment Requiredwith an x402 offer (JSON), or the sales page if they ask for HTML;robots.txtsaysDisallow: /+Allow: /crawl/crawlcoinpay x402 payor any x402 clientx-crawl-passLast week 65% of hits were training crawlers (Meta and Applebot). This is
@profullstack/x402-gateway0.1.0, settled by CoinPay.Files
src/lib/crawl-gateway.tsbuilds the gateway (site URL fromgetAppUrl(),siteName: "ugig", bulk-deal contactsupport@ugig.net) and exportsgate.proxy.tsrunsgate(request)first and returns its answer if there is one; otherwise everything works as before (TRACE block, polling throttle, Supabase session refresh,refcookie)./api/routes stay covered on purpose: a training crawler hitting the API is charged like everywhere else. The matcher additionally skips fonts, scripts, css, source maps and media by extension.src/app/robots.txt/route.tsreplaces the staticpublic/robots.txt(deleted: a static file would shadow the route) withrobotsRoute(gateway, { disallow: [...] }), so robots.txt and the gate are generated from one list; the previous private paths (/api/,/dashboard/,/settings/,/conversations/,/profile/edit) are repeated in every named group, since a crawler that finds its own name ignoresUser-agent: *.public/ai.txt:Training: disallow(wasallow),Retention/Commercial-Uselikewise, with a comment pointing at/crawl.crawlstays in Capabilities, Contact unchanged.src/lib/crawl-gateway.test.ts: robots output refuses GPTBot and meta-externalagent withDisallow: /, names OAI-SearchBot withAllow: /, keepsDisallow: /dashboard/; the gate 402s ameta-externalagentrequest to/gigs/anythingand to/api/gigs, serves HTML onAccept: text/html, lets crawlers readrobots.txt, passes Chrome/Googlebot/OAI-SearchBot/Claude-SearchBot/PerplexityBot through; the proxy answers a crawler beforeupdateSessionruns and hands a person's request on..env.exampledocuments the two new variables.Deployment needs two env vars
COINPAY_X402_KEYcp_live_…, from the business's API Keys tab, withpayments:create). The legacy business key is refused by CoinPay's x402 routes.CRAWL_PAY_TOUntil they are set, training crawlers still get 402, but with an empty offer: nothing is sold, nothing is given away. Set both on the Railway service before or after merging; the behaviour switches on at the next deploy. The pass signing secret defaults to the CoinPay key, so rotating the key invalidates outstanding passes (at most a day's worth).
Checks
pnpm type-check: cleanpnpm lint: 0 errors (40 pre-existing warnings, none in the changed files)pnpm test:run: 215 files, 2044 tests passedpnpm@9 install --frozen-lockfilefrom scratch: passes (lockfile diff is the single new entry)pnpm buildnot runWorth knowing
Applebot(Siri/Spotlight search) stays free; onlyApplebot-Extended(training) is charged. Same split for every operator pair.🤖 Generated with Claude Code
https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC