Pay per crawl

Pay-per-crawl (the Tollbooth)

Agent402 is two-sided. The main server lets agents buy tools. The agent402-tollbooth package is the inverse: it lets any site charge the AI bots that crawl it.

Put it in front of any site or API: human visitors browse free, while AI crawlers and agents pay per request — in USDC over x402, or for free by solving a proof-of-work. The open, self-hostable answer to Cloudflare's closed pay-per-crawl: no CDN lock-in, no Stripe, no Merchant-of-Record, no signup.

Two ways to run it

Express middleware — humans pass; known AI crawlers get 402:

import express from "express";
import { createTollbooth } from "agent402-tollbooth";

const app = express();
app.use(createTollbooth({ price: "$0.002" }));
app.get("/article", (_req, res) => res.send("…your content…"));
app.listen(3000);

Reverse proxy — wrap any existing site, any language, zero code changes:

TOLLBOOTH_UPSTREAM=https://your-site.com node tollbooth/index.js

curl -A "Mozilla/5.0" localhost:4021/article     # human -> 200, free
curl -A "ClaudeBot/1.0" localhost:4021/article   # bot   -> 402 Payment Required

How it works

• Who pays: by default, requests whose User-Agent matches a known AI/LLM crawler (GPTBot, ClaudeBot, CCBot, PerplexityBot, Bytespider, Google-Extended, …). Classic search indexers (Googlebot, Bingbot) are not charged, so SEO stays free. Override with botUserAgents, or a custom charge(req) predicate.
• Free rail (proof-of-work): works out of the box, no wallet. A crawler solves a single-use, resource-bound sha256 puzzle and retries with X-Pow-Solution: <token>:<nonce> — the same hardened scheme the main server uses (see Paying with Compute).
• Paid rail (x402 USDC): set payTo and supply a verifyX402 hook wired to the standard x402 stack — settlement is reused, not reinvented (see Paying with x402).

Why it exists

The big platforms shipped pay-per-crawl as a closed, fiat, you-must-be-on-our-CDN feature. This is the open, crypto-native, run-it-yourself version, built on the same 402 + proof-of-work machinery as the rest of Agent402. It turns the project into both sides of the x402 economy: agents buy capabilities, and sites charge agents.

Full docs and config table: tollbooth/README.md. MIT licensed.