feat(sdk): sanctionedFetch — the web charges your agents now; Sanction decides what they pay#183
Merged
Merged
Conversation
…n decides what they pay
Cloudflare's July 1 launch (pay-per-crawl + Monetization Gateway) meters the
open web for agents: paid URLs answer 402 with a crawler-price quote, settled
over x402, and agent crawlers are blocked by default on ad pages from Sept 15.
The protocol's buyer-side control is a static crawler-max-price header — a
cap, not governance.
sanctionedFetch wraps any fetch (pass your Web Bot Auth signing fetch —
identity stays upstream) and turns every quote into a governed spend decision:
- 402 + parseable crawler-price → POST /authorize with merchant = hostname,
category = content-access (overridable), tags {channel: pay-per-crawl, url}
- approved → retry echoing the site's own quote verbatim into
crawler-exact-price (exact-match semantics, never a max)
- escalated/denied → SanctionCrawlBlocked with status/code/requestId/price —
a planning outcome the crawler branches on, not a crash
- non-402s and 402s without a usable quote pass through untouched
Auto-approve bands, per-txn caps, daily/monthly/pooled budgets, human
escalation, freeze, and the audit trail all apply unchanged — a crawl quote
is just spend with the site as the merchant.
Also: AuthorizeInput gains `tags` (the REST API already accepted them; the
SDK now sends them). docs/PAY-PER-CRAWL.md guide (wire flow, policy recipe,
honest boundary: Sanction governs the decision; the billing relationship and
Web Bot Auth keys are yours/Cloudflare's) registered at /docs/pay-per-crawl.
Changelog entry; roadmap mandate-authority note marks slice 1 shipped;
backlog captures the arc (reconciliation, crawl policy pack, generic x402,
seller-side GTM question).
11 new adapter tests (parse forms, pass-throughs, approve/escalate/deny,
verbatim echo, tags/category/onDecision). Also fixes a pre-existing tsc
error on main in adapters.test.ts (untyped execute arg) so the gate runs
clean. 857 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UEw1RdHk5bmphKxr6N5mJY
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📝 WalkthroughWalkthroughThis change adds a pay-per-crawl SDK adapter that authorizes HTTP 402 crawler prices, retries approved requests, reports decisions, and exposes structured blocking errors. It also adds attribution tags, tests, a public guide, and related changelog, roadmap, backlog, and navigation updates. ChangesPay-per-crawl authorization and documentation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Crawler
participant sanctionedFetch
participant SanctionClient
participant WebOrigin
Crawler->>sanctionedFetch: Request URL
sanctionedFetch->>WebOrigin: Initial fetch
WebOrigin-->>sanctionedFetch: 402 with crawler-price
sanctionedFetch->>SanctionClient: Authorize crawl price
SanctionClient-->>sanctionedFetch: Decision
sanctionedFetch->>WebOrigin: Retry with crawler-exact-price
WebOrigin-->>Crawler: Approved response
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
sdk/src/adapters.ts (1)
231-237: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winNo idempotency key on the pay-per-crawl authorize call.
client.authorize()is called withoutidempotencyKey.SanctionClient.authorize/finishLocaluseidempotencyKey(ordecision.requestId) to make offline-queue replay safe, implying the contract is designed around idempotent retries. If the caller retries the wholesanctionedFetchcall for the same URL+quote (e.g. after a transient network failure on the paid retry step), this issues a brand-new authorization and double-counts againstdailySpentUsd/budgets for a single logical crawl.Consider deriving a stable key (e.g. hash of
host + url + priceUsd) and passing it through:const decision = await client.authorize({ action: "purchase", amountUsd: priceUsd, merchant: host, category: opts.category ?? "content-access", tags: { channel: "pay-per-crawl", url: url.slice(0, 80), ...opts.tags }, + idempotencyKey: `crawl:${host}:${url}:${priceUsd}`, })Please confirm
AuthorizeInput(sdk/src/types.ts) actually exposesidempotencyKeyas shown in theSanctionClientgraph context before applying.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@sdk/src/adapters.ts` around lines 231 - 237, Confirm that AuthorizeInput exposes idempotencyKey, then update the pay-per-crawl authorize call in sanctionedFetch to derive and pass a stable key from host, URL, and priceUsd. Ensure repeated retries for the same logical crawl reuse that key while preserving the existing authorization fields.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@sdk/src/adapters.ts`:
- Line 236: Update the attribution tags construction in the pay-per-crawl flow
to remove the URL’s search/query portion before truncating and storing it in the
audit tags. Preserve the existing 80-character limit and opts.tags merge
behavior while ensuring query parameters cannot reach the audit trail.
- Around line 210-244: Update sanctionedFetch to initialize the retry headers in
a way that preserves headers from a Request input, merging input.headers with
init?.headers before setting crawler-exact-price. Keep explicit init headers’
existing behavior and pass the merged headers to the final baseFetch retry.
In `@sdk/src/client.ts`:
- Line 100: Update syncOfflineDecisions() to include tags: item.tags in each
replay request body, preserving channel, URL, and caller attribution tags during
offline decision synchronization. Add or extend a test covering offline replay
to verify the tags are forwarded unchanged.
---
Nitpick comments:
In `@sdk/src/adapters.ts`:
- Around line 231-237: Confirm that AuthorizeInput exposes idempotencyKey, then
update the pay-per-crawl authorize call in sanctionedFetch to derive and pass a
stable key from host, URL, and priceUsd. Ensure repeated retries for the same
logical crawl reuse that key while preserving the existing authorization fields.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5ef76418-73e7-4d82-bc22-1cf961733fea
📒 Files selected for processing (11)
app/docs/page.tsxdocs/BACKLOG.mddocs/PAY-PER-CRAWL.mdlib/changelog.tslib/docs.tslib/roadmap.tssdk/src/adapters.test.tssdk/src/adapters.tssdk/src/client.tssdk/src/index.tssdk/src/types.ts
Comment on lines
+210
to
+244
| export function sanctionedFetch( | ||
| client: SanctionClient, | ||
| baseFetch: FetchLike = fetch, | ||
| opts: SanctionedFetchOptions = {}, | ||
| ): FetchLike { | ||
| return async (input, init) => { | ||
| const res = await baseFetch(input, init) | ||
| if (res.status !== 402) return res | ||
|
|
||
| const quote = res.headers.get("crawler-price") | ||
| const priceUsd = parseCrawlPrice(quote) | ||
| if (priceUsd === null) return res // 402 for some other reason — not ours to pay | ||
|
|
||
| const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url | ||
| let host: string | ||
| try { | ||
| host = new URL(url).hostname | ||
| } catch { | ||
| return res // unparseable URL — refuse to authorize what we can't attribute | ||
| } | ||
|
|
||
| const decision = await client.authorize({ | ||
| action: "purchase", | ||
| amountUsd: priceUsd, | ||
| merchant: host, | ||
| category: opts.category ?? "content-access", | ||
| tags: { channel: "pay-per-crawl", url: url.slice(0, 80), ...opts.tags }, | ||
| }) | ||
| opts.onDecision?.(decision, url, priceUsd) | ||
| if (decision.status !== "approved") throw new SanctionCrawlBlocked(url, priceUsd, decision) | ||
|
|
||
| const headers = new Headers(init?.headers) | ||
| headers.set("crawler-exact-price", quote as string) | ||
| return baseFetch(input, { ...init, headers }) | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the adapter implementation and nearby docs/comments.
git ls-files sdk/src | sed -n '1,200p'
printf '\n--- adapters.ts excerpt ---\n'
sed -n '180,280p' sdk/src/adapters.ts
printf '\n--- search for sanctionedFetch references ---\n'
rg -n "sanctionedFetch|SanctionedFetchOptions|crawler-exact-price|Web Bot Auth|signed Request|preserve" sdk -S
printf '\n--- relevant docs/file hits ---\n'
rg -n "Request|headers|init.headers|fetch\\(request, init\\)|replace.*header|merge.*header" sdk -SRepository: ericlovold/sanction
Length of output: 9297
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Probe Fetch/Request header semantics in the available runtime without modifying repo files.
node <<'JS'
const urls = ['https://example.com/']
const req = new Request(urls[0], { headers: { 'x-signed': 'yes' } })
const retryHeaders = new Headers(undefined)
retryHeaders.set('crawler-exact-price', '1.23')
const cloned = new Request(req, { headers: retryHeaders })
console.log('original x-signed:', req.headers.get('x-signed'))
console.log('cloned x-signed:', cloned.headers.get('x-signed'))
console.log('cloned crawler-exact-price:', cloned.headers.get('crawler-exact-price'))
console.log('same method:', cloned.method)
JSRepository: ericlovold/sanction
Length of output: 252
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# If Node's global Request isn't available, fall back to a minimal spec-oriented probe using undici-like behavior if present.
node <<'JS'
try {
const req = new Request('https://example.com/', { headers: { 'x-signed': 'yes' } })
const retryHeaders = new Headers()
retryHeaders.set('crawler-exact-price', '1.23')
const cloned = new Request(req, { headers: retryHeaders })
console.log(JSON.stringify({
original: req.headers.get('x-signed'),
clonedSigned: cloned.headers.get('x-signed'),
clonedPrice: cloned.headers.get('crawler-exact-price'),
method: cloned.method,
}, null, 2))
} catch (e) {
console.error(e && e.stack || e)
process.exit(1)
}
JSRepository: ericlovold/sanction
Length of output: 249
Preserve Request headers on retry.
new Headers(init?.headers) ignores headers already carried by a Request, so the retry can drop signing/auth headers and send only crawler-exact-price. Merge input.headers into the retry headers before calling baseFetch.
Suggested fix
- const headers = new Headers(init?.headers)
+ const headers = new Headers(input instanceof Request ? input.headers : undefined)
+ new Headers(init?.headers).forEach((v, k) => headers.set(k, v))
headers.set("crawler-exact-price", quote as string)
return baseFetch(input, { ...init, headers })📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| export function sanctionedFetch( | |
| client: SanctionClient, | |
| baseFetch: FetchLike = fetch, | |
| opts: SanctionedFetchOptions = {}, | |
| ): FetchLike { | |
| return async (input, init) => { | |
| const res = await baseFetch(input, init) | |
| if (res.status !== 402) return res | |
| const quote = res.headers.get("crawler-price") | |
| const priceUsd = parseCrawlPrice(quote) | |
| if (priceUsd === null) return res // 402 for some other reason — not ours to pay | |
| const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url | |
| let host: string | |
| try { | |
| host = new URL(url).hostname | |
| } catch { | |
| return res // unparseable URL — refuse to authorize what we can't attribute | |
| } | |
| const decision = await client.authorize({ | |
| action: "purchase", | |
| amountUsd: priceUsd, | |
| merchant: host, | |
| category: opts.category ?? "content-access", | |
| tags: { channel: "pay-per-crawl", url: url.slice(0, 80), ...opts.tags }, | |
| }) | |
| opts.onDecision?.(decision, url, priceUsd) | |
| if (decision.status !== "approved") throw new SanctionCrawlBlocked(url, priceUsd, decision) | |
| const headers = new Headers(init?.headers) | |
| headers.set("crawler-exact-price", quote as string) | |
| return baseFetch(input, { ...init, headers }) | |
| } | |
| export function sanctionedFetch( | |
| client: SanctionClient, | |
| baseFetch: FetchLike = fetch, | |
| opts: SanctionedFetchOptions = {}, | |
| ): FetchLike { | |
| return async (input, init) => { | |
| const res = await baseFetch(input, init) | |
| if (res.status !== 402) return res | |
| const quote = res.headers.get("crawler-price") | |
| const priceUsd = parseCrawlPrice(quote) | |
| if (priceUsd === null) return res // 402 for some other reason — not ours to pay | |
| const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url | |
| let host: string | |
| try { | |
| host = new URL(url).hostname | |
| } catch { | |
| return res // unparseable URL — refuse to authorize what we can't attribute | |
| } | |
| const decision = await client.authorize({ | |
| action: "purchase", | |
| amountUsd: priceUsd, | |
| merchant: host, | |
| category: opts.category ?? "content-access", | |
| tags: { channel: "pay-per-crawl", url: url.slice(0, 80), ...opts.tags }, | |
| }) | |
| opts.onDecision?.(decision, url, priceUsd) | |
| if (decision.status !== "approved") throw new SanctionCrawlBlocked(url, priceUsd, decision) | |
| const headers = new Headers(input instanceof Request ? input.headers : undefined) | |
| new Headers(init?.headers).forEach((v, k) => headers.set(k, v)) | |
| headers.set("crawler-exact-price", quote as string) | |
| return baseFetch(input, { ...init, headers }) | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@sdk/src/adapters.ts` around lines 210 - 244, Update sanctionedFetch to
initialize the retry headers in a way that preserves headers from a Request
input, merging input.headers with init?.headers before setting
crawler-exact-price. Keep explicit init headers’ existing behavior and pass the
merged headers to the final baseFetch retry.
| amountUsd: priceUsd, | ||
| merchant: host, | ||
| category: opts.category ?? "content-access", | ||
| tags: { channel: "pay-per-crawl", url: url.slice(0, 80), ...opts.tags }, |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Truncated URL in attribution tags may carry sensitive query params into the audit trail.
url.slice(0, 80) includes the query string verbatim. Pay-per-crawl URLs are usually public content, but any tracking/session tokens in the query string would end up logged in Sanction's audit feed/CSV export. Consider stripping the search/query portion before tagging.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@sdk/src/adapters.ts` at line 236, Update the attribution tags construction in
the pay-per-crawl flow to remove the URL’s search/query portion before
truncating and storing it in the audit tags. Preserve the existing 80-character
limit and opts.tags merge behavior while ensuring query parameters cannot reach
the audit trail.
| merchant: input.merchant, | ||
| category: input.category, | ||
| description: input.description, | ||
| tags: input.tags, |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Preserve tags when replaying offline decisions.
authorize() now forwards input.tags, but syncOfflineDecisions() omits tags from its replay body (Lines [182-188]). Offline pay-per-crawl decisions will therefore lose the channel, URL, and caller attribution tags in the audit record. Add tags: item.tags and cover this path with a test.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@sdk/src/client.ts` at line 100, Update syncOfflineDecisions() to include
tags: item.tags in each replay request body, preserving channel, URL, and caller
attribution tags during offline decision synchronization. Add or extend a test
covering offline replay to verify the tags are forwarded unchanged.
ericlovold
added a commit
that referenced
this pull request
Jul 12, 2026
…#193) Bump 0.6.0 → 0.7.0 and drain the truth surfaces (truthsync folded into the cut). The pack since v0.6.0: pay-per-crawl sanctionedFetch (#183), @sanction/sdk publish-ready (#169), Sanction Local install package (#168, #184), observe mode (#187) + observe digest & subtree rollup (#192), email-lands-on-the-decision (#190). Changelog gaps filled (verified against code, not the draft): - v0.7.0 release header (theme: adoption — watch-first, install, drop-in). - Observe mode entry (OBS-1 + OBS-2): the do-nothing on-ramp — real engine records the would-be outcome, blocks nothing, flip-to-enforce per pool with a revisioned timestamp; org roots read the whole subtree. - Email-on-decision entry (#190): escalation notices deep-link to the exact pending request (approveUrlFor), not the generic inbox. Roadmap rotated: Local install → shipped phrasing; observe-mode adoption added to Now as shipped; arc comment updated. (Demo companies, consulting page, and the skills work are infra/site/tooling — no product-changelog entry, by design.) Diff is package.json + changelog + roadmap only. Isolated branch — does not touch the concurrent MCP-demo work. Claude-Session: https://claude.ai/code/session_01UEw1RdHk5bmphKxr6N5mJY Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why now
Cloudflare's July 1 launch turned the open web into a metered resource for agents: pay-per-crawl + the Monetization Gateway let any site, dataset, API, or MCP tool behind Cloudflare charge per request —
402+ acrawler-pricequote, settled over x402 (the exact protocol in our roadmap's mandate-authority item) — and agent crawlers are blocked by default on ad pages from September 15. Cloudflare built the seller's rail. Nobody has built the buyer's side: the protocol's own control is a staticcrawler-max-priceheader. A cap is not governance.What
sanctionedFetch(client, signingFetch, opts?)— wrap any fetch; every pay-per-crawl quote becomes a governed spend decision:402+ parseablecrawler-price→POST /authorizewith merchant = the site's hostname, categorycontent-access(overridable), tags{channel: "pay-per-crawl", url}— so auto-approve bands, per-transaction caps, daily/monthly/pooled department budgets, human escalation on unusual prices, freeze, and the audit trail all apply unchanged.crawler-exact-price(exact-match semantics, never a max).SanctionCrawlBlockedcarrying status/code/requestId/price — a planning outcome the crawler branches on, not a crash.baseFetch; the payment header must be covered by itssignature-inputcomponents. Stated in code and guide.Plus:
AuthorizeInputgainstags(the REST API accepted them since the attribution work; the SDK now sends them) ·docs/PAY-PER-CRAWL.mdguide (wire flow, crawl-fleet policy recipe, honest boundary — Sanction governs the decision; crawler registration/keys/settlement are between the operator and Cloudflare) live at/docs/pay-per-crawland linked from the docs index · changelog entry · roadmap mandate-authority note marks slice 1 shipped · backlog captures the arc (settlement reconciliation ofcrawler-chargedreceipts vs decisions, crawl policy pack, generic x402 parsing, the seller-side GTM question).Verification
USD 0.01+ tolerant forms, junk/zero/non-USD rejected), non-402 and quote-less-402 pass-throughs with zero authorize calls, approved→verbatim-echo retry with full authorize-body assertion, escalated/denied→typed throw with no paid retry, custom category/tags merge +onDecision.adapters.test.ts(untypedexecutearg, verified present with this diff stashed) — one-line test fix so the gate runs.Wire facts verified against Cloudflare's docs before building: pay-per-crawl announcement · crawl-pages mechanics · new AI traffic defaults
🤖 Generated with Claude Code
https://claude.ai/code/session_01UEw1RdHk5bmphKxr6N5mJY
Generated by Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests