Skip to content

Charge AI training crawlers for access (x402 gateway) - #141

Merged
ralyodio merged 3 commits into
masterfrom
x402-gateway
Sep 5, 2026
Merged

Charge AI training crawlers for access (x402 gateway)#141
ralyodio merged 3 commits into
masterfrom
x402-gateway

Conversation

@ralyodio

@ralyodio ralyodio commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Wires @profullstack/x402-gateway in front of the middleware: training crawlers get 402 with an x402 offer or the sales page at /crawl; a paid pass opens the site for a day. robots.txt is generated from the same crawler lists. COINPAY_X402_KEY and CRAWL_PAY_TO are already set on the Railway service.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider,
Applebot-Extended) get 402 Payment Required with an x402 offer, or the sales
page at /crawl, and a paid pass opens the site for a day. People, search
engines and retrieval crawlers pass through untouched. robots.txt is now
generated from the same lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

ThreatCrush Security Scan

21 finding(s)

HIGH/CRITICAL: 3 | MEDIUM: 9 | LOW: 9

Severity Rule Location
HIGH secret-private-key plugins/credential-sharing/src/providers/ssh.ts:129
HIGH js-host-header-trust apps/commandboard-web/server.js:27
HIGH js-ssrf-outbound-request apps/pwa/public/sw.js:45
MEDIUM js-unescaped-html-sink apps/commandboard-web/src/main.ts:19
MEDIUM js-unescaped-html-sink apps/logicsrc-web/src/app/[[...slug]]/page.tsx:73
MEDIUM js-unescaped-html-sink apps/logicsrc-web/src/app/layout.tsx:83
MEDIUM js-unescaped-html-sink apps/logicsrc-web/src/app/pricing/page.tsx:46
MEDIUM js-unescaped-html-sink apps/logicsrc-web/src/components/site-shell.tsx:47
MEDIUM redos-nested-quantifier packages/opencontext/src/ids.ts:10
MEDIUM redos-nested-quantifier packages/opencontext/src/ids.ts:11
MEDIUM js-timing-unsafe-mac-compare packages/opencontext/src/validate.ts:208
MEDIUM redos-nested-quantifier packages/openontology/src/ids.ts:20
LOW secret-generic-credential apps/pwa/test/appbar.test.mjs:10
LOW secret-aws-access-key packages/opencontext/src/permissions.test.ts:260
LOW secret-private-key packages/opencontext/src/permissions.test.ts:261
LOW secret-stripe-key packages/opencontext/src/permissions.test.ts:262
LOW secret-aws-access-key packages/opencontext/src/security.test.ts:228
LOW secret-private-key plugins/credential-sharing/src/providers/ssh.test.ts:7
LOW secret-database-url plugins/credential-sharing/src/rekey.test.ts:33
LOW secret-openai-key plugins/credential-sharing/src/vault-encryption.test.ts:21
LOW secret-generic-credential plugins/credential-sharing/src/vault-encryption.test.ts:21

Snippets are redacted; ThreatCrush never prints matched credential material.

@socket-security

socket-security Bot commented Sep 5, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Added@​profullstack/​x402-gateway@​0.1.07610010088100

View full report

ralyodio and others added 2 commits September 5, 2026 21:39
The crawl gateway answers with a plain Fetch Response.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2
@ralyodio
ralyodio merged commit 93af477 into master Sep 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant