Summary
AI training crawlers — primarily GPTBot and ClaudeBot, with smaller contributions from Bytespider and YisouSpider — are crawling the public catalog SSR pages at very high volume. Each hit triggers a full SSR render plus Postgres queries, and is billed as Workers CPU time. This is the dominant contributor to the current Cloudflare bill.
Evidence (Workers Observability, last 24 h, logs head-sampled at 25%)
| User-Agent |
Sampled requests |
Estimated actual req/day |
Avg CPU/request |
Estimated CPU/day |
| GPTBot |
97,040 |
~390,000 |
140 ms |
~54M CPU-ms (~15 CPU-hours) |
| ClaudeBot |
19,816 |
~79,000 |
179 ms |
~14M CPU-ms |
| Bytespider |
1,208 |
~4,800 |
52 ms |
~0.25M CPU-ms |
| YisouSpider |
200 |
~800 |
34 ms |
negligible |
| Real users (browsers, dataaccessd, etc.) |
— |
— |
3–60 ms |
negligible |
Estimated cost: roughly 2B CPU-ms/month from GPTBot + ClaudeBot alone, i.e. on the order of $40–50/month at standard Workers CPU pricing — the bulk of the bill.
What they crawl
Almost exclusively catalog SSR pages:
/catalog/sections/{jwId} and /catalog/sections/{jwId}/{introduction,comments,exams,teachers,calendar}/catalog/courses/{jwId}, /catalog/courses/{jwId}/sections
These URLs are enumerated for the crawlers by sitemap.xml (src/routes/sitemap.xml/+server.ts lists every non-retired course/section/teacher), and robots.txt (src/routes/robots.txt/+server.ts) currently allows all user-agents to crawl /catalog/*.
Why each hit is expensive
Every crawler hit executes the full request pipeline with zero edge caching:
- HTML responses get
Cache-Control: no-store from src/hooks.server.ts:315-317 (catalog pages set no public cache headers of their own), so Cloudflare never serves them from cache.
- Each request instantiates a fresh
PrismaClient + pg pool (see related issue: Prisma client is re-created per request), adding tens to hundreds of ms of pure CPU.
- Full SvelteKit SSR + Postgres queries via Hyperdrive for content that changes at most a few times per day.
Proposed remediation (already applied)
Resolution
Blocked at Cloudflare on 2026-07-25. Closing; the structural follow-ups are tracked in their own issues.
Summary
AI training crawlers — primarily GPTBot and ClaudeBot, with smaller contributions from Bytespider and YisouSpider — are crawling the public catalog SSR pages at very high volume. Each hit triggers a full SSR render plus Postgres queries, and is billed as Workers CPU time. This is the dominant contributor to the current Cloudflare bill.
Evidence (Workers Observability, last 24 h, logs head-sampled at 25%)
Estimated cost: roughly 2B CPU-ms/month from GPTBot + ClaudeBot alone, i.e. on the order of $40–50/month at standard Workers CPU pricing — the bulk of the bill.
What they crawl
Almost exclusively catalog SSR pages:
/catalog/sections/{jwId}and/catalog/sections/{jwId}/{introduction,comments,exams,teachers,calendar}/catalog/courses/{jwId},/catalog/courses/{jwId}/sectionsThese URLs are enumerated for the crawlers by
sitemap.xml(src/routes/sitemap.xml/+server.tslists every non-retired course/section/teacher), androbots.txt(src/routes/robots.txt/+server.ts) currently allows all user-agents to crawl/catalog/*.Why each hit is expensive
Every crawler hit executes the full request pipeline with zero edge caching:
Cache-Control: no-storefromsrc/hooks.server.ts:315-317(catalog pages set no public cache headers of their own), so Cloudflare never serves them from cache.PrismaClient+ pg pool (see related issue: Prisma client is re-created per request), adding tens to hundreds of ms of pure CPU.Proposed remediation (already applied)
User-agent: GPTBot/ClaudeBot/BytespiderDisallow: /entries (or disallow/catalog/) inrobots.txtfor well-behaved crawlersResolution
Blocked at Cloudflare on 2026-07-25. Closing; the structural follow-ups are tracked in their own issues.