Accept Header Test - AI Agent Detection

A Cloudflare Worker that serves different content formats (HTML vs Markdown) based on the Accept header, designed to test and track what AI tools and agents request when fetching web content.

Features

• Smart Format Detection: Serves Markdown to AI tools, HTML to browsers
• Comprehensive Tracking: Records 25+ data points per visit in Cloudflare KV
• AI Agent Classification: Detects GPTBot, Claude, Perplexity, and 20+ other AI crawlers
• Rich Analytics Dashboard: View visits by agent type, location, network, protocol, and headers
• Click-to-Copy: Export any visit as JSON with a single click

Live Demo

• Test Page: https://accept-header-test.nikola-balic.workers.dev
• Stats Dashboard: https://accept-header-test.nikola-balic.workers.dev/stats

Testing

# Request HTML (browser-like)
curl https://accept-header-test.nikola-balic.workers.dev

# Request Markdown (AI agent-like)
curl -H "Accept: text/markdown" https://accept-header-test.nikola-balic.workers.dev

# Request with text/plain
curl -H "Accept: text/plain" https://accept-header-test.nikola-balic.workers.dev

Deployment

# Install Wrangler CLI
npm install -g wrangler

# Deploy to Cloudflare
wrangler deploy

What Gets Tracked

Each visit captures:

• Agent Info: Type, name, user-agent string
• Format: HTML or Markdown
• Location: Country, city (IP hidden in public view)
• Network: ISP, ASN, organization
• Protocol: HTTP version, TLS version
• Headers: Accept, Accept-Language, Accept-Encoding, DNT, Sec-Fetch-*, Referer
• Request: Method, path, timestamp

Detected Agent Types

• AI Crawler: GPTBot, Anthropic, CCBot, Perplexity, Google-Extended
• AI Agent: OpenAI Retriever, You.com RAG, Phind
• AI SEO: SEMrush AI, Moz RogerBot AI
• Headless: Playwright, Puppeteer, Selenium
• HTTP Lib: axios, Python requests, Go, Java HttpClient
• CLI: curl, wget, HTTPie
• Search Bot: Googlebot, Bingbot, DuckDuckBot
• Browser: Chrome, Firefox, Safari, Edge

Why This Matters

AI agents are consuming massive amounts of web content. This tool helps you:

• See which AI tools are fetching your content
• Test if your site properly serves Markdown to AI (10x token savings)
• Track agent behavior and adoption of Accept header negotiation
• Implement GEO (Generative Engine Optimization)

Inspired By

This project was inspired by this article about serving Markdown to LLMs.

License

MIT