v4.16.0 — Ground Truth

Fixed

• Bot roster corrected against current vendor documentation (#512). Googlebot was completely absent from AI_BOTS/CITATION_BOTS even though Google's own docs confirm AI Overviews are fed by Googlebot's crawl data, not by Google-Extended (which is a robots.txt opt-out token for Gemini/Vertex training, not a fetching agent). Claude-User (Anthropic's current on-demand fetch bot) was missing entirely, in its place were two identifiers — anthropic-ai, claude-web — that no longer appear in Anthropic's published crawler docs. CITATION_BOTS also mixed in ClaudeBot (training-only per Anthropic) while excluding Googlebot/Applebot; it now reads {OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot}, matching the crawlers that actually drive AI citations. Applebot moved from the training to the search bot tier (Applebot-Extended is the training-only one).
• geo access CDN/WAF live-fetch check only simulated 3 bots. audit_cdn_ai_crawler tested GPTBot/ClaudeBot/PerplexityBot — none of which include the actual search/citation crawlers the check exists to protect. Now tests the six bots that matter for citation-vs-training visibility: GPTBot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Applebot.

Added

• geo access reports TTFB and page weight. Two access-layer signals a non-rendering AI crawler is sensitive to were entirely unmeasured: time-to-first-byte (warns above 500ms) and raw initial-HTML weight (warns above 200KB). Both come from the single fetch geo access already performs — no extra network call.
• geo authority flags orphan pages. Previously reported interlinking only at the cluster/topic level; now also lists individual analyzed pages with zero inbound internal links from the other pages (homepage excluded — it's reachable by definition), since a page only reachable via the sitemap carries weak entity association even when crawled.