Repository navigation
v0.0.56: a disallowed training crawler is policy, not a critical finding
v0.0.56: a disallowed training crawler is policy, not a critical finding
A fix to rankcli geo and the MCP server's seo_geo_check, left over from the one bot table in 0.0.55. The geo analyzer took its crawler list from the audit's table but still treated every disallowed crawler the same way: GEO_AI_CRAWLERS_BLOCKED (critical) for any of them, ten GEO points off per crawler, "URGENT: Unblock AI crawlers", and a fix that told the site to allow Claude-Web. A site that disallows the training crawlers, which OpenAI and Anthropic document as costing no visibility, lost fifty points and got a critical finding for it.
By class now, the same way the audit reports it:
| Disallowed | Finding | GEO score |
|---|---|---|
| search, AI search (Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, YouBot) | GEO_AI_CRAWLERS_BLOCKED, critical, with the Allow group for those crawlers as the fix |
-10 each |
| user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) | GEO_AI_USER_FETCH_BLOCKED, warning (new) |
-5 each |
| training (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot, cohere-ai) | listed, with a line saying it costs no visibility | none |
In the JSON, aiCrawlerAccess.byClass carries the split and robotsTxtAllowsAI is true when nothing that costs visibility or a user's fetch is disallowed; blockedCrawlers and allowedCrawlers keep their shape. The geo command and the MCP tool print the three disallowed rows by class. The audit (rankcli audit, the GitHub Action) is unchanged: it already reported crawler access by class.
Packages: @rankcli/cli 0.0.56, @rankcli/agent-runtime 0.0.44, @rankcli/mcp-server 0.0.29.
Install
npm install -g @rankcli/cli # or: brew install integrallis/tap/rankcliStandalone binaries for macOS, Linux and Windows and the Claude Desktop extension (.mcpb) are attached below.