Skip to content

v0.0.56: a disallowed training crawler is policy, not a critical finding

Choose a tag to compare

@bsbodden bsbodden released this 03 Nov 03:48
· 17 commits to main since this release

v0.0.56: a disallowed training crawler is policy, not a critical finding

A fix to rankcli geo and the MCP server's seo_geo_check, left over from the one bot table in 0.0.55. The geo analyzer took its crawler list from the audit's table but still treated every disallowed crawler the same way: GEO_AI_CRAWLERS_BLOCKED (critical) for any of them, ten GEO points off per crawler, "URGENT: Unblock AI crawlers", and a fix that told the site to allow Claude-Web. A site that disallows the training crawlers, which OpenAI and Anthropic document as costing no visibility, lost fifty points and got a critical finding for it.

By class now, the same way the audit reports it:

Disallowed Finding GEO score
search, AI search (Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, YouBot) GEO_AI_CRAWLERS_BLOCKED, critical, with the Allow group for those crawlers as the fix -10 each
user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) GEO_AI_USER_FETCH_BLOCKED, warning (new) -5 each
training (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot, cohere-ai) listed, with a line saying it costs no visibility none

In the JSON, aiCrawlerAccess.byClass carries the split and robotsTxtAllowsAI is true when nothing that costs visibility or a user's fetch is disallowed; blockedCrawlers and allowedCrawlers keep their shape. The geo command and the MCP tool print the three disallowed rows by class. The audit (rankcli audit, the GitHub Action) is unchanged: it already reported crawler access by class.

Packages: @rankcli/cli 0.0.56, @rankcli/agent-runtime 0.0.44, @rankcli/mcp-server 0.0.29.

Install

npm install -g @rankcli/cli    # or: brew install integrallis/tap/rankcli

Standalone binaries for macOS, Linux and Windows and the Claude Desktop extension (.mcpb) are attached below.