Add AI crawler readiness check (llms.txt + robots.txt AI bot detection)#89
Open
alexander-k-eliot wants to merge 1 commit into
Open
Add AI crawler readiness check (llms.txt + robots.txt AI bot detection)#89alexander-k-eliot wants to merge 1 commit into
alexander-k-eliot wants to merge 1 commit into
Conversation
Adds AIReadinessChecker, a new module mirroring SitemapParser's session/base_domain/timeout convention. Checks whether a site publishes /llms.txt and parses /robots.txt for explicit Disallow rules against the major AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc). No existing test framework in this repo (no tests/ dir, no pytest in requirements.txt), so not introducing one unprompted -- instead verified manually against two real live sites before submitting: a site that publishes llms.txt and blocks nothing (llms_txt: True, blocked_ai_bots: []), and a site with an explicit AI-crawler-blocking robots.txt (blocked_ai_bots correctly lists GPTBot, ClaudeBot, Google-Extended, PerplexityBot, anthropic-ai). Standalone module, not yet wired into the crawl pipeline or web UI -- happy to wire it in wherever fits your architecture best if this direction looks useful, since I wasn't certain of the intended integration point in the Flask/JS layers.
alexander-k-eliot
pushed a commit
to alexander-k-eliot/ai-visibility-check-free
that referenced
this pull request
Jul 27, 2026
…thout merging Checked all 3 listed open-source PRs live via the GitHub API before touching anything, since 2 of the 3 real-world facts had changed since they were last written: - sethblack/python-seo-analyzer#132 (this session's own follow-up fix, opened and merged today by the repo owner) was missing from the 'in the wild' list entirely -- added. - nuxt-modules/sitemap#643 was listed as 'open' but is actually closed without merging (maintainer declined, no reason given, closed 2026-07-26T04:37Z) -- corrected to say so plainly rather than leave a stale, more-favorable-looking status up. - PhialsBasement/LibreCrawl#89 confirmed still genuinely open, no change needed. Also updated 'The Press Release About Nothing' page, which specifically cited 'one real, merged open-source contribution' in two places -- now two, with both PRs linked. This is exactly the kind of real number this operation is supposed to keep current rather than let go stale, the same standard just applied to press-kit/index.html in a separate repo earlier today.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds AIReadinessChecker, a new module mirroring SitemapParser's session/base_domain/timeout convention. Checks whether a site publishes /llms.txt and parses /robots.txt for explicit Disallow rules against the major AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc). No existing test framework in this repo (no tests/ dir, no pytest in requirements.txt), so not introducing one unprompted -- instead verified manually against two real live sites before submitting: a site that publishes llms.txt and blocks nothing (llms_txt: True, blocked_ai_bots: []), and a site with an explicit AI-crawler-blocking robots.txt (blocked_ai_bots correctly lists GPTBot, ClaudeBot, Google-Extended, PerplexityBot, anthropic-ai). Standalone module, not yet wired into the crawl pipeline or web UI -- happy to wire it in wherever fits your architecture best if this direction looks useful, since I wasn't certain of the intended integration point in the Flask/JS layers.