Skip to content

Mydentify AI Crawler Access Checker v1.0.0

Choose a tag to compare

View all tags
@mitdralla mitdralla released this 12 Aug 18:19
· 6 commits to main since this release
v1.0.0
a3b13f3

First stable release of the dependency-free Node.js crawler-access checker.

  • Evaluates robots.txt rules for six documented OpenAI and Anthropic crawler names.
  • Separates search, user-requested retrieval, and model-training purposes.
  • Inspects robots meta tags and X-Robots-Tag headers.
  • Sends labeled user-agent probes and states their IP-range limitation.
  • Supports human-readable and JSON output.
  • Includes five passing Node tests and requires no API key or package installation.

Hosted report: https://mydentify.com/tools/ai-crawler-access-checker

Assets 2
Loading