AEO-CLI

Audit any URL for AI crawler readiness. Get a 0-100 AEO score.

What is AEO?

Agentic Engine Optimization (AEO) is the practice of making your website discoverable and accessible to AI agents and LLM-powered search engines. As AI crawlers like GPTBot, ClaudeBot, and PerplexityBot become major traffic sources, AEO ensures your content is structured, permitted, and optimized for these systems.

AEO-CLI checks how well a URL is prepared for AI consumption and returns a structured score.

Features

• Robots.txt AI bot access — checks whether major AI crawlers are allowed or blocked
• llms.txt presence — detects the emerging standard for LLM-specific site instructions
• Schema.org JSON-LD — extracts and evaluates structured data markup
• Content density — measures useful content vs. boilerplate via markdown conversion
• Rich CLI output — formatted tables and scores via Rich
• JSON output — machine-readable results for pipelines
• MCP server — expose the audit as a tool for AI agents via FastMCP
• AEO Compiler — LLM-powered llms.txt and schema.jsonld generation
• CI/CD integration — --fail-under threshold, --fail-on-blocked-bots, GitHub Step Summary
• GitHub Action — composite action for CI pipelines

Installation

pip install aeo-cli

AEO-CLI uses a headless browser for content extraction. After installing, run:

crawl4ai-setup

Development install

git clone https://github.com/your-org/aeo-cli.git
cd aeo-cli
pip install -e ".[dev]"
crawl4ai-setup

Quick Start

aeo-cli audit example.com

This runs a full audit and prints a Rich-formatted report with your AEO score.

CLI Usage

Single Page Audit

Audit only the specified URL (skip multi-page discovery):

aeo-cli audit example.com --single

Multi-Page Site Audit (default)

Discover pages via sitemap/spider and audit up to 10 pages:

aeo-cli audit example.com

Limit Pages

aeo-cli audit example.com --max-pages 5

JSON Output

Get structured JSON for CI pipelines, dashboards, or scripting:

aeo-cli audit example.com --json

CSV / Markdown Output

aeo-cli audit example.com --format csv
aeo-cli audit example.com --format markdown

Verbose Mode

Show detailed per-pillar breakdown with scoring explanations:

aeo-cli audit example.com --single --verbose

CI Mode

Fail the build if the AEO score is below a threshold:

aeo-cli audit example.com --fail-under 60

Fail if any AI bot is blocked:

aeo-cli audit example.com --fail-on-blocked-bots

Exit codes: 0 = pass, 1 = score below threshold, 2 = bots blocked.

When running in GitHub Actions, a markdown summary is automatically written to $GITHUB_STEP_SUMMARY.

Quiet Mode

Suppress output, exit code 0 if score >= 50, 1 otherwise:

aeo-cli audit example.com --quiet

Use --fail-under with --quiet to override the default threshold:

aeo-cli audit example.com --quiet --fail-under 70

Start MCP server

aeo-cli mcp

Launches a FastMCP stdio server exposing the audit as a tool for AI agents.

MCP Integration

To use AEO-CLI as a tool in Claude Desktop, add this to your Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "aeo-cli": {
      "command": "aeo-cli",
      "args": ["mcp"]
    }
  }
}

Once configured, Claude can call the audit_url tool directly to check any URL's AEO readiness.

AEO Compiler (Generate)

Generate llms.txt and schema.jsonld files from any URL using LLM analysis:

pip install aeo-cli[generate]
aeo-cli generate example.com

This crawls the URL, sends the content to an LLM, and writes optimized files to ./aeo-output/.

BYOK (Bring Your Own Key)

The generate command auto-detects your LLM provider from environment variables:

Priority	Env Variable	Model Used
1	OPENAI_API_KEY	gpt-4o-mini
2	ANTHROPIC_API_KEY	claude-3-haiku-20240307
3	Ollama running locally	ollama/llama3.2

Override with --model:

aeo-cli generate example.com --model gpt-4o

Industry Profiles

Tailor the output with --profile:

aeo-cli generate example.com --profile saas
aeo-cli generate example.com --profile ecommerce

Available: generic, cpg, saas, ecommerce, blog.

GitHub Action

Use AEO-CLI in your CI pipeline:

- name: Run AEO Audit
  uses: hanselhansel/aeo-cli@main
  with:
    url: 'https://your-site.com'
    fail-under: '60'

The action sets up Python, installs aeo-cli, and runs the audit. Outputs score and report-json for downstream steps. See docs/ci-integration.md for full documentation.

Score Breakdown

AEO-CLI returns a score from 0 to 100, composed of four pillars:

Pillar	Max Points	What it measures
Content density	40	Quality and depth of extractable text content
Robots.txt AI bot access	25	Whether AI crawlers are allowed in robots.txt
Schema.org JSON-LD	25	Structured data markup (Product, Article, FAQ, etc.)
llms.txt presence	10	Whether a /llms.txt file exists for LLM guidance

Scoring rationale (2026-02-18)

The weights reflect how AI search engines (ChatGPT, Perplexity, Claude) actually consume web content:

• Content density (40 pts) is weighted highest because it's what LLMs extract and cite when answering questions. Rich, well-structured content with headings and lists gives AI better material to work with.
• Robots.txt (25 pts) is the gatekeeper — if a bot is blocked, it literally cannot crawl. It's critical but largely binary (either you're blocking or you're not).
• Schema.org (25 pts) provides structured "cheat sheets" that help AI understand entities (products, articles, organizations). Valuable but not required for citation.
• llms.txt (10 pts) is an emerging standard. No major AI search engine heavily weights it yet, but it signals forward-thinking AI readiness.

AI Bots Checked

AEO-CLI checks access rules for these AI crawlers:

• GPTBot
• ChatGPT-User
• Google-Extended
• ClaudeBot
• PerplexityBot
• Amazonbot
• OAI-SearchBot

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Lint
ruff check src/ tests/

License

MIT