Skip to content

v3.6.0 — AI licensing, cloaking detection, crawl efficiency & full documentation

Latest
Latest

Choose a tag to compare

View all tags
@lucioduran lucioduran released this 09 Jun 16:12
· 1 commit to main since this release
v3.6.0
e1984dc

This release consolidates everything since v3.1.0 (v3.2.0 → v3.6.0): four new checks, Content Signals support, parallel batch auditing, fetch retries, a Markdown reporter, and a complete documentation set. ax-audit goes from 15 to 18 checks, and from 229 to 301 tests.

All four new checks are informational in 3.x — they run and report full findings but carry weight 0, so your existing scores and baselines are unchanged. They gain weight in v4.0.

✨ New checks

content-negotiation — Markdown for Agents (v3.2 surface, shipped 3.1; hardened since)

Probes the homepage with Accept: text/markdown — the pattern served by Cloudflare and Vercel and requested by Claude Code, Cursor, and OpenCode (~80% fewer tokens than HTML). Validates the negotiated Content-Type, that the body is real Markdown (not relabeled HTML), Vary: Accept for cache correctness, and reports the size reduction vs HTML. Partial credit for a <link rel="alternate" type="text/markdown"> fallback.

rsl — Really Simple Licensing (v3.3)

Validates RSL 1.0, the machine-readable content-licensing standard endorsed by 1,500+ publishers (Reddit, Yahoo, Medium, O'Reilly). Discovery via all three spec mechanisms — robots.txt License: directive, Link: rel="license" header, and <link rel="license" type="application/rsl+xml">. Document validation: namespace, <content url> requirement, <license> presence, permits/prohibits vocabulary (usage/user/geo), and payment types. Flags pre-1.0 draft tokens with migration hints.

agent-access — Cloaking detection (v3.4)

Probes the homepage with realistic user-agents for the 8 core AI crawlers and compares status and visible-text volume against the baseline. Catches the failure mode invisible to operators: robots.txt allows GPTBot while your WAF returns it a 403 (Cloudflare's "Block AI Crawlers" toggle produces exactly this). Blocks consistent with an explicit robots.txt Disallow are treated as intentional. Includes a verified-bots caveat for WAFs using Web Bot Auth.

crawl-efficiency (v3.5)

Measures the cost of crawling your pages: compression (rewards Brotli, accepts gzip/deflate/zstd), conditional GET (verifies an ETag/Last-Modified validator and that the server answers If-None-Match/If-Modified-Since with a real 304), and response size.

🔧 Improvements

Content Signals Policy in robots-txt (v3.2)

The robots-txt check now parses Content-Signal: directives (contentsignals.org, CC0) — the search / ai-input / ai-train preferences Cloudflare serves by default on 3.8M+ managed domains. Declared signals are reported per User-agent group; malformed segments, unknown names, and out-of-group placement produce warnings. Informational — no score impact.

Infrastructure (v3.6)

  • Fetch retries with exponential backoff for transient failures (network errors, timeouts, 408/425/429/5xx). --retries <n> (default 2). Previously a single transient timeout scored a check 0.
  • Parallel batch auditing via --concurrency <n> and the new BatchOptions type, with order-preserving output. Default remains sequential.
  • Markdown reporter — --output markdown for CI logs and PR comments (single + batch). New exports: renderMarkdown, renderBatchMarkdown.
  • Added Google's official signed AI-agent user-agent Google-Agent (agent.bot.goog) to the known-crawler list.
  • CLI now validates --retries, --concurrency, and --output.

📚 Documentation

A complete documentation set under docs/, shipped in the npm package and mirrored at lucioduran.com/projects/ax-audit/docs:

  • getting-started — first audit, reading the report, impact-ordered remediation, baselines
  • concepts — the AX standards landscape (discovery, interaction, governance/licensing, transport)
  • checks — exact per-finding scoring for all 18 checks
  • cli / api / ci / architecture / faq — full reference, with an API-stability policy
  • New CONTRIBUTING.md and SECURITY.md

Every new finding has a matching remediation guide at /projects/ax-audit/guides.

🐛 Fixes

  • Scorer division by zero: running only weight-0 checks (e.g. --checks rsl) returned NaN; now falls back to a plain average.

⚙️ Compatibility

  • No breaking changes. New checks are informational (weight 0); scores and baselines are unchanged from 3.1.x. Retries can raise scores on flaky endpoints that previously timed out, but the scoring model itself is unchanged.

📦 Install

npx ax-audit@3.6.0 https://your-site.com

Full changelog: v3.1.0...v3.6.0

Assets 2
Loading