v1.16.0

Fixed: markdown-negotiation middleware returned 404 to agents (Cloudflare Pages, Netlify)

The middleware enhance writes for Cloudflare Pages (functions/_middleware.js) and Netlify
(netlify/edge-functions/) treated every markdown-preferring request as a page request. For any
URL that already has a file extension it looked up <path>.md, found nothing and served the
/404.md mirror with a 404:

• .md URLs requested with Accept: text/markdown (Claude Code's WebFetch sends
  text/markdown, text/html, */*), so the .md links your llms.txt advertises were broken for
  those clients.
• robots.txt, sitemap.xml and nested llms*.txt indexes requested by an AI crawler UA
  (GPTBot, ClaudeBot, ...).

The templates now skip negotiation for any path with a file extension. Claude-User (Claude's
user-initiated fetcher) is added to the bot list. Vercel's Proxy was not affected.

If you already ran enhance: it never overwrites an existing middleware, so your site still
has the old one. Either delete the generated file and re-run siteready enhance, or add this
guard before the markdown branch:

const url = new URL(request.url);
const hasExtension = /\.[a-z0-9]+$/i.test(url.pathname);
if (wantsMarkdown && !hasExtension) {

New: agent-headers scanner (on by default)

The external scanners test one header pattern per URL type and follow redirects, so they all
passed the broken middleware above. agent-headers requests every kind of URL your llms.txt
links to (index files, .md links, pages, plus robots.txt/sitemap.xml, following nested
llms*.txt one level) with real agent headers (Claude Code's Accept, a GPTBot UA) and without
following redirects. A check fails when a URL that works for a plain request returns an error, or
redirects to a different path, for an agent.

It's plain fetch() with no hosted API behind it, so it runs by default for scan/rescan and
for scan-local/rescan-local/loop. Run it alone with --scanners agent-headers.