Agentic Browsing audit fails llms.txt for spec-compliant files on PSI Chrome 146

[VinzSoul]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).
• Yes, my issue is not answered by other FAQs.

URL

https://www.spinoplex.com/

What happened?

The Agentic Browsing audit reports "llms.txt does not follow recommendations - Fetch of llms.txt failed" on a file that:

1. Is fully spec-compliant per llmstxt.org — independently validated by a third-party llms.txt validator with 0 errors, 0 warnings, STATUS: Pass (882 links across 18 sections)
2. Returns HTTP 200 OK with Content-Type: text/plain; charset=utf-8 to all tested user agents (Chrome-Lighthouse, PTST, default curl)
3. Was successfully fetched by Google's PSI runner during the audit — confirmed via my CDN edge logs, which show two distinct PSI requests from Google datacenter IPs (ASN 15169, Finland) on 2026-06-18 at 00:38:11 UTC and 01:06:21 UTC, both delivering 200 OK with the spec-compliant file content

The error message "Fetch of llms.txt failed" is misleading — the fetch demonstrably succeeded twice. The actual failure must be in post-fetch content validation that isn't documented in https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt, which only mentions server errors as a failure condition.

Additionally, the PSI runner reports "Using HeadlessChromium 146.0.7680.177" in the report footer, while https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring states "Testing this category requires Chrome 150 or later". The audit appears to be running in an unsupported configuration but reports a misleading error instead of marking the check as Not Applicable.

What did you expect?

For a file that:

• Returns HTTP 200 OK with text/plain content-type
• Is valid spec-compliant Markdown with an H1 first, followed by a blockquote summary and bulleted section lists
• Is reachable from Google's PSI runner (verified in CDN edge logs)

I would expect one of:

1. Pass — the audit recognizes the file as valid and counts it toward 3/3 Agentic Browsing.

2. Not Applicable (N/A) — if the audit cannot run reliably on Chrome 146 (as the official docs indicate Chrome 150+ is required), the check should be skipped rather than marked as failed.

3. Accurate error message — if there is genuinely a content validation issue, the error text should describe what actually failed (e.g., "missing H1", "invalid Markdown", "exceeds size limit") rather than the misleading "Fetch of llms.txt failed", which is provably incorrect.

The current behavior — reporting "Fetch failed" when the fetch demonstrably succeeded, on a spec-compliant file, while running on a Chrome version below the documented minimum — produces a false negative that's confusing for site owners and unactionable.

What have you tried?

Extensive debugging to rule out site-side causes:

1. Verified file accessibility from multiple user agents — curl -I with default UA, Chrome-Lighthouse UA, and PTST UA all returned HTTP 200 OK with Content-Type: text/plain; charset=utf-8 from both www and apex hostnames.

2. Confirmed file is not blocked by CDN/WAF — added a WAF Skip rule in Cloudflare scoping /llms.txt to bypass all security layers (Managed Rules, Bot Fight Mode, Browser Integrity Check, rate limiting). After deploying, Cloudflare firewall event logs explicitly captured two PSI fetch requests from Google datacenter IPs (66.102.9.98 and 66.102.9.100, both Finland, ASN 15169) at 2026-06-18 00:38:11 UTC and 01:06:21 UTC. Both logged as "action: skip" + 200 OK delivered. Audit still reported 2/3.

3. Validated file structure against the llms.txt spec — used an independent third-party llms.txt validator. Result: STATUS: Pass, 0 errors, 0 warnings, 882 links across 18 sections, H1 first, blockquote summary, bulleted lists, UTF-8 without BOM.

4. Cleaned file format end-to-end — removed plugin-added preamble line that violated llmstxt.org's "title-first" rule, decoded HTML entities, separated H1 from inline description into proper H1+blockquote structure, added missing bullet prefixes. Re-validated after each change.

5. Re-ran PSI multiple times to rule out transient flakiness — consistent 2/3 across runs.

6. Verified Chrome version against docs — PSI footer reports HeadlessChromium 146.0.7680.177; official Agentic Browsing scoring docs specify Chrome 150+ as minimum requirement.

At this point all variables on the site/CDN side have been ruled out. The remaining failure mode must be inside the Lighthouse audit logic itself, likely related to either the Chrome version gap or undocumented content validation that the docs don't describe.

How were you running Lighthouse?

PageSpeed Insights

Lighthouse Version

13.4.0

Chrome Version

146.0.7680.177 (as reported by PSI footer: "Using HeadlessChromium 146.0.7680.177 with lr"). Not run locally — this is the version PSI's runner is using.

Node Version

N/A — issue reproduced via PageSpeed Insights web UI, not CLI/Node.

OS

Windows 11 (client side — but the audit itself runs on Google's PSI infrastructure)

Relevant log output

CDN edge logs (Cloudflare) captured two successful PSI fetches of /llms.txt:

Fetch 1: 2026-06-18 00:38:11 UTC
  Source IP: 66.102.9.98 (Google LLC, ASN 15169, Finland)
  User-Agent: Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Mobile Safari/537.36 Chrome-Lighthouse
  Protocol: HTTP/2
  Method: GET
  Path: /llms.txt
  Outcome: 200 OK delivered, Content-Type: text/plain; charset=utf-8

Fetch 2: 2026-06-18 01:06:21 UTC
  Source IP: 66.102.9.100 (Google LLC, ASN 15169, Finland)
  User-Agent: same as above
  Protocol: HTTP/2
  Method: GET
  Path: /llms.txt
  Outcome: 200 OK delivered, Content-Type: text/plain; charset=utf-8

Despite both successful fetches, the audit reports "Fetch of llms.txt failed" in the PSI report.

Note: the UA string contains "Chrome/136.0.0.0" but the PSI footer reports the runner as Chrome 146.0.7680.177. The 136 in the UA appears to be a hardcoded Lighthouse emulation string and does not reflect the actual Chrome binary running the audit.

[connorjclark]

Confirmed: https://pagespeed.web.dev/analysis/https-spinoplex-com/od5dxc6uot?form_factor=mobile

[connorjclark]

Can repro even w/ CLI (so not related to PSI)

LH:LlmsTxt:error Error: Timed out fetching resource
  LH:LlmsTxt:error     at file:///Users/cjamcl/src/lighthouse/core/gather/fetcher.js:130:46
  LH:LlmsTxt:error     at new Promise (<anonymous>)
  LH:LlmsTxt:error     at Fetcher._wrapWithTimeout (file:///Users/cjamcl/src/lighthouse/core/gather/fetcher.js:129:28)
  LH:LlmsTxt:error     at Fetcher._fetchResourceOverProtocol (file:///Users/cjamcl/src/lighthouse/core/gather/fetcher.js:111:33)
  LH:LlmsTxt:error     at Fetcher.fetchResource (file:///Users/cjamcl/src/lighthouse/core/gather/fetcher.js:37:17)
  LH:LlmsTxt:error     at LlmsTxt.getArtifact (file:///Users/cjamcl/src/lighthouse/core/gather/gatherers/agentic/llms-txt.js:24:39)
  LH:LlmsTxt:error     at file:///Users/cjamcl/src/lighthouse/core/gather/runner-helpers.js:106:45 +0ms

[connorjclark]

@VinzSoul we use a timeout of 2s for these fetches, but this llms.txt takes more.

time curl https://www.spinoplex.com/llms.txt

0.01s user 0.01s system 0% cpu 2.342 total

Do you know why the server takes so long?

[VinzSoul]

Hey @connorjclark — thanks for the precise diagnosis, that explains everything I was chasing.

The root cause on my side was two compounding factors:

1. Rank Math (WordPress SEO plugin) generates llms.txt dynamically from the database on every request — 882 links across 18 sections, plus a small PHP filter I added to clean its output formatting.
2. Cloudflare wasn't caching the file at edge — cf-cache-status: DYNAMIC meant every request was hitting LiteSpeed origin.

Deployed a Cloudflare Cache Rule with 1-day edge TTL for /llms.txt.

Before fix:

time curl https://www.spinoplex.com/llms.txt
0.01s user  0.01s system  0% cpu  2.342 total
cf-cache-status: DYNAMIC

After fix:

Measure-Command { curl.exe -s https://www.spinoplex.com/llms.txt | Out-Null }
TotalMilliseconds : 118.8
cf-cache-status: HIT

~20x faster, well within the 2s budget.

However — PSI still reports Agentic Browsing 2/3 on subsequent runs. The likely reason: Cloudflare's edge cache is per-data-center. My request hits a warm edge, but PSI's runner fetches from a different Cloudflare edge (presumably closer to Google's data centers), which may still be cold and exceed 2s on first request from that region.

If you can re-run your local CLI test against https://www.spinoplex.com/llms.txt now, you should see the dramatically improved response time — and if that edge is already warm, the audit should pass. If it's still slow, that confirms the per-edge cache warming problem affects real-world site owners using CDNs (which is most of them).

Two suggestions still worth considering on the Lighthouse side, since this trap will hit other site owners too:

1. Error message clarity — "Fetch of llms.txt failed" is misleading when the actual cause is timeout. Something like "Fetch timed out (exceeded 2s budget)" would tell site owners to optimize delivery speed rather than investigate WAF/CDN/file content. Most of my 6-hour debugging cycle went into ruling out things that weren't the cause — clearer wording would have saved that time.

2. Timeout reasonableness — 2s is aggressive for the real world. WordPress sites using Rank Math, Yoast, or AIOSEO to dynamically generate llms.txt from database queries will commonly exceed it on uncached or cold-cached requests. Bumping to 5s or making it configurable would catch more legitimate files. Even with a CDN cache, the first cold-edge request from PSI's specific data center can exceed 2s.

Thanks again for the fast triage on this.

[connorjclark]

It shouldn't take 2+ seconds to fetch or generate a small a text file, cached or not. I still spuriously gets more than 2s when running time curl even w/ your cache additions.

We won't increase our timeout because that would degrade services slightly, but moreso real users would be waiting 2s+ occasionally and I don't want to hide that detail, so ideally that can be resolved on the server side.

Better error messages will land soon.

---

Also, please don't respond with an LLM, I won't respond again unless it's clear I'm responding to a human :)

[VinzSoul]

Hey @connorjclark, sorry for the AI reply, I was the one debugging and filing this, just used AI to clean the writing. Will be myself from now :)

your timeout reasoning makes sense, I won't push on that. and the PR for better error messages is huge, would have saved me hours.

about my llms.txt being too big, fair point. 127kb with 882 links is way too much. I'll cut it down to ~100 high signal URLs (top casinos, key pages, sitemap) and that should fix the >2s problem from your location too. thanks for digging into this.