Repository navigation
v2.2.0 — re-checked against the engines' own documentation, a sitemap ramp-up for new sites, robots.txt read the way Google reads it
The knowledge base re-checked claim by claim against the engines' own
documentation, a sitemap ramp-up for new sites, and an auditor that reads
robots.txt the way Google and Bing do.
seo-audit.mjs
- A robots.txt answering 429 is a server error. It was reported as an
ordinary 4xx ("your rules are ignored"); Google treats 429 like a 5xx and
stops crawling the site, so it is now a P1 that says so. - Bing's Crawl-delay from the default group is caught. Bing applies a
Crawl-delayinUser-agent: *even when abingbotgroup exists. - The AIPREF training token (
User-agent: __usage_ai-training) counts as
a deliberate AI-crawler position, like per-bot groups andContent-Signal. - 30 crawlers, up from 24: Meta-WebIndexer, Meta-ExternalFetcher,
Amzn-SearchBot, Amzn-User, MistralAI-Index and MistralAI-Training, with the
Amazonbot, MistralAI-User and Bingbot descriptions corrected.
Knowledge base — every doc checked against the engines' current documentation
All 18 docs and the 24 verticals were re-read claim by claim against Google,
Bing, Apple, Yandex and the AI operators' own pages (November 2026). The
changes that matter most:
- Grow a new site's sitemap, don't dump it (
docs/06, the workflow, the
sitemap prompt,verticals/15,16,23): a seed set of the best few
hundred URLs, then small batches that pass the index threshold, paused when
Discovered – currently not indexed grows — with Google's documented
reasons, and the schedule clearly marked as practice. - FAQ rich results are gone for every site (7 May 2026), as are the
sitelinks search box and the rich results retired in 2025. Product,
review, LocalBusiness, SoftwareApplication, Recipe, JobPosting and Article
required fields now match Google's pages; ratings aggregated from other
websites are never allowed, even with attribution. - AI features have an opt-out: the Search Console "Search generative AI"
setting (docs/09,docs/15), and Google's own Generative AI performance
report. Snippet controls govern AI answers at Google, Bing and Apple. - robots.txt status codes as Google documents them (429, the 30-day rule,
redirects), 429/503 for one or two days at most, and Google's real
crawl-budget thresholds (docs/12). - AI crawler table (
docs/10): Google's user-triggered fetchers, Amazon,
Mistral and Meta search crawlers, AIPREF, OpenAI's ads bot. - Smaller corrections across
docs/01,03,05,07,11,13,14,
16,17,18and the verticals — policy names, Google News inclusion,
Change of Address, disavow, redirects, moved and dead source links. - Google's update timeline is current again.
docs/09listed core updates
only up to August 2024; it now carries every ranking update from March 2024 to
the November 2026 core update (started 10 November, still rolling out), with
the durations Google published, and points at the Search Status Dashboard as
the authoritative list. - How to measure an update (
docs/09): wait for completion plus a full
week, compare with the week before the rollout began, and don't compare a
seasonal peak against an ordinary week. Smaller core updates run unannounced,
so recovery need not wait for the next named one (docs/09,docs/14). - Discover core updates (
docs/09,verticals/12): the February 2026
update, Google's stated goals, and what they mean for images, titles and
clickbait. Traffic-drop diagnosis (docs/15, the monitoring prompt) now
separates a Discover drop from a Search drop. - Site reputation abuse in the EEA (
docs/14): since 30 August 2026, its
manual actions no longer affect results for searches in the EEA. - Fixed:
docs/09said the helpful content system was "folded into quarterly
quality signals through 2024–2025"; it became part of core ranking in March
2024, asdocs/14already said.docs/14no longer dates the "Who, How, and
Why" guidance to December 2025, and no longer presents a press reading of the
June 2026 spam update's targets as Google's own statement.
Security
- TLS certificates are verified by default, and
--insecuresays what it
does. The flag remains an explicit opt-in for your own self-signed staging
server; it turns verification off for the whole run, so the tool now warns on
stderr, recordstlsVerifiedin the JSON and Markdown reports, and the help
and guides recommendNODE_EXTRA_CA_CERTS, which keeps verification on. - HTML is read the way a browser reads it. An end tag with whitespace or
attributes before its>(</script >) no longer leaks a script into the
page text — which could make a client-rendered shell look server-rendered —
and comments are stripped. A tag name ends where a browser ends it, so a
"</script-x>"string does not close a script and<title-bar>is neither
the title nor stripped; a comment, script or style left open runs to the end
of the page. Character references are decoded in one pass:&lt;is the
text<, not<; numeric references are decoded too. - The CI workflow's token is limited to
contents: read.