Skip to content

v2.2.0 — re-checked against the engines' own documentation, a sitemap ramp-up for new sites, robots.txt read the way Google reads it

Choose a tag to compare

@umutxyp umutxyp released this 11 Nov 04:35
· 1 commit to main since this release

The knowledge base re-checked claim by claim against the engines' own
documentation, a sitemap ramp-up for new sites, and an auditor that reads
robots.txt the way Google and Bing do.

seo-audit.mjs

  • A robots.txt answering 429 is a server error. It was reported as an
    ordinary 4xx ("your rules are ignored"); Google treats 429 like a 5xx and
    stops crawling the site, so it is now a P1 that says so.
  • Bing's Crawl-delay from the default group is caught. Bing applies a
    Crawl-delay in User-agent: * even when a bingbot group exists.
  • The AIPREF training token (User-agent: __usage_ai-training) counts as
    a deliberate AI-crawler position, like per-bot groups and Content-Signal.
  • 30 crawlers, up from 24: Meta-WebIndexer, Meta-ExternalFetcher,
    Amzn-SearchBot, Amzn-User, MistralAI-Index and MistralAI-Training, with the
    Amazonbot, MistralAI-User and Bingbot descriptions corrected.

Knowledge base — every doc checked against the engines' current documentation

All 18 docs and the 24 verticals were re-read claim by claim against Google,
Bing, Apple, Yandex and the AI operators' own pages (November 2026). The
changes that matter most:

  • Grow a new site's sitemap, don't dump it (docs/06, the workflow, the
    sitemap prompt, verticals/15, 16, 23): a seed set of the best few
    hundred URLs, then small batches that pass the index threshold, paused when
    Discovered – currently not indexed grows — with Google's documented
    reasons, and the schedule clearly marked as practice.
  • FAQ rich results are gone for every site (7 May 2026), as are the
    sitelinks search box and the rich results retired in 2025. Product,
    review, LocalBusiness, SoftwareApplication, Recipe, JobPosting and Article
    required fields now match Google's pages; ratings aggregated from other
    websites are never allowed, even with attribution.
  • AI features have an opt-out: the Search Console "Search generative AI"
    setting (docs/09, docs/15), and Google's own Generative AI performance
    report. Snippet controls govern AI answers at Google, Bing and Apple.
  • robots.txt status codes as Google documents them (429, the 30-day rule,
    redirects), 429/503 for one or two days at most, and Google's real
    crawl-budget thresholds (docs/12).
  • AI crawler table (docs/10): Google's user-triggered fetchers, Amazon,
    Mistral and Meta search crawlers, AIPREF, OpenAI's ads bot.
  • Smaller corrections across docs/01, 03, 05, 07, 11, 13, 14,
    16, 17, 18 and the verticals — policy names, Google News inclusion,
    Change of Address, disavow, redirects, moved and dead source links.
  • Google's update timeline is current again. docs/09 listed core updates
    only up to August 2024; it now carries every ranking update from March 2024 to
    the November 2026 core update (started 10 November, still rolling out), with
    the durations Google published, and points at the Search Status Dashboard as
    the authoritative list.
  • How to measure an update (docs/09): wait for completion plus a full
    week, compare with the week before the rollout began, and don't compare a
    seasonal peak against an ordinary week. Smaller core updates run unannounced,
    so recovery need not wait for the next named one (docs/09, docs/14).
  • Discover core updates (docs/09, verticals/12): the February 2026
    update, Google's stated goals, and what they mean for images, titles and
    clickbait. Traffic-drop diagnosis (docs/15, the monitoring prompt) now
    separates a Discover drop from a Search drop.
  • Site reputation abuse in the EEA (docs/14): since 30 August 2026, its
    manual actions no longer affect results for searches in the EEA.
  • Fixed: docs/09 said the helpful content system was "folded into quarterly
    quality signals through 2024–2025"; it became part of core ranking in March
    2024, as docs/14 already said. docs/14 no longer dates the "Who, How, and
    Why" guidance to December 2025, and no longer presents a press reading of the
    June 2026 spam update's targets as Google's own statement.

Security

  • TLS certificates are verified by default, and --insecure says what it
    does.
    The flag remains an explicit opt-in for your own self-signed staging
    server; it turns verification off for the whole run, so the tool now warns on
    stderr, records tlsVerified in the JSON and Markdown reports, and the help
    and guides recommend NODE_EXTRA_CA_CERTS, which keeps verification on.
  • HTML is read the way a browser reads it. An end tag with whitespace or
    attributes before its > (</script >) no longer leaks a script into the
    page text — which could make a client-rendered shell look server-rendered —
    and comments are stripped. A tag name ends where a browser ends it, so a
    "</script-x>" string does not close a script and <title-bar> is neither
    the title nor stripped; a comment, script or style left open runs to the end
    of the page. Character references are decoded in one pass: &amp;lt; is the
    text &lt;, not <; numeric references are decoded too.
  • The CI workflow's token is limited to contents: read.