You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
X-Robots-Tag parsing: a bot prefix applies to its own directive only. googlebot: noindex, noindex (two headers
joined by the header line) hid the second, global noindex, and the page was submitted.
RobotsCache keys by origin (robots.<scheme>_<host>_<port> for anything but https on 443): staging on http://host:8080 and production on https://host no longer share one robots.txt for an hour.
The pre-flight transport reads at most 1 MiB of a page (VerifyConfig::BODY_LIMIT) instead of the 50 MiB of the core's
GET limit; the signals live in the first 256 KiB.
Requires indexnowkit/core ^0.11.
Added
verify.time_budget (60 s): the pre-flight of one batch stops when the budget is spent and the remaining URLs are
sent unverified with one warning, so a queue job never runs past its visibility timeout into a second worker.
Check\DispatchCheck (verify.dispatch, with the adapter's queue mode) and Check\TransportCheck
(verify.transport: an application http.client keeps its own settings, so verify.redirect, verify.max_redirects, verify.timeout and the host check on redirect targets may not apply) — the adapters register both.