SpiderHead 3.1.0

SpiderHead 3.1.0 — first tagged release after the rename from SEOHEAD Tools.

Changes since 3.0.0

Entries for merged, unreleased work live one file per change in changelog.d/, not here:
python scripts/build_changelog.py folds them in between the markers below at release
time, so two branches never edit this file and can never conflict over it (#638).

• Keep fingerprinting a page whose site sends a valueless Set-Cookie header (#651). A header
  with no name=value pair at all -- Set-Cookie: Secure; HttpOnly, which emall.by sends
  beside two well-formed cookies -- becomes Cookie(name='Secure', value=None) in the jar, and
  httpx.Cookies.__getitem__ raises KeyError for it. dict(resp.cookies) therefore failed on
  a name the mapping's own iterator had produced: tech-detect exited 1 with a bare
  error: 'Secure' and no JSON on stdout for a page that had been fetched successfully, MCP
  seo_tech_detect raised the KeyError to its caller, and site-audit quietly came back with
  its entire technology section missing. detect_tech now walks the cookie jar instead. A
  valueless name is kept with an empty value rather than dropped, because several fingerprints
  (BITRIX_SM_GUEST_ID, craft_session, wfvt_) match on cookie name alone and dropping the
  entry would lose that signal -- but it is never silently normalized into an ordinary cookie:
  the names are reported in malformed_cookies and named in a finding, so a header the site
  sent broken stays a fact about the site. A response with only well-formed cookies is
  unaffected and reports an empty list.

• render-check measures compressed sites again instead of calling them broken (#650). The
  pinned render route read the origin with httpx's undecoded stream and handed those bytes to
  Playwright together with the origin's content-encoding. route.fulfill never applies a
  content coding to the body it is given, so Chromium parsed gzip as text/html and built a DOM
  out of the compressed stream: no title, no h1, no canonical, no links, and a rendered size
  equal to the transfer. Every site that compresses its HTML -- nearly all of them -- came back
  ok: false, reason: incomplete_render, and the message blamed the render, suggesting
  --wait domcontentloaded or a longer --timeout when the render had finished perfectly.
  seohead.tech reported 22 435 rendered bytes against 121 793 raw; it now reports 123 048 with
  the real title and 31 links. The route now reads the decoded entity, drops content-encoding
  from what it forwards (content-length was already dropped as hop-by-hop, so Playwright states
  the length of the body it actually received), and asks the origin only for the codings this
  client can decode rather than forwarding Chromium's own invitation to br and zstd. A coding
  that arrives undecoded anyway aborts the request and names itself, because passing it through
  would reproduce the bug in silence. The byte cap now counts decoded bytes, which is the body
  the renderer sees -- a compression bomb previously passed the cap compressed.

• Read Open Graph off a native crawl by the key the parser actually writes (#646).
  parse_html stores each tag under its full property name -- {"og:title": ...}, as its own
  docstring says -- and crawl/collect.py asked the same dict for "title", "description" and
  "image". dict.get answered None every time, so og_title, og_description and og_image
  were empty on every page of every native crawl ever run: 0 of 40 920 pages carried one on the
  run that surfaced this, on a site whose pages do publish Open Graph. That is the repository's
  central rule inverted -- an unmeasured field read as a measured one -- and it travelled, because
  crawl/evidence.py projects those columns into OG:Title, OG:Description and OG:Image of
  the Internal:All frame, so a check saw blank on a native crawl and the real value on a Screaming
  Frog export of the same site. check_og's honesty contract hid the damage rather than showing
  it: with no OG column populated anywhere it skipped, so a site with Open Graph audited
  identically to a site without it. The parser is unchanged; only the read side moved. A new
  tests/test_open_graph_seam.py crawls two loopback fixture sites and follows the value from
  parser to pages.jsonl to the pages table of the scan artifact to the OG:* frame columns,
  crawls a second site with no Open Graph to hold "empty because absent" apart from "empty because
  unread", and pins the parser's key shape in the same file -- the failure was two correct modules
  disagreeing about a key name with no test spanning both.

• Answer "is this site linked well, and where is it linked badly", taking the registry from 157
  to 161 checks and adding an internal-linking skill (#634). The pieces existed -- every edge
  with its position, anchor and nofollow flag in the links table, ORPHAN_PAGE,
  INLINK_BOILERPLATE_ONLY, LOW_LINK_SCORE -- and nothing turned them into an answer about the
  graph. A 40 920-page audit carried the wrong headline for hours because of it: 66% of the
  sitemap was reported unreachable by internal links, a figure produced by subtracting a crawl's
  URL count from the sitemap's, which measures "the crawler did not get there". Walking the
  artifact's own 5 252 235 edges from the home page put reachability at 91.9%. The real defect
  was depth: 23 742 pages -- 58% of the site -- were more than ten clicks from the home page, and
  the deepest was 3 005, because the archive is a linked list of "next post" links rather than a
  tree. Click depth is now the headline the audit prints. summary.internal_linking carries the
  histogram, the maximum, the reachable/unreachable split, internal edges by page position, and
  how many edges repeat another edge, and audit.md renders it beside the health summary. The
  walk starts at the URL the crawl actually began from, never at pages.crawl_depth: that column
  records where the crawler happened to reach a URL, and a sitemap-seeded crawl records 0 for
  most of a site, so a walk seeded from it describes somewhere else. When no start URL was
  recorded and more than one page claims depth 0, the measurement refuses to run and says which.

• Four checks read those numbers. DEEP_CLICK_DEPTH fires on an indexable page whose shortest
  route from the start URL exceeds thresholds.click_depth_max (default 10, and every finding
  names the number it used). DUPLICATE_INTERNAL_LINK fires on a page that writes the same
  destination and anchor more than once: on the site above, 509 744 of 1 134 307 edges -- 44.9%
  -- were repeats, and they were one masthead emitted twice with the second copy removed by
  JavaScript, a template defect that had taken a hand-written script to find. LINK_INSIDE_HEADING
  and IMAGE_LINK_WITHOUT_TEXT need what a link's page region cannot carry -- the anchor's own
  ancestor chain and contents -- so a native crawl now records a small, capped per-page
  link_placement_json beside the heading outline, and both checks skip by name on a Screaming
  Frog export rather than run clean on nothing.

• The honesty rules the rest of the analyzer holds apply throughout. An edge with no recorded
  position is counted as unclassified and never folded into content: link_position.classify
  defaults to off, which leaves that on every edge, and a run in that state must read as "nobody
  looked" rather than as a site whose links are all body copy. A position absent from the
  distribution is reported as "no edges classified here", not as a region the site lacks -- rules
  are tried in order and a menu inside <header> matches nav first. A partial crawl withdraws
  the click-depth verdict whether or not it fired, because a frontier never fetched may hold the
  shorter route, and a check silently finding nothing would read as "every page is within the
  floor".

• Stop reporting a requirement-gated check as clean when it was never evaluated (#635).
  H2_MISSING only fires when a configuration sets requirements.require_h2, and the default
  is false, so on an ordinary run its branch is unreachable. It left no trace of that: coverage
  counted it among the silent checks -- the bucket that means "ran over every page and found
  nothing" -- and a reader was told the site passed a check nobody ran. On the audit that
  surfaced this, 61% of the site's pages had no H2 at all and 5 066 of those were longer than
  800 words, so the check would have fired thousands of times. It is now declared skipped by
  name, with a reason naming the setting that turned it off, so the answer is "not evaluated,
  and here is how to evaluate it" rather than silence. require_canonical is the same shape
  and gets the same treatment; its default is true, so it was not producing a wrong reading
  yet, and now cannot start.

• Fix the task backlog heading reading "Audit Tasks — None" for any audit whose run
  recorded no project (#640). render_tasks_md built the heading with
  src.get("project", "site"), but "project" is always a present key in the backlog's
  source block -- it is set at build time from run.get("project"), so a run with no
  project stores the value None there rather than leaving the key absent, and
  dict.get's default only fires for a missing key, never for one holding None. A
  native crawl and a reanalysis both leave project unset, so every backlog built from
  either got the literal word "None" as its heading. The heading now falls back with
  or, which does catch a stored None, and reaches for the site's start_url or
  source before giving up -- the same order seohead.reports.facts.crawl_domain
  already used to resolve a site's name from its run block, now exported and reused
  here instead of reinvented, so a ba...