sigwood v0.2.9

Added

• The DNS suspicion score's measured catch rates are now written down. The FAQ and the
  known-issues list state what the score actually catches, from a seeded measurement against
  the live scorer (1,000 samples per label length, eleven lengths from 6 to 63): the
  vowel-free digit-and-consonant shape the score is tuned toward clears the candidate bar
  about 19-36% of the time at typical DGA lengths (10-16 characters); a uniformly random
  letter-digit label, 6.6-14.8% at the same lengths; and random all-letter labels never clear
  it at any length - zero of 11,000 samples, with the best possible all-letter label below
  the bar by arithmetic. Boosting the letter-only class lexically was measured and rejected:
  every boost rule tested either flagged real benign names - the strictest still crossed 57
  in a benign reference week - or caught under half of its target, because the benign score
  curve decays smoothly through the region with no gap to put a threshold in. The docs pair
  the numbers with what one name's score does and does not decide: a score over the bar makes
  a candidate, families group by parent, and the below-gate family check - five or more
  low-scoring subdomains under one parent on Zeek data, nearly all of whose lookups fail to
  resolve - surfaces as one INFO finding regardless of score, while a family spread across
  many parents, or one whose names resolve, is outside it. The hex-tunneling entry is
  corrected by the same measurement: a long hex label straddles the bar - roughly half of
  samples clear it - rather than sitting just under it; the conclusion that a high-volume hex
  tunnel can slip the dense-cluster scan is unchanged, since half is well short of the member
  share the scan requires.

• Every expandable system-log finding now shows its vocabulary above the fold. The short
  tokens: line that previews what a capsule contains was reaching grouped families and
  bursts but not recognized administrative sessions or update runs - so the most structured
  finding sigwood produces was the one row on the page with nothing under it until you opened
  it. Recognized sessions now carry the same preview, distilled from the lines they group,
  and it appears on both the plain-text and HTML reports. A capsule whose content is entirely
  opaque identifiers still shows no preview rather than an empty one.

• A field-validation kit for independent testers. tools/fieldkit.py is one standalone
  script a collaborator can download and run wherever the sigwood command is installed: it
  proves the install with a small synthetic canary, runs exactly one ordinary default hunt,
  offers an optional local triage pass, and writes one reviewable Markdown report. The
  automated projection never copies a log-derived string - it keeps enumerated numeric
  measurements, counts, and fixed tokens, and groups anything unexpected under other; the
  three typed answers are the sole free-text exception, and the file is created privately and
  reviewed before the collaborator chooses to email it. docs/FIELDKIT.md states the complete
  field list and the exact privacy boundary. The kit is a repository tool, not a sigwood
  command, and nothing about the product changes. A quiet network now reads as a quiet
  network: a hunt that completes and reports nothing says so in its own words, distinct from
  a run where sigwood produced no readable report at all. The collaborator page also names
  pipx as a precondition, since several distributions do not ship it and block the
  pip-install alternative.

• Families of low-scoring names whose lookups nearly all fail to resolve are now visible. A single
  odd-looking domain that scores below the detector's bar stays quiet, as before - but when
  one parent domain carries several such names and nearly all of their lookups come back
  NXDOMAIN, sigwood now reports the family as one INFO finding. That is the shape of a
  rotating rendezvous or a subdomain generator whose infrastructure is dead, and it was
  previously invisible no matter how many names were involved. It reads the whole log rather
  than the clustered subset, so a busy family cannot hide by being big. Zeek only - Pi-hole
  logs carry no response code, so the check stays silent there rather than guessing. Tunable
  under [detectors.dns] (promote_below_gate, promote_min_subdomains,
  promote_min_nxdomain_fraction), and off by one setting if you would rather not see them.

• The known-issues list now records what a very thin allowlist costs the DNS path. Turning
  suppression off, or running with an unusually short allowlist over a large window, makes DNS
  clustering markedly more expensive - on one seven-day corpus of about 2.2 million rows the DNS
  detector on its own took about eight minutes unsuppressed and sat near 8 GiB of memory, and
  running the whole default hunt over that same window unsuppressed did not finish inside a
  nine-minute bound. The entry gives the measured figures, says plainly that they come from one
  corpus on one machine, and offers the three things that reliably help: narrow the window, run
  the detector on its own, or leave suppression on. Nothing about detection changed - this is a
  limitation that was always there and is now written down.

• The known-issues list now records how a recognized admin session or update run groups
  lines. Grouping claims the rare lines that fall inside the session's window - that is what
  turns one administrator's work into a single row instead of nineteen - and it means a line
  that merely happened at the same time is grouped too, and can set the unit's severity if it
  came from a privileged program. Every grouped line is still listed under the unit with its own
  time and program, so the entry tells you to read a unit as "these things happened together"
  rather than "these things are the same event". Nothing about detection changed; narrowing the
  grouping by program was measured and rejected because it dissolved genuine sessions.

• Beacon findings carry their own event times. Each finding now records when the
  periodic flow was first and last seen, how long it ran, and roughly how many cycles
  that span covers - so a ten-minute cadence that ran for an hour reads differently
  from one that ran all week, without leaving the finding. The first-seen time and the
  span show at --verbose and in the CSV worklist; the full set is in -vv and the
  JSON feed. Detection is unchanged.

• Every grouped syslog finding now says what it is a fraction of. Bursts, reboots,
  and recognized transactions carry the host's full analyzed line count beside their own,
  so "12 rare lines" can be read against the 40 lines that host produced or the 400,000
  it produced. Rare-line and per-program findings already carried the equivalent figure.
  Visible with -v and in the CSV worklist; the default one-line rows are unchanged.

Changed

• DNS clustering now treats response outcomes as evidence, not geometry. Response
  codes no longer act like ordered numbers in the clustering matrix, suffix categories
  keep stable identities when counts tie, and a missing round-trip time stays
  distinguishable from a median observation without overpowering the other features.
  Cluster membership near the candidate bar can shift slightly as a result: on a benign
  reference week this meant four fewer barely-over-the-bar findings, with every
  corroborated and grouped finding unchanged.
• DNS severity is earned by behaviour now, not by the label score alone. A
  high-scoring domain name on its own is a lead, not a verdict, so it reports
  MEDIUM. HIGH additionally requires corroboration from what the queries actually
  did: most lookups under the name failing to resolve (a majority of responses,
  and at least two failures), or the name belonging to a dense high-entropy
  cluster - the shape of tunneling. One failed lookup is an event, not a pattern,
  so a single dead or mistyped hostname no longer reaches HIGH. Every DNS finding
  now records which corroboration applied, and says plainly when none did.
• DNS findings carry their resolution evidence. Where it can be measured,
  findings report the share of lookups that failed to resolve and how many failed.
  Pi-hole logs carry no response code, so on a Pi-hole-only run that corroboration
  is unavailable and DNS findings top out at MEDIUM - add Zeek DNS logs for the
  fuller picture. Whether a domain was blocked remains evidence you can read; it
  never decides severity.
• Beacon severity follows the same rule: timing alone tops out at MEDIUM. A
  periodic connection train is one category of evidence - a shape worth review,
  not a corroborated verdict - so a beacon finding on its own no longer reaches
  HIGH, whatever its score. The finding's prose now says what was measured (the
  regular cadence of an automated check-in) instead of asserting a C2 beacon,
  and its suggested next steps lead with local evidence - the process, dns.log,
  and conn.log history - before any external reputation lookup. Detection is
  unchanged: the same flows surface with the same scores.

Fixed

• Imitating a service role name no longer excludes a CloudTrail event from analysis.
  Events are treated as automated service activity - and so left out of the per-principal
  scoring - when a role name in the record carries the AWSServiceRoleFor marker AWS uses
  for its own service roles. That marker was matched anywhere in the name, so a role called
  MyAWSServiceRoleForSomething qualified and its events were set aside with no note and no
  count. The marker must now begin a path segment of the role's identifier, and the
  comparison is case-sensitive; anything else is treated as interactive and is analyzed.
  Genuine service roles are unaffected. A role named exactly like a service role still
  qualifies, which is recorded in the known-issues list with the reason it is not closed.

• A CloudTrail record can no longer exclude itself from analysis by imitating an AWS
  service name. An event is treated as automated service activity - and so left out of
  the per-principal scoring entirely - when the field naming what invoked it ends in
  amazonaws.com. That test had no boundary, so a value such as not-a-host/amazonaws.com
  satisfied it and the event was set aside with no note and no count. The field must now be
  exactly amazonaws.com or end with .amazonaws.com, and the comparison is
  case-sensitive; anything else is treated as interactive and is analyzed. Genuine service
  activity is unaffected, because every value AWS writes there has the form
  <service>.amazonaws.com.

• A malformed log file can no longer stop a run in two specific ways. A connection log
  whose timestamps span an implausible range - ten lines dated years apart - made the beacon
  detector ask for billions of memory slots before it scored anything, which ended the
  process. It now declines to score that flow and carries on, which is what it already did
  for every other thing it cannot measure. Separately, a CloudTrail file containing a number
  with thousands of digits, or JSON nested thousands of levels deep, produced a raw Python
  error instead of a warning; both are now reported as an unreadable file and skipped, and a
  single bad line in a newline-delimited file no longer discards the good lines around it.
  A Zeek TSV log whose column-type header declares a container nested thousands of levels
  deep did the same thing; that line is now skipped and counted with the file's other
  malformed lines, and the rest of the file still loads. A column with an unusual type that
  never carries a value keeps parsing exactly as before - the limit applies to values, not to
  the header. None of these change what sigwood reports on ordinary logs.

• A system-log capsule that shows a sample now says so. Opening a finding that reported
  97 rare lines showed twenty of them and gave no sign the rest existed, which reads as a
  miscount or a broken control. The capsule now closes with showing 20 of 97 rare lines
  whenever the lines on screen are fewer than the lines counted, and says nothing at all when
  the capsule is complete. A grouped administrative session or update run discloses per
  grouped entry rather than once at the top, using the same count that entry already prints.
  The twenty-line sample itself is unchanged: those are the lines sigwood carries, so the
  disclosure states the shortfall rather than offering a switch that cannot recover it.

• A Zeek syslog.log line with no message no longer becomes a finding about text
  that never existed. A row whose message field is empty of content - JSON null,
  or - in TSV - was turned into the literal words None or nan, which then read as
  a program name, templated like any other line, and could surface as a rare finding.
  Those rows are now dropped before that can happen, and the count is reported:
  syslog.log: skipped 3 rows with a missing or non-text message. A genuinely empty
  message is kept, and so is a real log line whose text happens to read None - the
  check is on the field's type, not on how it prints.

• A mistyped syslog setting now says which setting is wrong, instead of failing
  strangely later. [detectors.syslog] values are checked before the detector
  runs: an out-of-range rarity_pct used to crash mid-run with an unhelpful
  message, max_count = 0 silently switched rare-line detection off entirely, and
  writing privileged_programs = "useradd" instead of a list quietly emptied the
  privileged tier - a string is a sequence of letters, so the roster became a,
  d, e, r, s, u. Each is now reported against the setting that caused it,
  the other detectors still run, and no traceback reaches the terminal. This
  repairs crash and silent-disable shapes; it does not change detection on a valid
  configuration, and rarity_pct remains inert at the shipped max_count = 1.

• Beacon says why it scored nothing, instead of just reporting nothing. When the
  connection log is missing a column beacon needs, when every eligible connection
  lacks byte counts, or when rows carry no source, destination, port or protocol,
  the run summary now says so and names the cause - previously the detector
  returned an empty result that read exactly like "nothing was periodic." The
  counts describe the connections beacon actually received, after allowlist
  suppression, so a note never blames missing bytes for rows your allowlist
  removed. A configured min_connections below the scorer's own floor is now
  disclosed as well, rather than quietly having no effect.

• A malformed [detectors.beacon] setting fails with a message that names the
  key. A zero bin_seconds, a non-numeric threshold, or a text value where a
  number belongs used to surface as an opaque detector crash partway through a
  run; it is now reported against the setting that caused it, the other detectors
  still run, and no traceback reaches the terminal.

• A connection log missing an optional column no longer crashes beacon. Absent
  bytes, conn_state, port or proto columns are a fidelity limitation of the
  source, so beacon now abstains and says which columns were missing.

• The beacon run-time disclosure describes the excluded connections truthfully.
  It now says unscored connections were outside the Zeek SF/S1 states beacon
  analyzes, rather than calling them all "not established" - reset and
  half-closed connections are established states, and the old wording misfiled
  them. The known-issues entry carries the same correction, and the eligibility
  it states now matches the code exactly.

• Beacon's suggested history pivot no longer prints a command that hangs. The
  old zeek-cut suggestion piped nothing into zeek-cut and assumed a TSV file
  in the current directory; the step is now a plain instruction to review the
  destination's history in conn.log.