v4.10.2

Fixed

• petit no longer logs on the root logger. Each module logs through
  logging.getLogger(__name__), so a host that calls
  logging.basicConfig(level=…, format=…) after using petit gets its level
  and format. Before, the first logging.info() from Filter installed a
  default stderr handler on the root logger and the host's basicConfig()
  became a silent no-op. The CLI's -v and -d output is unchanged. For #89.

Changed

• CI: constitution validation runs through constitution's reusable
  validate.yml pinned to a release (constitution.yml), replacing the job
  that checked the validator out at HEAD. Dependabot auto-merges green
  GitHub Actions minor/patch bumps (dependabot-automerge.yml).

v4.10.1

Changed

• The README is Markdown now (README.md), with a logo, a recorded terminal
  demo, and the capability-indexed layout the crunchtools constitution
  (section XI) asks for. The reference that was in it moved, unchanged
  apart from structure, into docs/: hashing, reports, graphs, reading
  logs, library, install and philosophy each have a page. PyPI renders the
  new README as Markdown. For #54.
• docs/drivers.md moved to docs/internal/drivers.md: it is for
  contributors.
• The rpm and deb install README.md under /usr/share/doc/petit in place
  of README.

Added

• docs/history.md: where petit has lived since 2009, from the Perl
  script lt through eyemg, Google Code, Fedora, EPEL, Debian and Ubuntu
  to crunchtools, with sources, and what has been written about it.
• docs/demo/petit.tape and the Containerfile that renders it, with petit
  installed from the signed apt repository.

v4.10.0

Added

• max_identifiers on analyze_text, analyze_lines, hash_text and
  hash_lines. For JSON records, identifier-shaped strings (PROJ-1234,
  "10234", #42, a UUID or a git SHA) leave the fingerprint as <ID>
  and are listed in the new Group.identifier_fields (JSON Pointers) and
  Group.identifiers (one row per record, up to that many per group; a
  record past that keeps its identifiers in its fingerprint). 200 Jira
  issues that differ only in their keys used to be 200 groups; they are now
  one, with all 200 keys. Off by default. A record with more than four
  identifier fields keeps them in its fingerprint, so a list never becomes
  most of what it replaced. For mcp-trentina#173.
• petit --hash --identifiers N does the same from the command line,
  printing the identifiers on a line under each group.
• pull_identifiers() and IDENTIFIER, the same step on a single JSON
  value, for callers that fingerprint JSON themselves.
• The Fingerprints workflow archives every capture to the public
  crunchtools/petit-captures
  repo, under <date>/<run>/<release>/. That includes the whole journal of both boots, scrubbed and
  xz-compressed, not just the cut-out reboot, so stop words and corpora can
  be retuned from real logs later. refresh.py capture writes these under
  raw/.

v4.9.0

Fixed

• words.stopwords matched its words anywhere inside a word, not as the
  whole word, so --wordcount cut them out of other words. "target"
  became tar#, "command" became comm#, and 55 of the listed words
  didn't even stop themselves ("there" became t#e). Each word now
  matches only as a whole word, in any case, with punctuation (not a
  scrubbed number) around it allowed.
• hash.stopwords let through noise that differs from one boot or machine
  to the next, measured against two reboots of every supported release and
  one real-hardware RHEL 10 reboot:
  • UUIDs in upper case, or with a group made only of letters
  • vfat volume serials (F96D-44AD) and 0x hex values
  • Python tempfile and snap mount names
  • negative numbers and decimals
    Keys from two boots of the same image now agree more closely (69
    unstable keys down to 43). Of the fingerprint lines the real laptop
    reboot missed, the ones missed only through normalisation went from 10
    down to 1.

Changed

• --wordcount also drops English one- and two-letter words, bare
  punctuation, empty audit fields (addr=?), whole UUIDs, and systemd's
  lifecycle words ("Started", "Stopped", "Reached", "target",
  "Deactivated", "successfully" and so on). These filled the top of every
  modern reboot's count.
• daemon.stopwords rules for classic syslog pseudo-daemons ("last
  message repeated", "-- MARK --") now match only the whole field.
• Removed two hash.stopwords rules that could never match: [a-f]{16}
  and a literal MAC address.

v4.8.0

Added

• Reboot fingerprint corpora for current Linux (#37): el8, el9, el10
  (captured on AlmaLinux), fedora44, debian12, debian13,
  ubuntu22.04, ubuntu24.04, ubuntu26.04, opensuse-leap16, arch
  and alpine. The 2009–2011 corpora never matched a systemd journal, so
  --fingerprint did nothing on a modern host. Each corpus has a second,
  independent reboot in test/data/verify/ that must name it first. Rocky
  and CentOS Stream verify the elN corpora, and Fedora 43 and Alpine 3.23
  verify the newest release's corpus.
• tools/fingerprints/refresh.py and the weekly fingerprints.yml
  workflow keep the reboot corpora current, the way Dependabot keeps
  dependencies current (#37). Each supported release is booted from its
  own cloud image under QEMU/KVM and its reboot captured and scrubbed. A PR
  opens when a release is new, when its reboot no longer matches its
  corpus, or when it reaches end of life.

Removed

• petit/data/fingerprint_library/, the 2009–2011 per-machine reboot logs
  the old corpora were merged from by hand. Nothing read them. Corpora now
  come from tools/fingerprints/refresh.py, and each one's independent
  second capture is kept in test/data/verify/.

v4.7.0

Changed

• Fingerprint corpora are voted on rather than tried largest first (#49).
  A corpus is present when more than 31% of its lines are, as before, and
  at least five of them. Among those present, the one that best accounts
  for the input wins. The score is a weighted F1 in which a line many
  corpora share counts for little, so the lines one distribution alone
  logs decide. When nothing in the input tells two corpora apart, both
  are named in one label, a|b, rather than one being guessed. The
  winner's lines are then set aside and the rest vote again, so two
  reboots in one log both collapse.

Fixed

• A rhel5 reboot was also labelled rhel4-reboot.fp, and an ubuntu9.04
  reboot ubuntu10.04-reboot.fp. The larger corpus claimed the reboot
  first, then the right one matched what was left.

Added

• Analysis.fingerprint_scores lists a FingerprintScore (name,
  detection, identity) for every corpus that was present, so a caller can
  see the vote behind fingerprints_matched. -v logs the same.

v4.6.0

Added

• Native packages (#48): a noarch .rpm for RHEL 8, 9 and 10 and their
  rebuilds, Fedora, Amazon Linux 2023 and SUSE, and an all .deb for
  Debian 12 and 13 and Ubuntu 24.04 and 26.04. Each release attaches them,
  and the signed dnf and apt repository at crunchtools.github.io/packages
  serves them, so dnf upgrade and apt upgrade pick up new versions.
  petit installs to /usr/lib/petit and runs on whichever Python 3.11 or
  newer the system has, so RHEL 8 and 9 use their python3.12 package.
• CI tests Python 3.11 through 3.14 (#51).

v4.5.0 — stream input in bounded memory

Added

• analyze_lines() and hash_lines() take lines one at a time — an open
  file, a pipe, a generator — so an embedder can analyse a log without
  reading it into a string (#38).

Changed

• petit streams its input in bounded memory (#38, replacing #5). It used to
  read the whole input into one string, build an entry for every line and
  keep every entry in its groups: 1.4 GB of memory for a 96 MB, 1M-line
  secure log. It now keeps each group's count and a few samples, and the
  same log takes 28 MB for --hash and every graph and 36 MB for
  --wordcount, and still 28 MB for a 384 MB, 4M-line log. Output is byte-identical
  for every mode and fixture.
• A file is read in passes: every framer surveys it line by line, the
  drivers vote on records sampled across the whole of it, then it is
  parsed. Framing and driver choice are exactly what they were.
• A pipe is read once. petit holds its first 4 MB (HEAD_CHARS, the same
  as the JSON framer's limit); a pipe that ends there is read exactly like
  a file. A longer one is framed and parsed by what its first 4 MB chose,
  and a later record the driver can't read falls back to RawEntry on its
  own rather than sending the whole input back to be re-read.
• --graph finds its range and counts in one pass, keeping counts per
  time bucket and merging buckets once they are too fine to be drawn.
• --hash is about 25% faster and --wordcount more than twice as fast:
  Filter.scrub() built a debug log message for every stopword on every
  key even when debug logging was off, and --wordcount scrubs each
  distinct word once.

Fixed

• The container image's version label said 3.1.1.

v4.4.1

Fixed

• SyslogEntry and SecureLogEntry accept fractional seconds, so
  journalctl -o short-precise (Sep 23 16:55:34.125733) is read as
  syslog instead of falling back to RawEntry (#45). The clock column still
  has to be a whole HH:MM:SS, optionally followed by . and 1-9 digits;
  10:00 and x10:00:01 are still rejected (#2).
• RSyslogEntry found the clock by splitting on -, which only works west
  of UTC: any +hh:mm offset or Z raised ValueError, and so did a
  fraction of anything but six digits, and one such line sank the whole log
  to RawEntry (#11). It now matches the whole RFC 3339 timestamp, and its
  vote requires the whole first column to be one.
• Fixtures test19 (short-precise journal, same content as test16: its hash
  and graph output are identical) and test20 (RFC 3339 with +02:00,
  +05:30, Z, -0400 and 1-9 digit fractions).

v4.4.0

Added

• multiline framer (--framer multiline, tried automatically after
  json and message). A record starts at a line that begins with a
  timestamp, and every other line belongs to the record above it, so a
  journalctl stack trace, a Python traceback or a Java exception with
  Caused by: is one record. It recognises 16 timestamp formats: syslog
  and journalctl, RFC 3339/5424, Python logging, log4j/logback, Go, nginx,
  Apache error and access, Snort, Kubernetes/glog, the kernel, Unix time,
  java.util.logging, Tomcat, US dates, Redis and logback's time-only layout.
  journald's -- Boot ... -- lines are skipped.
• README section "How petit reads a log": the framer, entry driver and
  hash driver stages, what a record is, and a worked multi-line example.
  docs/drivers.md documents the framer's rules and every timestamp format.
• Fixtures test16 (journalctl), test17 (Python traceback) and test18 (Java
  exception).

Changed

• journalctl -o short output with multi-line messages used to fall back
  to RawEntry as a whole, because its continuation lines have no time:
  four days of a Fedora journal, 367,070 lines, graphed as nothing. It now
  frames as 365,928 records read by SyslogEntry. Any input the new
  framer claims groups per message instead of per line. It claims only
  input with an indented continuation line; everything else, including all
  of test01-test15, frames exactly as before.