v0.5.3

Fixed

• A seeded run no longer re-issues a pseudonym the seed already spent.
  Anonymizer.from_mapping resumed each counter at the size of the mapping
  table, but the counters also advance for names that are never stored, so a
  mapping of 814 objects can hold OBJ-0822. Anonymizing the second member
  of an Active-Active pair with --seed-mapping therefore gave its new
  objects numbers the first member had used — OBJ-0817 designated two
  different objects across the two copies, so correlating them merged two
  distinct objects rather than merely missing a link. The user, serial, FQDN
  and e-mail counters carried the same shortcut. Counters now resume past the
  highest number actually issued.

v0.5.2

Fixed

• A directory member is never renamed. The rule that member names are
  rewritten on their last segment only — so a user named cli cannot turn
  tmp/cli/ into tmp/user83115/ — did not cover the member that is the
  directory, whose last segment is its own name. On a real PA-440,
  var/log/sa (the sysstat directory) came out as var/log/user001 because
  sa is a genuine PAN-OS account, while the 34 files under it kept
  var/log/sa/…: the archive declared an empty directory and no longer
  declared the one holding the files.

v0.5.1

Fixed

• The free-text pass no longer stalls on the vendor UI catalog. Real TSFs
  ship opt/pancfg/mgmt/tmp/ui_content/ui_predefined.js.gz — 29 MB of
  minified JavaScript whose strings spell <description> and close it
  <\/description>, so 38 280 tags never close. The lazy
  <tag>(.*?)</tag> used to find free-text fields rescanned the whole payload
  for each one: ~30 minutes per pass, matching nothing. Both halves now pair
  the delimiters in a single linear scan — 0.035 s on the same file, with
  output asserted identical to the regex it replaces. On a two-file batch of
  real archives this was 26 minutes of anonymize and 50 minutes of compare
  per job, entirely wasted.

v0.5.0

Added

• Free text is removed by default. The content of <description>,
  <comments>, <comment>, <login-banner> and the SNMP <location> — in
  every config (running, merged, archived, candidate, audit) — and of their
  set-format echo (description "…" in a show config dump) is replaced by
  REDACTED-FREE-TEXT. Measured on real archives: 70–75 % of those fields
  came through every pseudonym pass verbatim (1019/1363, 70/149, 516/587 on
  three trees), carrying exactly the attribution the tool exists to strip —
  people, companies, providers, ticket references — because none of it has a
  shape a pattern can recognise. After the change: 0 fields keep their
  content on the same three trees. One placeholder replaces each line of a
  field, so a multi-line description stays as many lines: no replacement ever
  contains a newline. Vendor containers (<predefined>, <threats>,
  <application-type>, the <global> catalog a candidate config embeds) are
  skipped — 55 % of all free-text fields on the corpus, prose that explains
  behaviour and names nobody. The set-format side is deliberately
  conservative: a line carrying anything else quoted (a JSON key, an escaped
  quote, a value that runs past the end of the line) is left intact rather
  than half-rewritten. Off with --keep-free-text (CLI), by unchecking
  Remove free text (UI) or redact_free_text=false (API); the choice is
  recorded in the mapping sidecar next to ip_seed.
• The compare mirrors it from the sidecar alone, with its own copy of the
  rule (never a call into the anonymizer): a removed field is an explained
  change, a field that kept its content is a warning
  (free_text_survivals in the summary, a KPI in the UI), and a placeholder
  in an archive whose sidecar does not declare the removal is a warning too.
  A line the two halves judge differently is still explained by the plain
  mapping, so a difference of judgement can never turn into unexplained noise.
• The compare's routing-coherence check now also proves coherence over
  time: var/log/pan/routed.log* (add/delete route events, plain or
  .gz-rotated), and the show routing protocol bgp loc-rib / rib-out and
  ospf dumplsdb (LSDB) CLI sections, feed the same route/nexthop/containment
  relations the static config and RIB snapshot already checked. Same
  conservative rule as the rest of the check: a line whose prefix or nexthop
  does not parse cleanly is skipped silently, never guessed at. No mapping,
  no anonymizer import — same code re-derives both trees independently.

Changed

• Public IPv4 pseudonyms now come from the same keyed-PRF prefix tree as
  private ones: one catch-all tree over 240.0.0.0/4, with the real top
  nibble folded into the PRF path seed and every bit down to the host octet
  tree-flipped. Two real addresses sharing a k-bit prefix (k ≥ 4) share the
  fake prefix to depth k, so public routing relations survive on the copy —
  the default route's nexthop stays inside its connected WAN /30, BGP/OSPF
  aggregates still contain their members (120 of 246 containment relations
  were lost on one real box under the previous per-/24 grouping), and
  prefixes announced mid-log stay coherent; masks never change. The
  compare's public_divergences count is now expected ≈ 0 (measured
  0 / 0 / 0 on three real trees that showed 101 / 5 / 0 before); it stays a
  counted number, never a hard error. The anti-reuse probe no longer hands
  out .0/.255 host octets. Minor-worthy: this changes what public
  pseudonyms look like, not what is mapped — a mapping sidecar written by
  the old scheme keeps its explicit pairs verbatim when seeding a new run
  (Anonymizer.from_mapping), only new allocations use the tree, and
  ip_seed rides the sidecar unchanged.

Fixed

• Routing-protocol names (bgp, ospf, ospfv3, rip, bfd, pim,
  igmp, msdp, vrrp) joined BUILTIN_OBJECTS and are never identities:
  a real box with a service object named bgp had every
  > show advanced-routing bgp loc-rib-detail command echo rewritten to
  … SVC-0368 loc-rib-detail, which lost the BGP sections to any reader of
  the copy — caught by the routing-coherence check's dynamic view as 29
  missing routes on the anonymized side. Such a bare name now stays in
  clear when a customer genuinely uses it (it identifies nobody — the same
  trade www, lan and the English stopwords already make); compounds
  like bgp-peering-lyon are still mapped.

v0.4.0

Changed

• The compare's scan — every mapping key over every payload, which is both
  how a line is explained and how a leak is found — now runs as an
  Aho-Corasick automaton (new dependency: pyahocorasick) instead of three
  compiled trie regexes. The automaton reports candidates and each one is
  revalidated against the very same boundary assertions the regexes spliced
  inline, so the semantics are unchanged: longest key first at a position
  (backtracking to a shorter key whose boundary holds), pass order FQDNs →
  objects → numeric, the lowered-copy scan for case-insensitive keys with the
  re.IGNORECASE fallback where that copy cannot serve. Measured on a real
  38 MB log, the three passes cost 1.0 s against 4.1 s, and building the
  index over 26 828 keys 0.04 s against 0.40 s — paid once per worker
  process; end to end, the compare of a real tree went from 145 s to 61 s
  and of another from 131 s to 55 s (single worker). The trie regexes stay
  the reference implementation and the fallback wherever the C extension
  cannot be installed (compare.USE_AHOCORASICK); a test forces both paths
  and asserts they produce byte-identical apply output, identical
  find_leaks findings and an identical CompareReport on the mock
  archive, and the same was verified on three real archives (both trees
  each, 1.6 GB of text on one of them): zero divergence.
• The XML prescan runs on the same process pool as the rest of the job
  (TSF_ANON_WORKERS, --workers), and each XML is parsed once. A worker
  parses one config and reports the identities it holds — it is handed no
  Anonymizer and allocates nothing; the parent registers those findings in
  path order, in the document order each file yielded them, so the mapping is
  the sequential one whatever the worker count. Measured on 37 MB of
  synthetic configs: 1.51 s to 0.57 s at four workers, mapping byte-identical
  at 1, 4 and 8. The compare's XML structure check no longer builds a DOM to
  read a tag sequence either — 0.83 s and +276 MB down to 0.39 s and +13 MB
  per 22 MB pair, with the same verdict on every shape (namespaces, comments,
  processing instructions, entities, encoding declarations, unparseable
  documents).
• tsf-anonymizer serve binds 127.0.0.1 by default instead of 0.0.0.0
  (minor: changes what a bare serve is reachable from). A tool that
  handles un-anonymized archives and the mapping that reverses them should
  not be reachable off the box until an operator asks for it with
  --host; the compose default already published the container on
  loopback (TSF_BIND_ADDR), so this closes the gap for anyone running the
  CLI directly. The Dockerfile's CMD already passed --host 0.0.0.0
  explicitly (the container binds every interface inside its own network
  namespace; it is the host-side port publish that stays loopback-only via
  compose), so the container's behaviour is unchanged.
• Passes that cannot match are no longer run. The username, e-mail and
  hostname patterns are preceded by a check for the literal every one of
  their matches contains, and the frozen rewrite skips the serial fallback
  regex, whose work the text prescan has already done (the known-serial trie
  still runs, and an unfrozen anonymizer still discovers serials with it).
  Measured on a real archive: the literal is absent from 72 % of the text for
  hostname, 50 % for @ and 38 % for user, and the serial fallback cost
  1 212 ms per 31 MB of log for nothing. Output unchanged — byte-identical
  anonymized trees on the real corpus, and identical detection findings file
  by file.
• The case-insensitive tries (FQDNs and e-mail domains in the anonymizer, the
  same keys in the compare's apply and leak scan) now match a lowered copy
  of the text with a trie compiled without re.IGNORECASE, and cut the
  output from the original text at those offsets. Measured on a real 31 MB
  log: the FQDN pass 2 096 ms to 1 262 ms, the compare's apply 6 440 ms to
  3 540 ms. Output is unchanged — byte-identical anonymized trees and
  identical integrity reports on eight real archives — and text holding one
  of the three codepoints where lower() and re.IGNORECASE disagree keeps
  the flag.
• Worker processes are now sized when each heavy phase starts, not once at
  startup. TSF_ANON_WORKERS / TSF_COMPARE_WORKERS are floors describing a
  machine whose every archive slot is busy; a phase that runs while fewer
  archives can — a lone upload, or a batch of one firewall, which is a chain
  and runs one archive at a time — takes the idle share of the cores instead,
  up to the core count. Setting either variable pins the count as before. The
  job log records what each phase settled on.
• IP pseudonyms now preserve prefix structure (minor: changes what fakes
  look like). Private addresses stay inside their own RFC 1918/CGNAT class
  with the host octet kept (10.1.2.3 → 10.x.y.3), driven by a keyed PRF
  prefix tree whose key travels in the mapping sidecar as ip_seed; every
  other address maps into 240.0.0.0/4 (class E), one fake /24 per real /24.
  Same real prefix → same fake prefix: subnets, static and dynamically
  learned route destinations, LSDB entries and nexthops stay mutually
  coherent — across files and across TSFs seeded from the same mapping. An
  address can no longer map to itself; a structural fake that would repeat
  a used value probes within its /24. The compare now applies mapping
  keys that collide with pseudonym values instead of dropping them (they
  are still excluded from the leak scan and reported as collisions).
• Archive extract and repack are now isal-backed when the isal package
  (PyPI project python-isal) has a wheel for the platform, falling back to
  stdlib gzip/zlib otherwise. Measured on a real 565 MB TSF, single core:
  extract 92s → ~35s, repack 66s → ~25s. isal's encoder only offers levels
  0-3 (not zlib's 0-9); level 3, its slowest/best-ratio setting, is used
  throughout, because even at level 3 isal still outruns stdlib zlib at
  level 6 (the level this module already uses over the default 9) by 2-3x on
  this kind of text — there is no speed left to buy by dropping a level, so
  the ratio isn't traded away as it is for zlib's 9 → 6. The trade that is
  made: isal's output runs somewhat larger than zlib's for the same content
  (~15-20% measured on synthetic archives) — the outer .tgz and any
  rewritten .gz member compress worse, never differently once decompressed.
  isal's GzipFile cannot be trusted with the backward seek extract_archive
  performs (getmembers() scans forward to the end, then extractall()
  restarts near the beginning) — confirmed misdecoding after such a rewind
  on isal 1.8.0 — so reading only ever drives it forward: the compressed
  archive is decompressed in one sequential pass to a plain temp .tar on
  the same volume, and ordinary (fast, unaffected) random-access tarfile
  reading happens against that. Writing has no such restriction (tarfile
  never seeks backward while it writes), so the repack path wraps isal's
  GzipFile directly as tarfile's own fileobj, exactly as tarfile
  itself wraps stdlib's GzipFile for mode="w:gz". Decompressed bytes are
  asserted identical to the stdlib path on the synthetic mock archive,
  outer and inner .gz alike.

Fixed

• A non-UTF-8 byte in an XML payload no longer fails the whole file's
  comparison report. Text is decoded with errors="surrogateescape", so such
  a byte survives as a lone surrogate; both expat.Parser.Parse and
  ET.fromstring re-encode a str argument to UTF-8 before parsing, which
  raises UnicodeEncodeError for a surrogate rather than the parser's own
  error — uncaught, it escaped _xml_structure and turned the file's report
  into a bare "comparison failed" instead of the graceful xml_structure = "unparseable" verdict a truncated or otherwise broken document already
  gets. _xml_structure now catches UnicodeEncodeError alongside
  expat.ExpatError.
• A bare common English word is no longer an identity, in any category:
  brute-force login guesses (failed authentication for user 'install',
  'up', 'inventory') and config entries genuinely named data or
  bytes were pseudonymized and then replaced corpus-wide, destroying
  command echoes (> show chassis inventory), status vocabulary
  (Connection status: up), fixed counter text (size (bytes),
  Resource monitoring sampling data) and the install verb of
  opt/panrepo/logs/history.log and of audit.log on real archives. A new
  _ENGLISH_STOPWORDS guard sits on the two allocation choke points
  (anon_user, register_named_object). Trade-off, documented in the
  invariants: an account or object named exactly a bare English word stays
  in clear — it identifies nobody, and any spelling with a digit, dot or
  hyphen (jmartin, data-01) is unaffected.
• PAN-OS interface names are preserved on the FQDN route too: vlan.800 /
  tunnel.2 are FQDN-shaped, and a subinterface entry name registered
  through the FQDN branch rewrote every zone <member> holding it into
  hostNNN.anon.internal on a real TSF. anon_fqdn now refuses interface
  names, as register_named_object always did.
• An original address equal to an already-handed-out pseudonym is now
  mapped instead of skipped. With same-class fakes the old skip left a real
  private address in clear and outside the mapping — invisible to the leak
  scan, caught on a real TSF by the new routing-coherence check. The
  compare reports the string ambiguity as a collision, as before.
• The routing-coherence check scopes its verdict to the private relations
  prefix preservation guarantees; divergences that involve public space
  (aggregation beyond the per-/24 grouping) are counted and shown, not
  errors. It prefers .merged-running-config.xml (Panorama-managed boxes
  ship an empty <interface> section in...

v0.3.0

Added

• tsf-anonymizer mock-tsf: a deterministic synthetic TSF (fictional company,
  reserved address ranges) to try the tool on, reproduce bugs and generate the
  documentation — real archives are customer material and never enter the repo.
• docs/user-guide.md and docs/architecture.md; screenshots of the web UI
  generated automatically from the mock archive (make screenshots,
  scripts/docs-screenshots.py, Playwright).
• Project files for public release: Apache-2.0 LICENSE, SECURITY.md,
  CONTRIBUTING.md, CODE_OF_CONDUCT.md, issue and pull request templates,
  CODEOWNERS, Dependabot.
• GitHub Actions: lint + tests on Python 3.11/3.12/3.13 + Docker build + UI
  smoke test on every PR; CodeQL; a release workflow on v* tags that builds
  the wheel, publishes the GitHub release from this file and pushes the image
  to GHCR.
• Makefile with the everyday targets and .pre-commit-config.yaml.

Changed

• Tooling moves to uv: uv.lock, PEP 735 dependency groups (dev, docs),
  .python-version; the Dockerfile builds from the lockfile in two stages.
• The version lives in pyproject.toml only (uv version --bump …);
  tsf_anonymizer.__version__ reads it from the installed metadata.
• Ruff now also enforces import order, modern syntax and bugbear checks.

Changed

• Binary redaction is on by default in the web UI and the API
  (redact_binaries): a binary member that embeds mapped identifiers
  (sslvpn-task GpTaskStat records, wtmp/btmp/lastlog, sar headers,
  rule-hit-count.bin) ships as a marker instead. Every family but the login
  history has a text twin that is anonymized normally. The CLI keeps its
  --redact-binaries flag.