Skip to content

v0.8.0 — overlapping cloud spans under one placeholder, LOCATION, quieter log redaction, UK UTRs and wrapped IBANs

Choose a tag to compare

View all tags
@qualityingrained qualityingrained released this 31 Oct 21:58
· 28 commits to main since this release
v0.8.0
d184309

euRedact 0.8.0 ships the euredact package on PyPI and on npm, from one tag. Every entry applies to both SDKs. It prepares the SDKs for the gen 14 cloud model, and fixes the rules engine.

Upgrading from 0.7.1

These changes can affect code that reads results:

  • Overlapping cloud spans are masked as one placeholder. From gen 14 the model can give one stretch of text more than one type. Overlapping spans are now masked together under one placeholder, so a result can have fewer placeholders than before. detections still lists every span with its own type. (#139)
  • Loopback and unspecified addresses are left in the text. 127.0.0.0/8, 0.0.0.0, ::1 and :: are no longer IP_ADDRESS / IPV6_ADDRESS. They exist on every machine and identify no one's device. (#136)
  • Fewer SECRETs on assigned values. A key: value / key=value is a SECRET only when the key, or the line before the value, names a credential. (#136)

Added

  • EntityType.LOCATION. A city, town, village, commune or district the document links to a person, returned by the cloud model from gen 13 on. It can now be named in exclude_types and compared against the enum. The rule engine never emits it. (#134)
  • UK Unique Taxpayer Reference. A UTR behind its label (UTR:, self_assessment_utr, "Unique Taxpayer Reference") is TAX_ID, ten digits with or without the space after the fifth. It is detected whether or not its check digit passes, in hinted and blind mode alike. (#137)

Fixed

Overlapping cloud spans (#139)

  • A special category no longer leaks through the placeholder. Before, a span starting inside another was masked from where the first ended: one value sat under two placeholders, the first named by the service's list order. Imam, listed as SENSITIVE_ATTRIBUTE and JOB_TITLE, became [SENSITIVE_ATTRIBUTE].
  • How the placeholder is chosen. A non-sensitive type now names the placeholder before MEDICAL_CONDITION, SENSITIVE_ATTRIBUTE or BIOMETRIC_REF, so Imam is [JOB_TITLE].
  • Excluding a type never uncovers text another enabled type covers.
  • Tokens round-trip. Tokens and referential labels cover the whole region, so restore() returns the original text.
  • Batches use the same masking.

Log text (#136)

  • Sizes: 48213377 bytes is no longer a PHONE.
  • Config values: model='/var/lib/…' and other config values whose key names no credential are no longer SECRET. DB_PASSWORD=…, config['api_key'] = '…' and "Your new password is: …" stay masked.
  • Text masked once: a [TYPE] placeholder is no context keyword, so [SECRET] no longer makes the values near it secrets. On the gateway's second pass over SDK-masked text, that had typed names, customer ids and whole JSON log lines SECRET. On the IT, HR and medical benchmark sets (650 documents), such spans fell from 17 to 1.

Allowlist in cloud mode (#141)

  • An allowlisted value inside a longer span the cloud tier returned stays visible: with allowlist=["ACME NV"], a span ACME NV Brussel becomes ACME NV [ORGANISATION_NAME]. The value is cut out as a whole word, the rest stays masked under the span's type, and the cut is in exempted. Rules detections and allowlist_domains keep whole-span matching.

Wrapped IBANs (#138)

  • An IBAN broken over a line between its groups, with or without indentation, is one BANK_ACCOUNT. Before, the second line was left visible and part of it read as a PHONE.

Changed

  • Documentation. The open-source SDK is euRedact Core and the cloud tier euRedact Cloud. The package names and imports are unchanged. The README states that the rule engine emits 29 types; OTHER is the cloud client's fallback for a label it does not know. (#135, #124)

Accuracy

Measured with make verify on the release tree, rules engine only, over 152,300 synthetic documents and 667,268 labelled values, excluding dates of birth.

precision recall F1 false positives
With country hints 99.9% 99.8% 99.8% 720
Blind (no countries) 99.8% 99.7% 99.7% 1,645
  • Dates of birth: 100.0% recall.
  • Against 0.7.1:
    • SECRET recall 90.12% → 90.62%, precision 95.96% → 96.69%;
    • IP_ADDRESS mistyped 11 → 4;
    • LICENSE_PLATE mistyped 2 → 0;
    • false positives 759 → 720 with hints and 1,684 → 1,645 blind.
  • Per type: under the API's names, in docs/corpus-results.md and .json.
Assets 2
Loading