Skip to content

Rizzo PII 2.0.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 08 Aug 17:31
· 9 commits to main since this release
v2.0.0
aa7d133

Rizzo PII 2.0.0

PII anonymization that runs entirely on your machine, aimed at Italian legal documents. First
release since 1.0.0, and the major bump reflects how much moved: the app is now something you can
actually ship (desktop bundles, Docker, an open HTTP API), the deterministic regex+checksum net
recognizes identifiers in the shapes real documents print them in, and the synthetic data stopped
teaching the model structural shortcuts.

Model: rizzo-pii-0.3B v1.5.0 (app and model are versioned separately).

Highlights

  • Tag filter. Every one of the 23 tags can be switched off individually: a disabled tag is
    still detected but left in clear instead of being replaced. That is what you want when you need
    to compare the amounts in a contract, or keep age and gender readable in a clinical case. The
    legend (IT/EN descriptions plus examples) lives behind the 🏷️ icon in the header, and the choice
    persists in prefs.json. The reversible dictionary has its own switch: turn it off and the
    anonymization is final — no mapping is built, and the entity segments lose the field the
    dictionary could be rebuilt from.
  • The HTTP API is open. The same process is a plain HTTP service, so it can sit in a pipeline
    as a sidecar rather than being driven by hand: GET /health (readiness without inference, 503
    until the model is loaded), POST /analyze (JSON text or a .pdf/.md/.txt upload),
    POST /pdf, POST /preview, POST /pdf/preview, GET /settings, GET/POST /config. 
    curl -X POST localhost:5005/analyze -H 'Content-Type: application/json' \
     -d '{"text": "Mario Rossi, IBAN IT60X0542811101000000123456"}'
curl -F "file=@atto.pdf" localhost:5005/pdf -o atto_anonimizzato.pdf
  • The PDF is rendered on screen, before and after. Both columns show the document, not just
    its text: server-side rendering with PyMuPDF, lazy and cached pages (a 200-page PDF does not cost
    200 renders), documents held in an in-memory LRU and never written to disk. After
    anonymization a third view appears next to Preview/Text: the same document redacted, with
    placeholders where the PII was.
  • Download the anonymized PDF (POST /pdf). If the input was a PDF you get that document back
    truly redacted — the text leaves the content stream, no black box drawn on top — with the
    layout intact. Metadata, XMP, annotations, form-field values and bookmark titles are scrubbed
    too; embedded attachments are dropped. Two cases can leave a value in clear and both raise a
    warning (X-PII-Residual, X-PII-Skipped). The dictionary never crosses the wire in either
    direction.
  • Run it as a container. docker build -t rizzo-pii . then
    docker run --rm -p 127.0.0.1:5005:5005 rizzo-pii (or docker compose up -d --build). The model
    ships inside the image and HF_HUB_OFFLINE=1 is set at runtime: the container works with no
    network at all, which is the entire point of the project. gunicorn with 1 worker and 4 threads,
    because every worker is another ~1.2 GB copy of the model.

Accuracy: better where the data was thin

The model was retrained up to v1.5.0 with the changes that matter for the under-represented
tags — the open, high-variability classes and the rare ones, which is where the earlier checkpoint
was noisiest:

  • training now supervises all subwords of an entity instead of the first one only, so long
    codes and multi-word entities stop being learned half-way;
  • entity-level P/R/F1 at every eval instead of token accuracy alone, so a tag that regresses is
    visible while training rather than at the end;
  • the training pool is the union of the local synthetic data with the published Hugging Face
    dataset, and runs go two epochs — the validation loss of the 1.2.0 run was still falling when it
    stopped;
  • more, and more varied, source material for the classes that were thin: homocoded fiscal codes,
    new DOCID shapes (CIG, CUP, insurance policy, employee number), +60 offline legal templates,
    names no longer written in a single order, and a fixed augmenter that used to drop the second
    fragment inside the first.

On the largest validation set — validation_clean, 29,297 rows and 971,204 entities, no
truncation — v1.5.0 gains +22.2 points of micro-F1 over the previous checkpoint
(0.7778 → 0.9999), +25.8 points of macro-F1 (0.7423 → 0.9999) and token accuracy
0.9160 → 0.99995. The per-tag jumps are largest exactly where the data used to be thin:
AMOUNT +78.7, DOCID +71.9, TIME +65.9, CREDITCARDNUMBER +58.1, GENDER +49.2,
EMAIL +36.2, ORG +32.7. Twenty-one of 22 tags now sit at ≥ 0.9998.

Read that figure for what it is: that set is the distribution v1.5.0 was built on, so what it
mainly proves is that the older checkpoint did not know this taxonomy. On the neutral held-out
benchmark (validation_real, 7,000 real Italian rows) the two checkpoints are equivalent
(micro-F1 0.9887 vs 0.9886). The defensible gain there is masking coverage: 60 PII tokens left
in clear instead of 72 out of 54,093 — 99.89% coverage against 99.87%.

Full head-to-head, per tag, in the model's own
EVAL_v1.5.0_vs_main.md.

Correctness

  • _merge() was quadratic: 111 s → 0.4 s on a document with 40,000 entities, with identical
    output verified over 60,000 random cases and 48 constructed ones.
  • The regex net now reads IBANs written in groups, homocoded fiscal codes, credit cards, phone
    numbers and plates written with hyphens — the formats documents actually use.
  • Entity spans are aligned to word boundaries.
  • Pasting from Excel/TSV no longer destroys column alignment (#54).
  • "Clear" now really clears the dictionary (MAP + localStorage). Previously names, fiscal codes
    and IBANs stayed on disk in clear and could resurface in the next session.
  • A busy port is detected before the model loads (exit code 76), and the Tauri splash offers a
    host/port form instead of hanging.
  • Tauri: with host localhost the backend was never recognized.

Engineering

The regex+checksum net was extracted into detectors.py, testable in isolation, and the unittest
suite runs in CI on main and on every pull request.

Full detail, with the reasoning behind each change, in docs/CHANGELOG.md.

What's Changed

  • feat: configurable host/port with port-conflict recovery UI by @qzagarese in #2
  • [codex] add reproducible TARGA generator and stress test by @fabioscialanga in #3
  • perf(app): _merge() era quadratica, 111 s su un documento lungo by @pieronoviello in #64
  • Template scrivibili da un LLM locale (endpoint OpenAI-compatibile) by @maxxflyer in #20
  • contribute_dataset: scrittura incrementale invece di accumulo in RAM by @maxxflyer in #23
  • Una struttura, una riga: il volume ripetuto non è un contributo by @maxxflyer in #24
  • Rifiuta le label fuori tassonomia nei contributi by @maxxflyer in #26
  • Forme nuove sotto DOCID: CIG, CUP, polizza, matricola INPS by @maxxflyer in #28
  • fix(app): allinea le span delle entità ai confini di parola (rif. #11) by @Umberto65UT in #35
  • fix(app): detector regex DATE e DOCID contro la frammentazione delle … by @marcomodonesi in #38
  • feat(data_pipeline): offline legal template bank (+60 templates, no Gemini) by @marco88cappelli in #42
  • fix: "Pulisci" non cancellava il dizionario PII by @pieronoviello in #48
  • Aggiunta manuale delle PII + detector DATE soft (segue #10) by @FrankTheRope in #49
  • fix: la porta poteva risultare libera mentre era occupata by @pieronoviello in #51
  • fix(tauri): con host localhost l'app non riconosceva mai il backend by @pieronoviello in #52
  • fix(data): il secondo frammento dell'augment cadeva dentro il primo by @pieronoviello in #53
  • Codice fiscale omocodico: riconoscimento, sintetico e test by @tonytonycoder11 in #56
  • fix: preserve column/row alignment when anonymizing multi-row/tab-separated (Excel paste) input (#54) by @not-knope in #57
  • fix(app): detect IBAN, CF, card and phone in the formats used in real documents by @fabriziosalmi in #58
  • fix(app): la targa scritta col trattino (AB-123-CD) restava in chiaro by @pieronoviello in #62
  • fix(data): i sintetici scrivevano il nome solo come "Nome Cognome" (issue #40) by @pieronoviello in #63
  • [Add] docker-compose.yml for the webapp container by @andreampiovesana in #82

New Contributors

Full Changelog: v1.0.0...v2.0.0

Contributors

qzagarese, fabriziosalmi, and 10 other contributors
Assets 6
Loading