Skip to content

feat(pii): bundle multilingual spaCy models in precheck image#29

Merged
Shaivpidadi merged 1 commit into
devfrom
sage/3.5a-multilingual-pii-models
Apr 23, 2026
Merged

feat(pii): bundle multilingual spaCy models in precheck image#29
Shaivpidadi merged 1 commit into
devfrom
sage/3.5a-multilingual-pii-models

Conversation

@Shaivpidadi

Copy link
Copy Markdown
Member

Summary

  • Adds Spanish (es_core_news_sm), French (fr_core_news_sm), German (de_core_news_sm) and Mandarin (zh_core_web_sm) spaCy models to the precheck Docker image.
  • Installs them in a separate layer so the English-only base stays cache-hot for unrelated builds.
  • Adds scripts/smoke_multilingual_pii.py — loads every model and prints per-model cold-load times; the Dockerfile runs it so the image build fails if any model is broken.
  • Adds tests/test_multilingual_models.py gated by a new multilingual pytest marker; runs inside the precheck container (or any env where the models are installed) and skips cleanly otherwise.
  • No runtime code is wired to the new models yet — that is 3.5b. Presidio's NLP engine still boots with supported_languages=["en"].

Load-time impact

Local run (python 3.14, spaCy 3.8.13), per-model cold load:

model cold load
en_core_web_sm 308 ms
es_core_news_sm 99 ms
fr_core_news_sm 1208 ms
de_core_news_sm 252 ms
zh_core_web_sm 1022 ms
total ~2.9 s

These only register when the NLP engine is extended to actually load them (3.5b); today the service startup path is unchanged because build_presidio() still requests English only. Image-build time increases by the download + install of the four new small models (~70 MB on disk combined).

GovernsAI Tracker issue

GOV-585

Reviewers

Tagging Atlas — precheck change, required per SDLC rules.

Test plan

  • python scripts/smoke_multilingual_pii.py loads all 5 models locally (see table above).
  • pytest tests/test_multilingual_models.py — 9 passed.
  • CI image build completes with the new smoke RUN step.
  • docker run --rm <image> python scripts/smoke_multilingual_pii.py inside the built precheck image (spot check by reviewer).

Out of scope (follow-up tasks)

  • 3.5b: extend Presidio NlpEngine config + per-org language selection.
  • 3.5c: multilingual recognizers for existing entity types (phone numbers, IDs).

Adds Spanish, French, German and Mandarin Chinese spaCy models to the
precheck Docker image so the PII analyzer is ready to be configured for
non-English traffic (TASKS.md §3.5a). The image build now runs a smoke
script that loads every model and fails the build if any one errors, and
the same smoke covers a new pytest module gated by a `multilingual` marker.

Local run (python 3.14, spaCy 3.8.13) — cold load per model:
  en_core_web_sm   308 ms
  es_core_news_sm   99 ms
  fr_core_news_sm 1208 ms
  de_core_news_sm  252 ms
  zh_core_web_sm  1022 ms
  total           ~2.9 s

No runtime code is wired to the new models yet — that is 3.5b.

Refs: GOV-585
@Shaivpidadi
Shaivpidadi merged commit e900775 into dev Apr 23, 2026
5 checks passed
@Shaivpidadi
Shaivpidadi deleted the sage/3.5a-multilingual-pii-models branch April 23, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant