feat(pii): bundle multilingual spaCy models in precheck image#29
Merged
Conversation
Adds Spanish, French, German and Mandarin Chinese spaCy models to the precheck Docker image so the PII analyzer is ready to be configured for non-English traffic (TASKS.md §3.5a). The image build now runs a smoke script that loads every model and fails the build if any one errors, and the same smoke covers a new pytest module gated by a `multilingual` marker. Local run (python 3.14, spaCy 3.8.13) — cold load per model: en_core_web_sm 308 ms es_core_news_sm 99 ms fr_core_news_sm 1208 ms de_core_news_sm 252 ms zh_core_web_sm 1022 ms total ~2.9 s No runtime code is wired to the new models yet — that is 3.5b. Refs: GOV-585
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
es_core_news_sm), French (fr_core_news_sm), German (de_core_news_sm) and Mandarin (zh_core_web_sm) spaCy models to the precheck Docker image.scripts/smoke_multilingual_pii.py— loads every model and prints per-model cold-load times; the Dockerfile runs it so the image build fails if any model is broken.tests/test_multilingual_models.pygated by a newmultilingualpytest marker; runs inside the precheck container (or any env where the models are installed) and skips cleanly otherwise.supported_languages=["en"].Load-time impact
Local run (python 3.14, spaCy 3.8.13), per-model cold load:
en_core_web_smes_core_news_smfr_core_news_smde_core_news_smzh_core_web_smThese only register when the NLP engine is extended to actually load them (3.5b); today the service startup path is unchanged because
build_presidio()still requests English only. Image-build time increases by the download + install of the four new small models (~70 MB on disk combined).GovernsAI Tracker issue
GOV-585
Reviewers
Tagging Atlas — precheck change, required per SDLC rules.
Test plan
python scripts/smoke_multilingual_pii.pyloads all 5 models locally (see table above).pytest tests/test_multilingual_models.py— 9 passed.RUNstep.docker run --rm <image> python scripts/smoke_multilingual_pii.pyinside the built precheck image (spot check by reviewer).Out of scope (follow-up tasks)
NlpEngineconfig + per-org language selection.