feat: Add HuggingFaceNerRecognizer for direct NER model inference

[ultramancode]

Fixes #1833

Description

This PR adds HuggingFaceNerRecognizer, a recognizer that uses HuggingFace Transformers pipeline directly for NER, bypassing spaCy tokenizer alignment issues.

Why is this needed?

The standard approach using spaCy tokenizer with TransformersNlpEngine has alignment issues for agglutinative languages (Korean, Japanese, Turkish, etc.):

• Particles/postpositions attach to nouns
• spaCy tokenizer includes particles: "김태웅이고" (name + particle)
• NER model returns only the entity: "김태웅" (name only)
• char_span() alignment fails, causing entities to be skipped or incorrectly bounded

Solution

HuggingFaceNerRecognizer bypasses spaCy alignment by using HuggingFace pipeline directly.

Key Features

• Language-agnostic: Works with any HuggingFace NER model
• Direct inference: No spaCy tokenizer dependency for entity boundaries
• Extensible YAML Configuration: Implemented a dedicated Pydantic model HuggingFaceRecognizerConfig
  to strictly validate recognizer-specific parameters (e.g., model_name, device, label_mapping) directly from YAML, ensuring type safety and clear error messages.

Changes

• NEW: presidio_analyzer/predefined_recognizers/ner/huggingface_ner_recognizer.py
• NEW: tests/test_huggingface_ner_recognizer.py
• MODIFY: presidio_analyzer/input_validation/yaml_recognizer_models.py (Allow ML fields in config)
• MODIFY: presidio_analyzer/recognizer_registry/recognizers_loader_utils.py (Pass custom args to recognizers)
  • Updated argument preparation to correctly pass list-based arguments like supported_entities to the new recognizer.
• MODIFY: presidio_analyzer/predefined_recognizers/ner/__init__.py (export)
• MODIFY: presidio_analyzer/predefined_recognizers/__init__.py (export)
• MODIFY: presidio_analyzer/conf/default_recognizers.yaml (config, disabled by default)
• MODIFY: docs/analyzer/recognizer_registry_provider.md (Added configuration example)
• MODIFY: CHANGELOG.md

Verification Example: Side-by-Side Comparison

To prove the necessity of this feature, I configured a test environment where both the new HuggingFaceNerRecognizer and the default SpacyRecognizer are enabled for Korean. This allows for comparing their performance directly on the same text.

Test Configuration:

recognizers:
  - name: "HuggingFace NER KR"
    class_name: "HuggingFaceNerRecognizer"
    model_name: "Leo97/KoELECTRA-small-v3-modu-ner"
    supported_languages: ["ko"]
  
  - name: "SpacyRecognizer" # Intentionally enabled for comparison
    class_name: "SpacyRecognizer"
    supported_languages: ["ko"]
    enabled: true

Request:

curl --location 'http://localhost:3000/analyze' \
--header 'Content-Type: application/json' \
--data '{
    "text": "제 이름은 김태웅이고 서울에 살고 있습니다.",
    "language": "ko",
    "return_decision_process": true
}'

Result Analysis:
The response contains three entities. The first two are correct detections enabled by this PR, while the third is the incorrect "noise" produced by the default system.

• Result 1 & 2 (Success): The HuggingFaceNerRecognizer correctly splits and identifies "김태웅" (Kim Taewoong) as PERSON and "서울" (Seoul) as LOCATION.
• Result 3 (Failure): The SpacyRecognizer fails to handle Korean agglutination, capturing the phrase "이름은 김태웅이고..." (My name is Kim Taewoong and...) as a single PERSON entity.

[
    {
        "analysis_explanation": {
            "original_score": 0.9791115522384644,
            "pattern": null,
            "pattern_name": null,
            "recognizer": "HuggingFace NER KR",
            "regex_flags": null,
            "score": 0.9791115522384644,
            "score_context_improvement": 0,
            "supportive_context_word": "",
            "textual_explanation": "Identified as PERSON by Leo97/KoELECTRA-small-v3-modu-ner (original label: PS)",
            "validation_result": null
        },
        "end": 9,
        "entity_type": "PERSON",
        "score": 0.9791115522384644,
        "start": 6
    },
    {
        "analysis_explanation": {
            "original_score": 0.9564878344535828,
            "pattern": null,
            "pattern_name": null,
            "recognizer": "HuggingFace NER KR",
            "regex_flags": null,
            "score": 0.9564878344535828,
            "score_context_improvement": 0,
            "supportive_context_word": "",
            "textual_explanation": "Identified as LOCATION by Leo97/KoELECTRA-small-v3-modu-ner (original label: LC)",
            "validation_result": null
        },
        "end": 14,
        "entity_type": "LOCATION",
        "score": 0.9564878344535828,
        "start": 12
    },
    {
        "analysis_explanation": {
            "original_score": 0.85,
            "pattern": null,
            "pattern_name": null,
            "recognizer": "SpacyRecognizer",
            "regex_flags": null,
            "score": 0.85,
            "score_context_improvement": 0,
            "supportive_context_word": "",
            "textual_explanation": "Identified as PERSON by Spacy's Named Entity Recognition",
            "validation_result": null
        },
        "end": 23,
        "entity_type": "PERSON",
        "score": 0.85,
        "start": 2
    }
]

Production Configuration Tip

Although the HuggingFace recognizer functions independently, the Presidio Analyzer Platform requires a default NLP engine declaration for startup. For production environments where Spacy NER is not needed, I recommend explicitly disabling the default SpacyRecognizer to avoid duplication and performance overhead.

- name: "SpacyRecognizer"
  type: "predefined"
  supported_languages: ["ko"]
  enabled: false

Testing

Added unit tests covering:

• English person/location/organization detection

• Korean text with particles (agglutinative language demo)

• Multiple entity types

• Low confidence filtering

• Custom label mapping

• Empty text handling

• Model name validation

• Long text chunking & truncation safety

• I have reviewed the contribution guidelines

• I have signed the CLA (if required)

• My code includes unit tests

• All unit tests and lint checks pass locally

• My PR contains documentation updates / additions if required

[ultramancode]

@microsoft-github-policy-service agree

[ultramancode]

I’ll reuse PR #1805’s chunking logic once it’s merged to keep things consistent. Any other feedback is welcome.

[ultramancode]

Hi @omri374 I’ve updated the PR to adopt the BaseTextChunker chunking logic from #1805. When you have time, could you please review it?

[omri374]

Thanks! Sure, will do.

[ultramancode]

@omri374
I added a few more tests so the CI checks pass. Thanks!

[omri374]

Thanks! This is a great addition! I tested it on some of the newer OpenMed transformer models (OpenMed/OpenMed-PII-SuperClinical-Base-184M-v1) and it worked well. I did leave some comments on making it more robust to different uses. Happy to help with anything needed!

[Copilot]

Pull request overview

This pull request adds HuggingFaceNerRecognizer, a new recognizer that uses HuggingFace Transformers pipeline directly for NER inference, solving spaCy tokenizer alignment issues with agglutinative languages (Korean, Japanese, Turkish, etc.). The PR also implements an inspect-based dynamic parameter passing mechanism in the recognizer loader to support extensible YAML configuration.

Changes:

• Adds HuggingFaceNerRecognizer with direct HuggingFace pipeline integration and character-based text chunking
• Implements inspect-based parameter filtering in RecognizerListLoader._prepare_recognizer_kwargs for dynamic argument passing while maintaining backward compatibility
• Adds ML model configuration fields to PredefinedRecognizerConfig YAML validation model

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
presidio-analyzer/presidio_analyzer/predefined_recognizers/ner/huggingface_ner_recognizer.py	New recognizer implementation with HuggingFace pipeline integration, device handling, label mapping, and chunking support
presidio-analyzer/tests/test_huggingface_ner_recognizer.py	Comprehensive unit tests covering initialization, analysis, chunking, device parsing, error handling, and edge cases
presidio-analyzer/presidio_analyzer/recognizer_registry/recognizers_loader_utils.py	Modified _prepare_recognizer_kwargs to use inspect for smart parameter filtering, enabling dynamic kwargs passing
presidio-analyzer/presidio_analyzer/input_validation/yaml_recognizer_models.py	Added ML model configuration fields to PredefinedRecognizerConfig for HuggingFace and similar recognizers
presidio-analyzer/presidio_analyzer/predefined_recognizers/ner/init.py	Exported HuggingFaceNerRecognizer
presidio-analyzer/presidio_analyzer/predefined_recognizers/init.py	Exported HuggingFaceNerRecognizer
presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml	Added HuggingFaceNerRecognizer to default configuration (disabled by default)
docs/analyzer/recognizer_registry_provider.md	Added configuration example and tip for handling agglutinative languages
CHANGELOG.md	Documented the new feature

[ultramancode]

@omri374
Thank you for the detailed feedback — it was really helpful!
I’ve addressed all the comments and pushed the updates. Could you please take another look when you have a moment?

[omri374]

Great progress! another round of feedback, but just to emphasize the importance, we see this addition as strategic for next steps of Presidio, hence the detailed review. Thanks again!

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

[ultramancode]

@omri374
Thanks again for the thorough review! it really helped clarify my understanding of Presidio’s internals.
I’ve addressed the latest round of comments and pushed the updates. Could you please take another look when you have a moment?

[omri374]

Hi @ultramancode, there are some linting issues and one comment I left on the recognizer loading utils (making sure this didn't change any previously working behavior. Other than that, looks great! Thanks

[ultramancode]

Hi @omri374, thanks for the feedback.
I’ve replied to your comments—appreciate you taking the time!

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

[omri374]

Thanks! Sorry it took so long to finalize the review

[ultramancode]

Thanks @omri374! Your feedback helped me a lot. I really appreciate you taking the time to review this so carefully.
I learned a lot from your review.
Also, it looks like the approval may have been dismissed after I resolved the merge conflict in CHANGELOG.md.

[omri374]

Thanks @ultramancode! This is a great addition to Presidio, and we'd love for you to continue helping us shape the tool!