Use Phrases for Context matching

[dhruvmullick]

Instead of using words for context boost, would it be useful to allow for phrases too?

e.g. have a context hit (boost) if the phrase "zip code" is found in the vicinity of the entity.

[omri374]

Thanks @dhruvmullick. Although it is not provided out of the box, the context mechanism allows you to create your own context enhancement implementation. Replacing the LemmaContextAwareEnhancer with a different implementation would allow you to support phrases or substring matching instead of the existing lemma-based enhancement.

[omri374]

Not sure I follow. Could you please give an example?

[dristysrivastava]

So, I will go with the example given on presidio website only with a little modification

from presidio_analyzer import PatternRecognizer, RecognizerRegistry, AnalyzerEngine, Pattern

from presidio_analyzer.context_aware_enhancers import LemmaContextAwareEnhancer

# Define the regex pattern
regex = r"(\b\d{5}(?:\-\d{4})?\b)"
zipcode_pattern = Pattern(name="zip code", regex=regex, score=0.01)

# Define the recognizer with the defined pattern
zipcode_recognizer = PatternRecognizer(supported_entity="US_ZIP_CODE",
                                       patterns=[zipcode_pattern],
                                       context=["zip", "zipcode"])


registry = RecognizerRegistry()
registry.add_recognizer(zipcode_recognizer)
analyzer = AnalyzerEngine(
    registry=registry,
    context_aware_enhancer=LemmaContextAwareEnhancer(
        context_similarity_factor=0.85,
        min_score_with_context_similarity=0.8,
        context_prefix_count=2,  # how many words before the entity to match context
        context_suffix_count=0  # how many words after the entity to match context
    )
)


text = """
90210 is my zip code
my name is 12345
My zip code is 90210
"""

# Test
results = analyzer.analyze(text=text,language="en")
print(f"Result:\n {results}")

So here output is coming as

[type: US_ZIP_CODE, start: 54, end: 59, score: 0.86, type: US_ZIP_CODE, start: 1, end: 6, score: 0.01, type: US_ZIP_CODE, start: 33, end: 38, score: 0.01]

Although in 1st line of text zip code is there but since I have given context_suffix_count as 0 it is not matching it, so here my point is, is there any ML / intelligent solution or suggestion for this

[dristysrivastava]

And I have one more query, how to pass context as case insensitive

[omri374]

On case insensitive, you could consider to create a new ContextAwareEnhancer which inherits from LemmaContextAwareEnhancer and changes the logic in this method to match in a case insensitive way:
https://github.com/microsoft/presidio/blob/48d3556a609be8ae032d1c3a668243acfe4ec8db/presidio-analyzer/presidio_analyzer/context_aware_enhancers/lemma_context_aware_enhancer.py#L151

That's also likely where you'd create new logic to improve context handling. For example, you could skip matches if there's a \n, match based on substrings, use word embeddings to assess the similarity etc.

[dristysrivastava]

In our case the input text is unstructured, Can you please suggest how to use context words feature in that case.
#1043 (reply in thread)