What's the recommended process for improving Presidio's detection rates?

[omri374]

In very high level, the process for improving the PII detection rate with Presidio is the following:

1. Come up with a dataset with labels for PII entities to be used for evaluation.
2. Decide on an evaluation metric. In presidio-research we use F2.5 or F2 which give more weight to recall over precision.
3. Run and evaluate the vanilla Presidio (with the default spaCy model or with another spaCy model if more languages are required).
4. Calculate metrics and analyze false positive or false negative examples. See this notebook for example.
5. Decide whether results could be improved using rule-based approaches. Some examples:
   • Remove existing recognizers.
   • Modify existing recognizers for logic, patterns, confidence, which entities are being detected by which recognizer.
   • Add new logic (regex, deny-list, rule-based logic) which specifically addresses the entities in hand. See this doc to get started and this one for best practices.
   • Experiment with different configurations of these recognizers and how they should play together.
   • Replace the underlying spaCy/stanza model with another one. See more information here.
   • Connect to a 3rd party service like Text Analytics PII.
   • Add a new recognizer leveraging a ML model from a framework other than spaCy or Stanza (like flair).
6. Decide if model fine-tuning/training is necessary. If yes, either train/fine-tune a spaCy/stanza model, or use some other framework like transformers, flair or CRF.
7. Additional options for improvement:
   • Collect and label more data.
   • Create more synthetic data (e.g. the presidio-research data generator).
   • Train a model using weak supervision.
   • Separate entity detection into different models, each detecting a one entity (see this paper for inspiration).
   • Evaluate different fine-tuning and transfer learning techniques.
   • In structured/semi-structured data settings, the actual entity value could appear elsewhere in the data. In that case, ad-hoc recognizers could be used to automatically create deny-lists of entity values. For example, if a row contains first name, last name and free text, one could create an ad-hoc recognizer with the first and last names as an ad-hoc deny-list, and these would automatically be identified as PII if they exist in the free text.