Skip to content

feat: add score threshold to AnalyzerEngine #845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Jan 21, 2025
Merged

feat: add score threshold to AnalyzerEngine #845

merged 7 commits into from
Jan 21, 2025

Conversation

Pouyanpi
Copy link
Collaborator

@Pouyanpi Pouyanpi commented Nov 7, 2024

Description

Currently the AnalyzerEngine uses default_score_threshold of 0. This PR adds score_threshold to SensitiveDataDetectionOptions so that the user can set it in the config.yml. The default value is set to 0.2 but it requires further experimentation.

Example configuration:

rails:
  config:
    sensitive_data_detection:
      input:
        score_threshold: 0.4
        entities:
          - PHONE_NUMBER
          - EMAIL_ADDRESS
          - IN_PAN
          - IN_AADHAAR

      output:
        entities:
          - PHONE_NUMBER
          - EMAIL_ADDRESS
          - IN_PAN
          - IN_AADHAAR

Related Issue(s)

Checklist

  • I've read the CONTRIBUTING guidelines.
  • I've updated the documentation if applicable.
  • I've added tests if applicable.
  • @mentions of the person or team responsible for reviewing proposed changes.

@Pouyanpi Pouyanpi added the enhancement New feature or request label Nov 7, 2024
@Pouyanpi Pouyanpi self-assigned this Nov 7, 2024
@Pouyanpi Pouyanpi added the good first issue Good for newcomers label Nov 7, 2024
@Pouyanpi Pouyanpi mentioned this pull request Nov 7, 2024
Closed
4 tasks
@Pouyanpi Pouyanpi force-pushed the fix/presidio-min-threshold branch from 5b77022 to addad9d Compare November 7, 2024 12:07
@Pouyanpi Pouyanpi marked this pull request as ready for review November 10, 2024 10:35
@Pouyanpi Pouyanpi removed the good first issue Good for newcomers label Dec 3, 2024
@Pouyanpi
feat: add score threshold to AnalyzerEngine
166544c 
feat: add score_threshold parameter to _get_analyzer

- Added score_threshold parameter to _get_analyzer function with type hint.
- Included validation to ensure score_threshold is a float between 0 and 1.
- Updated detect_sensitive_data to pass score_threshold to _get_analyzer.
@Pouyanpi
feat: add score_threshold to SensitiveDataDetectionOptions
9a4a13a
@Pouyanpi
fix: replace assert with ValueError for source validation
3a61c90
@Pouyanpi
feat(docs): add score threshold section to handle false positives
626744d 
fix
@Pouyanpi
fix(docs): correct broken link in presidio.md
8bb8a60
@Pouyanpi Pouyanpi force-pushed the fix/presidio-min-threshold branch from addad9d to 8bb8a60 Compare December 3, 2024 08:15
@Pouyanpi Pouyanpi added this to the v0.12.0 milestone Dec 6, 2024
@Pouyanpi Pouyanpi added status: waiting confirmation Issue is waiting confirmation whether the proposed solution/workaround works. status: in review labels Jan 8, 2025
@Pouyanpi Pouyanpi force-pushed the fix/presidio-min-threshold branch from 9ec17dd to 7e4a3a6 Compare January 9, 2025 11:39
@Pouyanpi Pouyanpi force-pushed the fix/presidio-min-threshold branch from 7e4a3a6 to 883639f Compare January 9, 2025 11:47
@Pouyanpi Pouyanpi mentioned this pull request Jan 9, 2025
Merged
4 tasks
schuellc-nvidia
schuellc-nvidia approved these changes Jan 13, 2025
View reviewed changes
Copy link
Collaborator

@schuellc-nvidia schuellc-nvidia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have much context here, but from what I can see it looks good to me.

@Pouyanpi Pouyanpi merged commit f7db90b into develop Jan 21, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cparisien cparisien cparisien approved these changes

@schuellc-nvidia schuellc-nvidia schuellc-nvidia approved these changes

Assignees

@Pouyanpi Pouyanpi

Labels
enhancement New feature or request status: in review status: waiting confirmation Issue is waiting confirmation whether the proposed solution/workaround works.
Projects
None yet
Milestone
v0.12.0
Development

Successfully merging this pull request may close these issues.

bug: PII Filter Incorrectly Masks the Word 'individual' as Sensitive Data
3 participants
@Pouyanpi @cparisien @schuellc-nvidia