Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented general filtering (replaces blame filtering) #28

Merged
merged 6 commits into from Oct 4, 2023

Conversation

abyss638
Copy link
Member

@abyss638 abyss638 commented Sep 25, 2023

BREAKING CHANGE

Fixes #13

Implemented general filtering for any field in a results object. This replaces blame filtering which filtered only by author emails in blame details.

The filter format is switched from plain text to YAML.

Here is an example of a filter file (from updated README.md):

# Lines beginning with # are interpreted as comments and ignored.
# Optional description for the filter.  If no title is specified, the filter file name is used.
description: Example filter from README.md

# Items in `include` list are interpreted as inclusion filtering rules. 
# Items are treated with OR operator, the filtered results includes objects matching any rule.
# Each item can be one rule or a list of rules, in the latter case rules in the list are treated with AND operator - all rules must match.
include:
  # The following line includes issues whose author-mail field contains "@microsoft.com" AND found in Java files. 
  # Values with special characters `\:;_()$%^@,` must be enclosed in quotes (single or double):
  - author-mail: "@microsoft.com"
    locations[*].physicalLocation.artifactLocation.uri: "*.java"
  # Instead of a substring, a regular expression can be used, enclosed in "/" characters.  Issues whose committer-mail field includes a string matching the regular expression are included.  Use ^ and $ to match the whole committer-mail field.
  - committer-mail: "/^<myname.*\\.com>$/"

# Lines under `exclude` are interpreted as exclusion filtering rules.
exclude:
  # The following line excludes issues whose location is in test Java files with names starting with the "Test" prefix.
  - location: "Test*.java"
  # The value for the field can be empty, in this case only existence of the field in 
  - suppression:

Field names must be specified as a JSONPath expression, substrings and Regex are supported as before.

The following shortcuts are supported (from updated README.md):

Shortcut Full JSONPath
author properties.blame.author
author-mail properties.blame.author-mail
committer properties.blame.committer
committer-mail properties.blame.committer-mail
location locations[*].physicalLocation.artifactLocation.uri
rule ruleId
suppression suppressions[*].kind

For location which represents a file location wildcards are supported:

  • ? - a single occurrence of any character in a directory or file name
  • * - zero or more occurrences of any character in a directory or file name
  • ** - zero or more occurrences across multiple directory levels

Added a Pytest UTs for filtering code.

FilterStats is extracted to a standalone file and simplified (removed few no longer needed counters).

'GeneralFilter' is based on an extracted BlameFilter.

@balgillo
Copy link
Contributor

Looks great, thanks for this!

A few comments to discuss/resolve

@balgillo
Copy link
Contributor

I think we should do something to help ease the transition for users, e.g. write a page about migrating from blame-filter to filter, check the --blame-filter argument and point people to the page. But that might be better as a separate PR to keep this one small.

README.md Outdated Show resolved Hide resolved
sarif/filter/filter_stats.py Show resolved Hide resolved
sarif/filter/filter_stats.py Outdated Show resolved Hide resolved
sarif/filter/general_filter.py Outdated Show resolved Hide resolved
sarif/filter/general_filter.py Outdated Show resolved Hide resolved
sarif/filter/general_filter.py Outdated Show resolved Hide resolved
sarif/filter/general_filter.py Outdated Show resolved Hide resolved
sarif/filter/general_filter.py Show resolved Hide resolved
@balgillo balgillo merged commit b6342df into microsoft:main Oct 4, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to remove specific entries
2 participants