Add an "exists" check for "not" condition in sigma rules

[jowg-amazon]

Description

Currently the not clause generates a finding if an ingested document does not match whatever is specified in the detection section. However, this also includes cases where that field itself doesn’t exist in the ingested document. This PR adds an additional _exists_ check to the query so that if the field itself is not present in the ingested document, a finding is not generated incorrectly.

Example:
OLD: not selection1 -> NOT field1:val1
NEW: not selection1 -> NOT field1:val1 AND _exists_: field1

Additionally, distributes the NOT across parenthesis i.e. applies De Morgan's Theorem.

Example:
OLD: NOT(a and b)
NEW: (NOT a and NOT b)

Issues Resolved

#854

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[eirsep]

why is the applyDeMorgans always false? do we even need it??

[jowg-amazon]

applyDeMorgans starts off as false but we need it since we are recursively traversing the tree. It gets set to true (ie. we need to apply deMorgans) only if there is an AND or OR condition when it is an ancestor of a NOT.

Ex.

1. NOT A OR B -> do not apply deMorgans
2. NOT(A OR B) -> need to apply deMorgans

• This is because when we append the AND exists:field to the query, if we don't do deMorgans first but directly apply it directly to the query, an incorrect query would be constructed.

Ex.
Incorrect: NOT(A OR B) -> NOT(A AND exists:A OR B AND exists:B) -> NOT A OR exists:A AND NOT B OR exists:B
Correct: NOT(A OR B) -> NOT A AND NOT B -> NOT A AND exists:A AND NOT B AND exists:B

[eirsep]

are both these newly added bool params even necessary?
can't we just set them as necessary for convertConditionNot

[jowg-amazon]

see above

[codecov]

Codecov Report

Attention: Patch coverage is 71.73913% with 26 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (main@689760e). Click here to learn what that means.

Files	Patch %	Lines
...ecurityanalytics/rules/backend/OSQueryBackend.java	61.90%	15 Missing and 9 partials ⚠️
.../securityanalytics/rules/backend/QueryBackend.java	93.10%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #852   +/-   ##
=======================================
  Coverage        ?   25.02%           
  Complexity      ?     1044           
=======================================
  Files           ?      277           
  Lines           ?    12753           
  Branches        ?     1391           
=======================================
  Hits            ?     3192           
  Misses          ?     9288           
  Partials        ?      273           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eirsep]

check build and test CI

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-852-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 656a5fecbc2e09212b2d621c86b525fcbf9e4086
# Push it to GitHub
git push --set-upstream origin backport/backport-852-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-852-to-2.x.

[jowg-amazon]

CI's passed successfully

[opensearch-trigger-bot]

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.11 2.11
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.11
# Create a new branch
git switch --create backport-852-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 656a5fecbc2e09212b2d621c86b525fcbf9e4086
# Push it to GitHub
git push --set-upstream origin backport-852-to-2.11
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport-852-to-2.11.