Change generated LogQL queries to be case insensitive by default #53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SigmaString values should be treated as case insensitive. But Loki treats equality as being case sensitive. To resolve this, the backend will now convert all SigmaStrings into regular expressions with the case insensitive flag set (?i). Such queries will be less performant - hence the introduction of a "case_insensitive" boolean flag (default: True). If the performance of a query is important and the rule is believed to accurately match the case of the log data, setting this to False will mean the generated query uses string equality instead (i.e., the query will be case sensitive).
Largely involved changing a=`b` to a=~`(?i)b`, but a few more involved changes (especially including the line filter functionality, which does not currently handle case-insensitive searches as effectively as it could). Added new test files that were basically identical to the previous files, but with the case_insensitivity flag set to False to ensure both cases are covered.
Switching most tests to case insensitive meant that there were fewer tests on strings, slightly reducing the coverage of the test suite.
kelnage
changed the title
Pull Request Test Coverage Report for Build 4990008363
💛 - Coveralls |
…ase-insensitive-default
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
String values in Sigma rules must be interpreted as case insensitive, but when checking equality with strings in LogQL, those values are treated as case sensitive. To workaround this and ensure broad compatibility, any such string values must be converted into regular expressions that have the case insensitive flag
(?i)at the start.However, such LogQL queries will inherently be slower than the same queries using string equality, and hence I have added a
case_insensitiveflag (default: True) in the backend, which can be used to toggle back to the previous behaviour. This should only be used when the rule converter is confident that the Sigma rule's string values will have the same case as the data they are querying over.Note: at this point, Loki has an open issue grafana/loki#9294 around its handling of case-insensitive field equality - so this PR should not be merged into main until it is resolved.
Closes #50.