[Bug] EQL Parsing Error on String where condition #5236
Description
Describe the Bug
When we have queries that specify a double quotes string as the source of a where, the detection rules EQL parser throws an error. However, this is allowed in the stack.
To Reproduce
"ec2.amazonaws.com" where event.dataset == "aws.cloudtrail"
and event.action == "ModifySnapshotAttribute"
and event.outcome == "success"
and stringContains (aws.cloudtrail.request_parameters, "attributeType=CREATE_VOLUME_PERMISSION")
and stringContains (aws.cloudtrail.request_parameters, "add={items")
Will result in an error whereas:
any where event.dataset == "aws.cloudtrail"
and event.action == "ModifySnapshotAttribute"
and event.outcome == "success"
and stringContains (aws.cloudtrail.request_parameters, "attributeType=CREATE_VOLUME_PERMISSION")
and stringContains (aws.cloudtrail.request_parameters, "add={items")
Will not. See #5229 for examples, or using the following rule and attempt to load it via view rule (or other means)
python -m detection_rules view-rule rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml
Will result in:
detection-rules on tune_aws_ebs_snapshot_rules [!?] is v1.5.0 via v3.12.12 (detection-rules-build) on eric.forte
❯ python -m detection_rules view-rule rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
Error loading rule in /tmp/detection-rules/rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/tmp/detection-rules/detection_rules/__main__.py", line 31, in <module>
main()
File "/tmp/detection-rules/detection_rules/__main__.py", line 28, in main
root(prog_name="detection_rules")
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/main.py", line 462, in view_rule
rule = RuleCollection().load_file(rule_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule_loader.py", line 507, in load_file
return self.load_dict(obj, path=path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule_loader.py", line 483, in load_dict
contents = TOMLRuleContents.from_dict(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/mixins.py", line 144, in from_dict
return schema.load(obj)
^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 973, in load
all_loaded = super().load(data, many=many, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 792, in load
return self._do_load(
^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 974, in _do_load
self._invoke_schema_validators(
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 1267, in _invoke_schema_validators
self._run_validator(
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 845, in _run_validator
validator_func(output, partial=partial, many=many)
File "/tmp/detection-rules/detection_rules/rule.py", line 1550, in post_conversion_validation
data.validate_query(metadata)
File "/tmp/detection-rules/detection_rules/rule.py", line 760, in validate_query
validator.validate(self, meta)
File "/tmp/detection-rules/detection_rules/rule_validators.py", line 599, in validate
all_targets = self.build_validation_plan(data, meta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule_validators.py", line 401, in build_validation_plan
is_sequence = getattr(data, "is_sequence", False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/functools.py", line 998, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule.py", line 923, in is_sequence
return eql.utils.get_query_type(self.ast) == "sequence" # type: ignore[reportUnknownMemberType]
^^^^^^^^
File "/usr/lib/python3.12/functools.py", line 998, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule.py", line 766, in ast
return validator.ast
^^^^^^^^^^^^^
File "/usr/lib/python3.12/functools.py", line 998, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule_validators.py", line 362, in ast
return eql.parse_query(self.query) # type: ignore[reportUnknownVariableType]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/eql/parser.py", line 1415, in parse_query
return _parse(text, "piped_query", implied_any=implied_any, implied_base=implied_base, preprocessor=preprocessor,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/detection_rules/rule_validators.py", line 139, in wrapper
result = func(query, start=start, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/eql/parser.py", line 1367, in _parse
raise exc
eql.errors.EqlSyntaxError: Error at line:1,column:19
Invalid syntax
^^^^^Expected Behavior
Rule is expected to load successfully.
Screenshots
No response
Desktop - OS
None
Desktop - Version
No response
Additional Context
Verbose error messages via debugger:
Details
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 126, in feed_token
action, arg = states[state][token.type]
~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'NAME'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/eql/parser.py", line 1340, in _parse
tree = lark_parser.parse(text, start=start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/lark.py", line 581, in parse
return self.parser.parse(text, start=start, on_error=on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parser_frontends.py", line 106, in parse
return self.parser.parse(stream, chosen_start, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 41, in parse
return self.parser.parse(lexer, start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 171, in parse
return self.parse_from_state(parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 188, in parse_from_state
raise e
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 179, in parse_from_state
state.feed_token(token)
File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 129, in feed_token
raise UnexpectedToken(token, expected, state=self, interactive_parser=None)
lark.exceptions.UnexpectedToken: Unexpected token Token('NAME', 'where') at line 1, column 21.
Expected one of:
* VBAR
* $END