Escaping of chars in rule supplied regular expressions

[stbe]

Hello! I've got an issue with a couple of rules and wanted to get an opinion on whether these are bugs in the rules, needs improvement in the backend, or in general what your advice may be how to deal with it.

My problem is that some rules provide pre-supplied regular expressions in field|re conditions that - when generated with the Elastic DSL backend - throw errors on the Elastic side. The source of the issue seems to be the escaping of characters that may be valid characters to be used in a regular expression, but are reserved special characters in a DSL and thus would need escaping.

I have found a few rules in which this poses a problem. For example:

sigma/rules/windows/builtin/security/win_invoke_obfuscation_via_stdin_services_security.yml

Line 25 in a502f31

ServiceFileName|re: '(?i).*(set).*&&\s?set.*(environment|invoke|\${?input).*&&.*"'

This rule's regex has two problems: The { and the " towards the end of the string. Both trigger an error when parsed in an elastic query.

The question now is: Is this a bug in the rule, and the escaping should be done in the rule? Or, should this be done in the respective backend? I think one would favor the second option since each backend can have different reserved, DSL specific characters. So regex escaping should be in the rule, and the backend should apply the escaping of the backend specific chars.

I'm just openly asking what is preferred because I've found multiple rules that do indeed escape such characters, for example, the following one contains the escaped \" at the end of the string.

sigma/rules/windows/builtin/security/win_invoke_obfuscation_var_services_security.yml

Line 25 in a502f31

ServiceFileName|re: '.*cmd.{0,5}(?:\/c|\/r)(?:\s|)\"set\s[a-zA-Z]{3,6}.*(?:\{\d\}){1,}\\\"\s+?\-f(?:.*\)){1,}.*\"'

(This rule by the way has another issue that triggers an ES error: the | in position 28 of the regex should either be escaped to serve as literal, or have an option followed after it)

Here is a list of rules that I found are affected by the escaping issue:
Note: I've only checked the windows rules

• builtin/security/win_invoke_obfuscation_var_services_security.yml
• builtin/security/win_invoke_obfuscation_via_stdin_services_security.yml
• builtin/security/win_invoke_obfuscation_via_use_mshta_services_security.yml
• builtin/security/win_invoke_obfuscation_via_use_rundll32_services_security.yml
• builtin/system/win_invoke_obfuscation_var_services.yml
• builtin/system/win_invoke_obfuscation_via_stdin_services.yml
• builtin/system/win_invoke_obfuscation_via_use_mshta_services.yml
• builtin/system/win_invoke_obfuscation_via_use_rundll32_services.yml
• powershell/powershell_module/powershell_invoke_obfuscation_var.yml
• powershell/powershell_module/powershell_invoke_obfuscation_via_stdin.yml
• powershell/powershell_module/powershell_invoke_obfuscation_via_use_mhsta.yml
• powershell/powershell_module/powershell_invoke_obfuscation_via_use_rundll32.yml
• powershell/powershell_script/powershell_invoke_obfuscation_var_in_scriptblocktext.yml
• powershell/powershell_script/powershell_invoke_obfuscation_via_stdin_in_scriptblocktext.yml
• powershell/powershell_script/powershell_invoke_obfuscation_via_use_mhsta_in_scriptblocktext.yml
• powershell/powershell_script/powershell_invoke_obfuscation_via_use_rundll32_in_scriptblocktext.yml
• process_creation/win_class_exec_xwizard.yml
• process_creation/win_invoke_obfuscation_var.yml
• process_creation/win_invoke_obfuscation_via_stdin.yml
• process_creation/win_invoke_obfuscation_via_use_mhsta.yml
• process_creation/win_invoke_obfuscation_via_use_rundll32.yml
• process_creation/win_powershell_cmdline_special_characters.yml
• process_creation/win_run_powershell_script_from_input_stream.yml

[Knorke22222]

Hello,
yes DSL has a few problems. Try it with Lucene Syntax If possible.

[stbe]

Unfortunately, this would only shift the problem towards the querystring backend. The problematic rules from my post above would indeed work, but those rules that have an escape character in place, e.g. a \" would likely not work as expected. They would not need the escape char in the query string (as they are enclosed in /) probably resulting in undesired query behavior - i.e. the query being valid but not matching due to the regex trying to also match the backslash.

(additional downside is that one would also lose the aggregation feature)

[Neo23x0]

(This rule by the way has another issue that triggers an ES error: the | in position 28 of the regex should either be escaped to serve as literal, or have an option followed after it)

I'd say the |) means "or nothing" and works as intended by the author.

I'm no fan of these regex rules neither.

[nasbench]

Hi and thanks for opening this issue. I got some updates regarding this.

The following rules don't use regular expressions anymore. So their good to go

• win_security_invoke_obfuscation_var_services_security.yml
• win_security_invoke_obfuscation_via_stdin_services_security.yml
• win_security_invoke_obfuscation_via_use_mshta_services_security.yml
• win_security_invoke_obfuscation_via_use_rundll32_services_security.yml
• win_system_invoke_obfuscation_var_services.yml
• win_system_invoke_obfuscation_via_stdin_services.yml
• win_system_invoke_obfuscation_via_use_mshta_services.yml
• win_system_invoke_obfuscation_via_use_rundll32_services.yml
• posh_ps_invoke_obfuscation_via_use_mhsta.yml
• posh_ps_invoke_obfuscation_via_use_rundll32.yml
• proc_creation_win_invoke_obfuscation_var.yml
• proc_creation_win_invoke_obfuscation_via_stdin.yml
• proc_creation_win_invoke_obfuscation_via_use_mhsta.yml
• proc_creation_win_invoke_obfuscation_via_use_rundll32.yml

The following rules are better not transformed since the fixed strings used in the regex are too common and due to the nature of the PowerShell logs we can't ensure their location with a contains so they are kept as regex.

• posh_pm_invoke_obfuscation_via_var.yml
• posh_pm_invoke_obfuscation_via_stdin.yml
• posh_ps_invoke_obfuscation_var.yml
• posh_ps_invoke_obfuscation_via_stdin.yml

The following rules were left as regex as the logic is best described that way

• proc_creation_win_lolbin_class_exec_xwizard.yml
• proc_creation_win_powershell_cmdline_special_characters.yml
• proc_creation_win_run_powershell_script_from_input_stream.yml

Regarding the escaping of some characters in the regex. PR #3744 has addressed some issues and made improvements to the rules that still use regular expressions.

Hope this help.

I will close this issue as completed for now. If you have any other issues with any of the rules mentioned above or new ones. Please don't hesitate to open a new one and reference this is one if they are related.