[Bug] EQL `Sets` optimization creates parsing bug

[brokensound77]

related to #2593 (comment)

Overview

The python EQL optimization for combining Sets of values assumes that the field is single-valued, which leads to an improper optimization of the logic (EQL on elasticsearch treats sets as multi-value).

This does not impact the query that makes it to the final rule build, but has implications on validation and testing. The query is parsed in multiple places, but can be found under rule.contents.data.ast.

Details

The source of the optimization can be found in the eql ast code.

Ex:

process where host.os.type == "macos" and event.type == "start" and
 process.name == "defaults" and process.args == "write" and process.args in ("LoginHook", "LogoutHook") and
 not process.args :
       (
         "Support/JAMF/ManagementFrameworkScripts/logouthook.sh",
         "Support/JAMF/ManagementFrameworkScripts/loginhook.sh",
         "/Library/Application Support/JAMF/ManagementFrameworkScripts/logouthook.sh",
         "/Library/Application Support/JAMF/ManagementFrameworkScripts/loginhook.sh"
       )

Will get optimized to:

process where false

Since process.args cannot be equal to both "write" and ("LoginHook", "LogoutHook")

(Note: there is no optimization for wildcard sets, which is why this doesn't impact most of the rules)

To reproduce, run:

import eql

query = """process where host.os.type == "macos" and event.type == "start" and
 process.name == "defaults" and process.args == "write" and process.args in ("LoginHook", "LogoutHook") and
 not process.args :
       (
         "Support/JAMF/ManagementFrameworkScripts/logouthook.sh",
         "Support/JAMF/ManagementFrameworkScripts/loginhook.sh",
         "/Library/Application Support/JAMF/ManagementFrameworkScripts/logouthook.sh",
         "/Library/Application Support/JAMF/ManagementFrameworkScripts/loginhook.sh"
       )"""

with eql.parser.elasticsearch_syntax, eql.parser.ignore_missing_functions:
    print(eql.parse_query(s))

process where false

details from original PR

error:

FAILED tests/test_all_rules.py::TestEndpointQuery::test_os_and_platform_in_query - AssertionError: 'host.os.name' not found in [] : 5d0265bf-dea9-41a9-92ad-48a8dcd05080 - Persistence via Login or Logout Hook -> missing required field for endpoint rule

No fields were parsed: []

rule.name
'Persistence via Login or Logout Hook'

rule.id
'5d0265bf-dea9-41a9-92ad-48a8dcd05080'

rule.contents.data.query
'process where host.os.name == "macos" and event.type == "start" and\n process.name == "defaults" and process.args == "write" and process.args in ("LoginHook", "LogoutHook") and\n not process.args :\n       (\n         "Support/JAMF/ManagementFrameworkScripts/logouthook.sh",\n         "Support/JAMF/ManagementFrameworkScripts/loginhook.sh",\n         "/Library/Application Support/JAMF/ManagementFrameworkScripts/logouthook.sh",\n         "/Library/Application Support/JAMF/ManagementFrameworkScripts/loginhook.sh"\n       )\n'

rule.contents.data.ast
PipedQuery(first=EventQuery(event_type='process', query=Boolean(value=False)), pipes=[])

str(rule.contents.data.ast)
'process where false'

The original logic is parsed simply into process where false

Originally posted by @brokensound77 in #2593 (comment)

[traut]

Since we have a workaround for this EQL lib bug (optimisation was turned off in #3443), and fixing it properly in the EQL lib would require significant archeological efforts, I'm closing this issue.