[FR] Add ESQL rules to dataset exception

[eric-forte-elastic]

Pull Request

Issue link(s):

Resolves #5246

Summary - What I changed

The prior logic is to not have related_integrations populated for rules that do not have an event.dataset even if it is specified in the integration. (not specific to ESQL).

For example an EQL query with the following:

[metadata]
creation_date = "2024/04/17"
integration = ["aws"]

...

query = '''
any where event.provider == "s3.amazonaws.com"
    and event.action == "PutBucketPolicy" and event.outcome == "success"
'''

Will also not have related integrations populated (as we enforce the need for event.dataset for all integrations that are not part of NON_DATASET_PACKAGES). This is due to us not being able to use an AST for ESQL validation. As such we need to check for integrations directly and deduplicate with event datasets (as done in this draft PR).

However, we also need to handle cases like this in deduplication:

"related_integrations": [
    {
      "package": "aws_bedrock",
      "version": "^1.1.0"
    },
    {
      "package": "aws",
      "version": "^4.0.0"
    },
    {
      "integration": "billing",
      "package": "aws",
      "version": "^4.0.0"
    },
    {
      "integration": "cloudtrail",
      "package": "aws",
      "version": "^4.0.0"
    }
  ],

Fortunately, Kibana handles this when a rule is imported.

Example Modified pre-built rule with the above related integrations.

test_esql_rule.ndjson.txt

Also added a unit test/integration test to check for this specific behavior. Before we were only validating the presence of related integrations for a query that had an event dataset.

Note the modification to remote filtered_keys.update(indices) from detection_rules/index_mappings.py is an unrelated fix to a typo where this should not have been added. This was inadvertently introduced when testing autogenerated schema compatibility with ESQL validation in this case is an NOP and can be removed. All unit tests pass with this removal and test rules all pass validation.

How To Test

Run remote unit tests

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[eric-forte-elastic]

Unit test pass, with new additional unit test.

detection-rules on  5246-bug-inconsistent-population-of-related_integrations-field-for-esql-rules [$?] is  v1.5.5 via  v3.12.12 (detection-rules-build) on  eric.forte took 4s 
❯ python -m pytest tests/test_rules_remote.py::TestRemoteRules
============================================================================================ test session starts ============================================================================================
platform linux -- Python 3.12.12, pytest-8.4.0, pluggy-1.6.0
rootdir: /home/forteea1/Code/dac_demo/dac_clean/detection-rules
configfile: pyproject.toml
plugins: typeguard-4.4.4
collected 11 items                                                                                                                                                                                          

tests/test_rules_remote.py ...........                                                                                                                                                                [100%]

============================================================================================= warnings summary ==============================================================================================
tests/test_rules_remote.py: 11 warnings
  /home/forteea1/Code/dac_demo/dac_clean/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py:399: SecurityWarning: Connecting to 'https://localhost:9200' using TLS with verify_certs=False is insecure
    _transport = transport_class(

tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index_endpoint_fields
tests/test_rules_remote.py::TestRemoteRules::test_esql_filtered_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_non_dataset_package_related_integrations
tests/test_rules_remote.py::TestRemoteRules::test_esql_related_integrations
  /home/forteea1/Code/dac_demo/dac_clean/detection-rules/detection_rules/index_mappings.py:344: ElasticsearchWarning: No limit defined, adding default limit of [1000]
    response = elastic_client.esql.query(query=query)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================================================== 11 passed, 16 warnings in 9.57s ======================================================================================