Add unit test for protected prebuilt-rules

[shashank-elastic]

Pull Request

Issue link(s): #4707

Summary - What I changed

• Add a test case to ensure upstream referenced (protected rules) rule IDs and rule names remain unchanged
• The only rule_id referenced so far appears to be 9a1a2dae-0b5f-4c3d-8305-a268d404c306.

How To Test

Protected Rule not modified

❯ pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                 

tests/test_all_rules.py .                                                                                                                  [100%]

=============================================================== 1 passed in 59.46s ===============================================================

detection-rules on  4707-fr-unit-test-for-elastic_endpoint_security-rule-id-change [$?] is 📦 v1.5.2 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m1s 
❯ 

Protected Rule ID modified

❯ pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                 

tests/test_all_rules.py F                                                                                                                  [100%]

==================================================================== FAILURES ====================================================================
_______________________________________ TestIntegrationRules.test_preserve_upstream_protected_rule_id_name _______________________________________

self = <tests.test_all_rules.TestIntegrationRules testMethod=test_preserve_upstream_protected_rule_id_name>

    def test_preserve_upstream_protected_rule_id_name(self):
        """
        Ensure upstream referenced rule IDs and rule names remain unchanged
        """
        protected_rules = {"9a1a2dae-0b5f-4c3d-8305-a268d404c306": "Endpoint Security (Elastic Defend)"}
    
        # map current rules by id and name for quick lookup
        current_rules = {rule.contents.data.get("rule_id"): rule.contents.data.get("name") for rule in self.all_rules}
        failures = []
        for rule_id, rule_name in protected_rules.items():
            if rule_id in current_rules:
                if rule_name != current_rules.get(rule_id):
                    failures.append(
                        f"Protected rule_id {rule_id} name modified from '{rule_name}' to '{current_rules.get(rule_id)}' - review upstream impact"
                    )
            else:
                failures.append(
                    f"Protected rule: {rule_name} rule_id: {rule_id} missing/modified - review upstream impact"
                )
    
        if failures:
            fail_msg = """
            The following protected prebuilt rules have missing/modified rule IDs or names \n
            """
>           self.fail(fail_msg + "\n".join(failures))
E           AssertionError: 
E                       The following protected prebuilt rules have missing/modified rule IDs or names 
E           
E                       Protected rule: Endpoint Security (Elastic Defend) rule_id: 9a1a2dae-0b5f-4c3d-8305-a268d404c306 missing/modified - review upstream impact

tests/test_all_rules.py:1139: AssertionError
============================================================ short test summary info =============================================================
FAILED tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name - AssertionError: 
=============================================================== 1 failed in 58.47s ===============================================================

detection-rules on  4707-fr-unit-test-for-elastic_endpoint_security-rule-id-change [$!?] is 📦 v1.5.2 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m 
❯ 

Protected Rule name modified

❯ pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                 

tests/test_all_rules.py F                                                                                                                  [100%]

==================================================================== FAILURES ====================================================================
_______________________________________ TestIntegrationRules.test_preserve_upstream_protected_rule_id_name _______________________________________

self = <tests.test_all_rules.TestIntegrationRules testMethod=test_preserve_upstream_protected_rule_id_name>

    def test_preserve_upstream_protected_rule_id_name(self):
        """
        Ensure upstream referenced rule IDs and rule names remain unchanged
        """
        protected_rules = {"9a1a2dae-0b5f-4c3d-8305-a268d404c306": "Endpoint Security (Elastic Defend)"}
    
        # map current rules by id and name for quick lookup
        current_rules = {rule.contents.data.get("rule_id"): rule.contents.data.get("name") for rule in self.all_rules}
        failures = []
        for rule_id, rule_name in protected_rules.items():
            if rule_id in current_rules:
                if rule_name != current_rules.get(rule_id):
                    failures.append(
                        f"Protected rule_id {rule_id} name modified from '{rule_name}' to '{current_rules.get(rule_id)}' - review upstream impact"
                    )
            else:
                failures.append(
                    f"Protected rule: {rule_name} rule_id: {rule_id} missing/modified - review upstream impact"
                )
    
        if failures:
            fail_msg = """
            The following protected prebuilt rules have missing/modified rule IDs or names \n
            """
>           self.fail(fail_msg + "\n".join(failures))
E           AssertionError: 
E                       The following protected prebuilt rules have missing/modified rule IDs or names 
E           
E                       Protected rule_id 9a1a2dae-0b5f-4c3d-8305-a268d404c306 name modified from 'Endpoint Security (Elastic Defend)' to 'Endpoint Security (Elastic Defends)' - review upstream impact

tests/test_all_rules.py:1139: AssertionError
============================================================ short test summary info =============================================================
FAILED tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name - AssertionError: 
=============================================================== 1 failed in 59.37s ===============================================================

detection-rules on  4707-fr-unit-test-for-elastic_endpoint_security-rule-id-change [$!?] is 📦 v1.5.2 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m1s 
❯ 

Retested After Code Changes All 3 use cases

❯ pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                                                                                                             

tests/test_all_rules.py F                                                                                                                                                                                                              [100%]

================================================================================================================== FAILURES ==================================================================================================================
_____________________________________________________________________________________ TestIntegrationRules.test_preserve_upstream_protected_rule_id_name _____________________________________________________________________________________

self = <tests.test_all_rules.TestIntegrationRules testMethod=test_preserve_upstream_protected_rule_id_name>

    def test_preserve_upstream_protected_rule_id_name(self):
        """
        Ensure upstream referenced rule IDs and rule names remain unchanged
        """
        protected_rules = {"9a1a2dae-0b5f-4c3d-8305-a268d404c306": "Endpoint Security (Elastic Defend)"}
    
        failures: list[str] = []
        for rule_id, rule_name in protected_rules.items():
            try:
                if rule_name != self.rc.id_map[rule_id].name:
                    failures.append(
                        f"Protected rule_id {rule_id} name modified from '{rule_name}' to '{self.rc.id_map[rule_id].name}' - review upstream impact"
                    )
            except KeyError:
                failures.append(
                    f"Protected rule: {rule_name} rule_id: {rule_id} missing/modified - review upstream impact"
                )
    
        if failures:
            fail_msg = """
            The following protected prebuilt rules have missing/modified rule IDs or names \n
            """
>           self.fail(fail_msg + "\n".join(failures))
E           AssertionError: 
E                       The following protected prebuilt rules have missing/modified rule IDs or names 
E           
E                       Protected rule: Endpoint Security (Elastic Defend) rule_id: 9a1a2dae-0b5f-4c3d-8305-a268d404c306 missing/modified - review upstream impact

tests/test_all_rules.py:1137: AssertionError
========================================================================================================== short test summary info ===========================================================================================================
FAILED tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name - AssertionError: 
============================================================================================================= 1 failed in 59.32s =============================================================================================================

detection-rules on  4707-fr-unit-test-for-elastic_endpoint_security-rule-id-change [$!?] is 📦 v1.5.4 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m1s 
❯ pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                                                                                                             

tests/test_all_rules.py F                                                                                                                                                                                                              [100%]

================================================================================================================== FAILURES ==================================================================================================================
_____________________________________________________________________________________ TestIntegrationRules.test_preserve_upstream_protected_rule_id_name _____________________________________________________________________________________

self = <tests.test_all_rules.TestIntegrationRules testMethod=test_preserve_upstream_protected_rule_id_name>

    def test_preserve_upstream_protected_rule_id_name(self):
        """
        Ensure upstream referenced rule IDs and rule names remain unchanged
        """
        protected_rules = {"9a1a2dae-0b5f-4c3d-8305-a268d404c306": "Endpoint Security (Elastic Defend)"}
    
        failures: list[str] = []
        for rule_id, rule_name in protected_rules.items():
            try:
                if rule_name != self.rc.id_map[rule_id].name:
                    failures.append(
                        f"Protected rule_id {rule_id} name modified from '{rule_name}' to '{self.rc.id_map[rule_id].name}' - review upstream impact"
                    )
            except KeyError:
                failures.append(
                    f"Protected rule: {rule_name} rule_id: {rule_id} missing/modified - review upstream impact"
                )
    
        if failures:
            fail_msg = """
            The following protected prebuilt rules have missing/modified rule IDs or names \n
            """
>           self.fail(fail_msg + "\n".join(failures))
E           AssertionError: 
E                       The following protected prebuilt rules have missing/modified rule IDs or names 
E           
E                       Protected rule_id 9a1a2dae-0b5f-4c3d-8305-a268d404c306 name modified from 'Endpoint Security (Elastic Defend)' to 'Endpoint Security (Elastic Defends)' - review upstream impact

tests/test_all_rules.py:1137: AssertionError
========================================================================================================== short test summary info ===========================================================================================================
FAILED tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name - AssertionError: 
======================================================================================================== 1 failed in 61.13s (0:01:01) ========================================================================================================

detection-rules on  4707-fr-unit-test-for-elastic_endpoint_security-rule-id-change [$!?] is 📦 v1.5.4 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m3s 
❯ pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                                                                                                             

tests/test_all_rules.py .                                                                                                                                                                                                              [100%]

======================================================================================================== 1 passed in 62.06s (0:01:02) ========================================================================================================

## Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[eric-forte-elastic]

Just a note for posterity, do we expect there to be substantially more protected rules, or generally is this a small list?

[shashank-elastic]

Technically for now 1 and may be a growing small list or not.

Initially I had checked to see if we wanna make all the rule_id immutable. But since there are no upstream impacts choose to stick to only rule that is referenced by its ID here https://github.com/elastic/kibana/blob/3f7184698faea27158fb47f397c35fec909a4ce3/x-pack/solutions/security/plugins/security_solution/common/detection_engine/constants.ts#L38

[shashank-elastic]

In Peer Review discussions we identified the root issue of ESQL validation not kicking off. @eric-forte-elastic Will shortly raise a fix and I will rebase and merge

[eric-forte-elastic]

I really like the idea of a quick lookup that you are using. Looking at an alternative.

        protected_rules = {"9a1a2dae-0b5f-4c3d-8305-a268d404c306": "Endpoint Security (Elastic Defend)"}

        # map current rules by id and name for quick lookup
        filtered_rules_collection = self.all_rules.filter(lambda r: r.id in protected_rules.keys())
        failures: list[str] = []
        for rule in filtered_rules_collection:
            rule_id = rule.contents.data.get("rule_id")
            rule_name = rule.contents.data.get("name")
            if rule_id in protected_rules.keys():
                if rule_name != protected_rules.get(rule_id):
                    failures.append(
                        f"Protected rule_id {rule_id} name modified from '{rule_name}' to '{protected_rules.get(rule_id)}' - review upstream impact"
                    )
            else:
                failures.append(
                    f"Protected rule: {rule_name} rule_id: {rule_id} missing/modified - review upstream impact"
                )

Your current lookup effectively costs O( 2 n ) due to:

• This is O(n): current_rules = {rule.contents.data.get("rule_id"): rule.contents.data.get("name") for rule in self.all_rules}
• This is O(n) for rule_id, rule_name in protected_rules.items():
  where we know n > 1,000 since we have ~1,500 rules

Whereas the above suggestion:

• This is technically O(n) although it may be more optimized filtered_rules_collection = self.all_rules.filter(lambda r: r.id in protected_rules.keys())
• This is O(m) for rule in filtered_rules_collection:
  where we know m < 10 (in this case 1).

Test for the above code time:

❯ python -m pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/detection-rules
configfile: pyproject.toml
plugins: typeguard-4.4.4
collected 1 item                                                                                                                                                                                       

tests/test_all_rules.py .                                                                                                                                                       [100%]

============================================================================ 1 passed in 121.18s (0:02:01) ============================================================================
⏎          

Test for current code time:

❯ python -m pytest tests/test_all_rules.py::TestIntegrationRules::test_preserve_upstream_protected_rule_id_name
================================================================================= test session starts =================================================================================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/detection-rules
configfile: pyproject.toml
plugins: typeguard-4.4.4
collected 1 item                                                                                                                                                                      

tests/test_all_rules.py .                                                                                                                                                       [100%]

============================================================================ 1 passed in 104.75s (0:01:44) ============================================================================

As we can see, the O( 2 n) algorithmic approach is notably faster than the theoretically more optimized approach 👍

LGTM 👍