Add Entity related integrations ML rules with _ea job IDs and min_stack_version 9.4.0

[susan-shu-c]

Pull Request

Issue link(s):

Related to:

• This PR updates rules to follow the changes made in Entity Analytics 9.4 updates for UEBA integrations integrations#17626
• Update Entity related Kibana prebuilt ML rules with new _ea ML job ID and update minimum stack versions #5794

Summary - What I changed

Update DED, DGA, LMD, PAD, and ProblemChild integration ML rules to use the new _ea ML job ID suffix and min_stack_version = "9.4.0". Corresponds to integrations#17626

How To Test

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Rule: Tuning - Guidelines

These guidelines serve as a reminder set of considerations when tuning an existing rule.

Documentation and Context

• Detailed description of the suggested changes.
• Provide example JSON data or screenshots.
• Provide evidence of reducing benign events mistakenly identified as threats (False Positives).
• Provide evidence of enhancing detection of true threats that were previously missed (False Negatives).
• Provide evidence of optimizing resource consumption and execution time of detection rules (Performance).
• Provide evidence of specific environment factors influencing customized rule tuning (Contextual Tuning).
• Provide evidence of improvements made by modifying sensitivity by changing alert triggering thresholds (Threshold Adjustments).
• Provide evidence of refining rules to better detect deviations from typical behavior (Behavioral Tuning).
• Provide evidence of improvements of adjusting rules based on time-based patterns (Temporal Tuning).
• Provide reasoning of adjusting priority or severity levels of alerts (Severity Tuning).
• Provide evidence of improving quality integrity of our data used by detection rules (Data Quality).
• Ensure the tuning includes necessary updates to the release documentation and versioning.

Rule Metadata Checks

• updated_date matches the date of tuning PR merged.
• min_stack_version should support the widest stack versions.
• name and description should be descriptive and not include typos.
• query should be inclusive, not overly exclusive. Review to ensure the original intent of the rule is maintained.

Testing and Validation

• Validate that the tuned rule's performance is satisfactory and does not negatively impact the stack.
• Ensure that the tuned rule has a low false positive rate.

[shashank-elastic]

@susan-shu-c

I have take time to review this PR and the upstream integration changes. Below are my observations / concerns

• When prebuilt ML rules set min_stack_version (e.g. 9.4) so they only target 9.4.0 stacks and install the latest version of the rule to match new ML job IDs.
• The upstream integration changes are not min-stacked to 9.4.0, so essentially the new integration changes is available for stacks say 9.2 and 9.3
• On those stacks, rules stay on an older package (old job IDs) while the integration may already expose new jobs.
• Detection rule repo unit tests such as test_ml_integration_jobs_exist only assert the job exists for one integration version chosen from manifests for the rule’s min_stack_version (or current package); they do not prove consistency across every older integration line customers might still run. And these will fail on 9.2 and 9.3 as the job IDs will no longer match as the rule updates are now min-stacked to 9.4 only but the integration version would be upgraded.
• If the customer choose to upgrade the integration in older stack versions, the rules may be inconsistent with job names.

I would recommend if we are not min-stacking the integration, let us also not minstack the rule, this will allow the new job-ids to be back-ported to older stack versions as well and the users if they choose to update the integration will have the right job ID's. Also am not sure why the rules were min-stacked originally but this will cause issues to users on older stack as they can upgrade the integrations.

cc @Mikaayenson @eric-forte-elastic

[shashank-elastic]

@susan-shu-c I have pulled in the latest manifest and schemas of the changed ML integrations and the Unit Tests are now passing on the PR. Please note this PR will be merged once we have the prep for release complete and 9.4 branch is available on Wednesday!  The PR may see some conflicts as we prepare manifests for other packages, I will take care of that and merge once ready!.