Optimize TI analytic rule deduplication and filters

[v-sabiraj]

Reorder and simplify the Threat Intelligence analytic query to deduplicate records earlier and apply active/expiration filters afterwards. Added summarize arg_max(TimeGenerated, *) by Id, Url and by Id, ObservableValue to get the latest record before filtering, moved IsActive/ValidUntil/ExpirationDateTime checks to follow deduplication, and removed redundant time-order checks and extra arg_max usages related to EmailUrlInfo. These changes ensure the latest valid indicators are retained and simplify the join logic. Bumped version 1.0.5 -> 1.0.6.

Required items, please complete

Change(s):

• See guidance below

Reason for Change(s):

• See guidance below

Version Updated:

• Required only for Detections/Analytic Rule templates
• See guidance below

Testing Completed:

• See guidance below

Checked that the validations are passing and have addressed any issues that are present:

• See guidance below

Guidance <- remove section before submitting

---

Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:

Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR. Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process. Providing associated Issues that this resolves also easily connects the reason.

Change(s):

• Updated syntax for XYZ.yaml

Reason for Change(s):

• New schema used for XYZ.yaml
• Resolves ISSUE Add Logstash required version to Readme file #1234

Version updated:

• Yes
• Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly. If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

• Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

• Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

• Guidance for Detection checks
• General contribution guidance
• PR validation troubleshooting

---

[contentautomationbot]

Hello how are you I am GitHub bot
😀😀
I see that you changed templates under the detections/analytic rules folder. Did you remember to update the version of the templates you changed?
If not, and if you want customers to be aware that a new version of this template is available, please update the version property of the template you changed.

[Copilot]

Pull request overview

This PR refactors the Threat Intelligence analytic rule query to deduplicate TI indicator records earlier (via arg_max) and streamline filtering/join logic, with a corresponding template version bump.

Changes:

• Deduplicate URL and domain TI indicators earlier using summarize arg_max(TimeGenerated, *) ....
• Reorder/adjust active/expiration filtering and simplify join-related logic with EmailUrlInfo_.
• Bump analytic rule template version from 1.0.5 to 1.0.6.

[Copilot]

Reordering to deduplicate before applying IsActive/ValidUntil can drop valid indicators: if the most recent record for a given Id, Url is inactive/expired but an earlier record is still active/valid, the arg_max keeps the inactive one and the subsequent where removes it entirely. To preserve the latest valid indicator, apply the IsActive/ValidUntil filter before arg_max, or compute arg_max over only the valid subset (e.g., filter first, then summarize).

[Copilot]

After switching to join kind=innerunique without re-selecting the latest EmailUrlInfo_TimeGenerated per indicator/URL, the chosen EmailUrlInfo_ row becomes effectively arbitrary when multiple email-URL records exist for the same Url. This can yield nondeterministic results and may associate older/newer email evidence incorrectly. Consider deduplicating EmailUrlInfo_ (e.g., arg_max(EmailUrlInfo_TimeGenerated, *) by Url) before the join, or restoring a post-join summarize arg_max(EmailUrlInfo_TimeGenerated, *) ... to make the selection deterministic.

[Copilot]

In the domain branch, tolower(ObservableValue) is applied after deduplication/grouping by ObservableValue. If the same domain appears with different casing, it will be treated as distinct groups during summarize, potentially producing duplicates downstream even though DomainName is later normalized. Normalize first (create a lowercased domain column) and then summarize ... by Id, <lowercasedDomain> (or summarize by the normalized key you actually use later).

Suggested change

	| summarize arg_max(TimeGenerated, *) by Id, ObservableValue // Get latest record per Id FIRST
	| where IsActive == true // THEN filter for active indicators
	| extend TrafficLightProtocolLevel = AdditionalFields.TLPLevel
	| extend DomainName = tolower(ObservableValue) // Convert domain names to lowercase
	| extend DomainName = tolower(ObservableValue) // Normalize domain names to lowercase before summarization
	| summarize arg_max(TimeGenerated, *) by Id, DomainName // Get latest record per Id and normalized domain
	| where IsActive == true // THEN filter for active indicators
	| extend TrafficLightProtocolLevel = AdditionalFields.TLPLevel

[rahul0216]

Please address discussed pointers.

[rahul0216]

Looks good.