IOC Deduplication

IOC Deduplication

Available from IRIS-NG-v2.2.0. See Changelog.

A case collects indicators from several directions — typed by hand, pasted from a report, imported from CSV, escalated from alerts, extracted from notes, promoted off the working timeline — and the same indicator arrives more than once, rarely spelled the same way twice: hxxp://evil[.]com/login.php beside http://evil.com/login.php, EVIL.COM beside evil.com, 203.0.113.9:8443 beside 203.0.113.9. Find duplicates, in the ⋮ menu of the case's IOC page, finds them and folds them together without losing anything that hung off the removed row.

What counts as a duplicate

Two rows are exact duplicates when they have the same IOC type and the same value after normalisation. Normalisation is deliberately generous, because that is how indicators are written in the wild:

Written	Normalised
hXXp://Evil[.]com/A	http://evil.com/a
EVIL(dot)COM	evil.com
user[at]corp.example	user@corp.example
8[.]8[.]8[.]8	8.8.8.8
hxxps://x[.]y[:]8443/p	https://x.y:8443/p
evil\.com	evil.com
ABCDEF0123 (a hash)	abcdef0123

Trim, Unicode normalisation, the common defanging conventions (hxxp/fxp, [.], (.), {.}, [dot], [at], [:], [://], regex-escaped dots), whitespace collapsed, then case-folded. Case-folding is intentional: hashes, domains, e-mails and URLs are case-insensitive in practice, and two rows that differ only by case are one indicator.

The first entry survives. In an exact group the row with the lowest id — the one added first — is kept; the later rows are folded into it.

The same rule is enforced at ingestion so new duplicates stop arriving: a CSV import skips a row the case already holds and names the existing indicator in its result (Skipped 1 duplicate(s): EVIL[.]COM already exists as IOC #42 (evil.com)), and escalating an alert into a case links the case's existing indicator instead of minting a second row. Adding one by hand was already refused on an exact string match; it still is.

The modal

Find duplicates scans the case and shows three sections.

Exact duplicates. Each group lists the survivor and the rows that will be folded into it, with what each row carries (assets, events, notes). Auto-resolve all collapses every group in one click.

Near duplicates. Candidates a string comparison can find but should not decide alone, each with the reason it was paired:

Reason	Example
URL host is another indicator	http://evil.com/login.php and evil.com
host:port beside host	203.0.113.9:8443 and 203.0.113.9
Same value under two types	evil.com as a domain and as a hostname
Close spelling within a type	cdn-updates-evil.com and cdn-update-evil.com (similarity ≥ 85 %)

AI pass. Run AI pass asks the configured model to read the whole list and propose the pairs a string comparison misses — a file by name and by path, a hash recorded under two types, HKLM beside HKEY_LOCAL_MACHINE — each with a confidence and a one-line reason. It is advisory and stateless; details on AI Features. Without an AI backend the button reports that and the other two sections work as before.

Every pair, in any section, offers the same four choices: Keep left, Merge into left, Merge into right, Keep right, and Keep both to dismiss the pair for this scan.

Keep versus merge — nothing is orphaned

Removing an indicator that has links would silently drop those links from the timeline, the assets and the notes that referenced it. So a keep and a merge both transfer the removed row's links onto the survivor before the row goes:

Carried over	Rule
Asset links	one link per asset — an asset both rows pointed at stays linked once
Timeline events	one link per event
Note provenance (which note the IOC came from)	one link per note
Alert association	the alert now points at the survivor
Comments	follow the indicator
MISP attribute link	moves only if the survivor has none; the MISP side is never touched

The difference is the fields:

• Keep discards the removed row's description, tags and other fields.
• Merge unions the tags, appends a differing description under a Merged from IOC #n line, and fills an empty TLP, MISP reference or enrichment from the removed row.

Auto-resolve all merges (fields unioned), on the reasoning that anything typed on the duplicate was typed for a reason. The survivor's History panel records every fold, and the case activity log shows it.

Scope and limits

• One case at a time. The same indicator appearing in two cases is not a duplicate — it is a correlation, and IOC Correlation is where that lives.
• Read access can scan; full access is needed to resolve. A user who cannot see the case gets nothing, not even a count.
• Keep both is remembered only for the open modal. The pair is listed again on the next scan.
• The near-candidate scan is bounded (the first 2000 indicators by id, at most 200 pairs) and the AI pass sends the first 400 rows; the modal says when a list was truncated.

API

All under /api/v2/cases/<cid>/iocs/dedup/, POST, session or API-key auth:

Path	Body	Returns
scan	—	{ioc_count, exact: [{keep, duplicates: [...]}], near: [{ioc_a, ioc_b, similarity, reason}]}
auto-exact	—	{removed, groups, transferred: {assets, events, notes, alerts, comments, misp, alert_similarity}}
resolve	{"action": "keep"|"merge", "keep_id": n, "delete_ids": [n, …]}	{kept, merged, transferred}
ai-scan	—	202 {task_id}; poll GET /api/v2/ai/jobs/<task_id> — result.pairs. ?sync=true runs inline

resolve refuses an id from another case, a self pair and an empty delete_ids with a 400 and touches nothing.