IOC Correlation

IOC Correlation

Overview

The Correlation tab on /dashboard surfaces IOCs that appear in multiple cases, groups them into clusters, and lets analysts apply campaign tags and generate AI narratives.

No new database tables — correlation is computed on-the-fly from the existing Ioc table using (ioc_value, ioc_type_id) equality.

How it works

1. Query Ioc for values present in two or more cases — that is what "shared" means, and it is fixed, not the threshold below
2. For every pair of cases, count how many distinct IOC values they have in common. A pair is linked when that count reaches the Min shared IOCs per case pair threshold
3. Build clusters from the surviving links via union-find; cluster_id = MD5[:8] of sorted case IDs
4. Compute per-cluster decay and confidence scores
5. Render cluster cards, the Shared IOCs table and the D3 force-directed graph

TLP filter: only TLP:GREEN (id=3) and TLP:CLEAR (id=4) IOCs participate in correlation surfaces. TLP:RED / AMBER / AMBER+STRICT IOCs are excluded to prevent cross-case information leakage on the shared dashboard.

Controls

• Quick-range buttons — 30d / 60d / 90d / 180d / all time
• Custom date range — From / To inclusive pickers
• Min shared IOCs per case pair — default 2; raise to focus on stronger overlaps

What the threshold counts. It is the number of distinct indicators two cases must have in common before they are linked. It is not a minimum number of cases an indicator must appear in. Every IOC in the Shared IOCs table is in at least two cases by definition, so a value present in exactly two cases is normal — it is the smallest thing a shared indicator can be.

A consequence worth internalising: an indicator's fate is decided by its neighbours, not by itself. Two indicators can each appear in exactly two cases, and raising the threshold drops one but keeps the other — because the pair of cases holding the first has little else in common, while the pair holding the second shares several more.

The threshold governs clustering. The Shared IOCs table follows it, listing the indicators behind the surviving clusters, so the table and the cluster cards can never disagree.

Cluster cards

Each cluster card shows:

• Case list with links
• Shared IOC chips
• Decay pill (● Active · 65%, color-coded green/amber/grey) — see below
• IOC confidence pill (⬡ 64% conf) — see below
• Tagged badge (green) if all cases in the cluster already carry the campaign tag
• Filter IOCs button — filters the Shared IOCs table to this cluster
• ✨ Analyze cluster button — generates or shows the AI cluster narrative
• Export STIX button — downloads a STIX 2.1 bundle for the cluster
• Push to MISP button — publishes the cluster to MISP as a campaign event, with entity names redacted; see MISP Cluster Publishing

Decay score

How fresh a cluster's indicators are, on a 0–100% scale. Exponential decay with a half-life set by IOC type:

Type	Half-life
ip-dst, ip-src	14 days
url, uri	30 days
domain, hostname	45 days
email-*	90 days
filename, regkey, mutex	120 days
hash variants (md5, sha1, sha256…)	180 days
anything else	60 days

Tags scale the half-life, not the score. An indicator attributed to a tracked threat actor stays useful roughly 1.4× longer; one flagged as CDN or public-resolver noise decays around twice as fast. Because the adjustment moves the half-life, the score cannot leave 0–100% however many tags apply.

Weights come from MISP galaxy attribution (threat-actor, ransomware, malpedia, tool), CIRCL incident classification, CSSA sharing class, and the extractor's noise flags. Free-text tags are matched on word boundaries as a lower-weighted fallback.

TLP does not affect decay. It expresses a sharing restriction, not fidelity or longevity — and correlation already excludes TLP:RED and AMBER entirely.

Age is measured from the most recent case containing the indicator, not the oldest. Re-observation is evidence that something is still live, so a campaign running across several months does not score staler the longer it persists.

Labels: Active (≥ 60%) / Aging (≥ 25%) / Stale. A cluster whose cases are all closed is capped at Aging — closed cases are not active work.

IOC confidence

How much the overlap justifies treating the cases as related. Not a function of the shared-IOC count alone — counting indicators treats a public DNS resolver and a unique C2 hash as equal evidence, which they are not.

Each shared indicator contributes according to two things:

• Rarity. An indicator present in most of your cases carries almost no weight; one present in only the cluster's cases carries a lot. This is inverse document frequency over the cases in scope, so 8.8.8.8 contributes close to nothing while a bespoke lookalike domain dominates.
• Credibility. Admiralty-scale reliability and information-credibility tags raise or lower an indicator's weight, and the extractor's noise flags (CDN, public resolver, parked, sinkhole) reduce it sharply.

The weighted total is then discounted by graph cohesion — how close the cluster is to fully connected. Clustering is single-linkage, so A–B plus B–C forms one cluster even when A and C share nothing; a chain is a weaker claim than a triangle of the same size.

Each cluster also reports cohesion, min_edge_weight (the weakest link holding it together) and distinctive_evidence (the rarity-and-credibility-weighted total, in units of "effective distinctive indicators"). These appear in the confidence pill's tooltip.

Confidence does not respond to the Min shared IOCs control. That is a view filter, and a measure of evidence should not move when you change how you are looking at it.

Expect lower figures than releases before IRIS-NG-v1.1.1, and markedly lower ones on instances with few cases — with a handful of cases, nothing is statistically distinctive yet. The earlier score counted indicators; this one weighs them.

D3 force-directed graph

• Cases = nodes (coloured by cluster)
• Shared IOCs = edges (thicker = more shared)
• Click a node to navigate to the case
• Drag to pan, scroll to zoom (0.2×–4×)

The graph only appears when the Correlation tab is active — hidden tab panes have offsetWidth = 0, causing D3 to draw all nodes at (0, 0). The graph defers its draw to the shown.bs.tab event.

D3 v7 is vendored locally at ui/public/assets/js/plugin/d3.v7.min.js — the script-src 'self' CSP blocks CDN loads.

Shared IOC click-through drawer

Clicking any row in the Shared IOCs table opens a slide-in drawer with per-case context:

• Case name / client / classification / severity / open-closed status
• IOC tags and case-level tags (violet chips)
• IOC description (ioc_description on the Ioc row)
• Linked notes with rendered markdown snippets

Ioc.ioc_tags is comma-separated (e.g. persistence,lateral-movement,T1053.005). Split on /[,|]/, not '|' alone.

The drawer is implemented in vanilla JS (document.createElement for escaping, fetch() for the API call) — jQuery ($) may not be loaded when inline scripts in content blocks run.

Note snippets are server-rendered via mistletoe (_render_md()). When the matched line is inside a markdown table, _extract_snippet() expands upward/downward to the full contiguous table block before rendering — a mid-table snippet produces a <table>, not <p>.

Per-IOC cross-case panel

On the edit-IOC modal (existing IOCs only), Check other cases → GET /api/v2/correlation/ioc-context renders the same enriched per-case cards. Note references use:

1. Formal IocNoteLink rows (labeled sourced from)
2. Substring fallback (labeled mentioned in)

Note chips show title only (per-IOC modal). The Shared IOC drawer shows title + snippet.

Applying a campaign tag

Apply campaign tag button on a cluster card:

POST /api/v2/correlation/apply-campaign-tag
{
  "tag": "campaign:cluster-28a0f9e1",
  "case_ids": [3, 7, 24],
  "shared_ioc_pairs": [{"ioc_value": "...", "ioc_type_id": 5}]
}

• Applies the tag to all cases in the cluster
• Also appends the tag to ioc.ioc_tags for each matching IOC in those cases
• Response includes cases_tagged and iocs_tagged counts

The cluster card re-renders with a green Tagged badge. The "Untagged clusters" summary card derives from the same per-item predicate — it counts clusters where NOT all cases already carry the suggested_campaign_tag.

AI cluster narrative

POST /api/v2/correlation/cluster-narrative

{
  "cluster": { ... },
  "case_meta": { ... },
  "force": false
}

Server-side cache — case_ai_artifact anchored to min(cluster.case_ids), kind = 'cluster_narrative:<cluster_id>', input_hash = MD5(payload+prompt+model). The cache invalidates naturally when cluster composition or the active model changes. force: true bypasses it.

Client-side cache — CORR._narrativeCache[cluster_id] for toggle-without-API-call within the page session.

Footer shows prompt_id · model · cached · generated_at · #<djb2-hash> · Re-run.

The suggested_name from the narrative is injected inline next to the cluster ID.

Entity-name prohibition (v2 prompt, load-bearing for STIX safety): the prompt explicitly forbids echoing any specific organization name, client name, or case identifier in its output. Victims are described by sector role only (e.g. "a water utility", "two energy-sector organizations"). This makes cached narratives safe to embed in STIX bundles shared with third parties.

When editing the prompt, bump both the # ClusterNarrativeSystemPrompt-<N> header in cluster_narrative.md AND the PROMPT_ID constant in cluster_narrative.py — existing v1 cache entries miss automatically on the next access (input_hash includes the full prompt text), so no manual cache clearing is needed.

Correcting a narrative by hand

An ✎ Edit button in the narrative header opens the campaign title and the prose for editing. This is for fixing a narrative that is mostly right rather than re-rolling the model and hoping for a better result.

• Saving marks the narrative as analyst-edited (pen icon, "edited by <user>", and an edited timestamp in the footer), and adds View AI original — a client-side toggle, no extra request — and Revert to AI.
• The corrected title also replaces the campaign name shown beside the cluster ID.
• confidence is not editable. It grades the underlying correlation data rather than the wording, and an analyst-authored value there is hard to interpret later. The edit badge carries the human-correction signal instead.
• The model's original output is never destroyed, so revert is always available.

Re-run is guarded. Because each generation inserts a new artifact row and reads take the latest, re-running over an edited narrative would silently discard the corrections. POST /cluster-narrative returns HTTP 409 (reason: manual_edit_present) unless the body carries discard_edit: true. The guard is server-side, so API clients get it too; the UI turns it into a confirm dialog, and declining restores the panel.

Shared mechanics and how to extend this to other AI surfaces: AI Features → Manual override.

Edits reach the STIX bundle — see the note in the next section.

STIX 2.1 export

Export STIX button on each cluster card downloads a self-contained STIX 2.1 bundle:

• identity — iris-ng identity object
• marking-definition — TLP:GREEN well-known object (34098fce-...)
• campaign — cluster summary (N cases, M shared IOCs)
• indicator per shared IOC — full pairs list, not the 20-value display truncation
• relationship — each indicator --indicates--> the campaign

AI narrative enrichment: if "Analyze cluster" has been run, the STIX endpoint looks up the most recent CaseAiArtifact (kind='cluster_narrative:<cluster_id>', anchored to min(cluster.case_ids)) before building the bundle. If found:

• suggested_name → campaign.name (the machine slug moves to campaign.aliases)
• Narrative prose → appended to campaign.description

The endpoint falls through silently when no narrative is cached — the bundle is always generated, enrichment is best-effort.

Manual edits are exported. The lookup reads display_content, so if the narrative has been corrected by hand your title becomes campaign.name and your prose becomes campaign.description. Partners receive the analyst's version rather than superseded model text. Note that the v2 prompt's entity-name prohibition constrains what the model writes; text you type by hand is your own responsibility, so keep organization and client names out of a narrative you intend to share. See AI Features → Manual override.

CaseAiArtifact field names (load-bearing): the model text is in .content (not .artifact_content) and the timestamp column is .generated_at (not .created_at). Read rendered text through .display_content, which returns the analyst edit when one exists and .content otherwise. Wrong field names cause AttributeError → HTTP 500 → Chrome shows "Site wasn't available" in the download history.

GET /api/v2/correlation/clusters/<cluster_id>/stix
  ?min_shared=2&start_date=YYYY-MM-DD&end_date=YYYY-MM-DD

The same filter params as /report are required so the cluster is reproducible. Returns 404 when cluster_id is not found under the current params.

IOC type → STIX pattern mapping is in source/app/iris_engine/stix_export.py::_ioc_pattern() (30+ type slugs; unknown types fall back to a valid x-iris-ng-indicator custom SCO). IDs are deterministic UUID v5 (namespace 00abedb4-...) — the same IOC always maps to the same indicator ID across exports. ioc_confidence (float 0–1) maps to STIX confidence (int 0–100). valid_from = earliest open_date from cluster's cases.

Endpoints

Method	Path	Description
GET	/api/v2/correlation/report	Full correlation report (clusters + graph data + IOC table)
GET	/api/v2/correlation/ioc-context	Cross-case context for a single IOC (per-IOC modal)
POST	/api/v2/correlation/apply-campaign-tag	Tag all cases in a cluster + their shared IOCs
POST	/api/v2/correlation/cluster-narrative	Generate (or return cached) AI narrative for a cluster. Returns 409 if the stored narrative was manually edited — pass discard_edit: true to override
PUT	/api/v2/correlation/cluster-narrative/edit	Save an analyst correction (body: cluster_id, case_ids, suggested_name, narrative)
DELETE	/api/v2/correlation/cluster-narrative/edit	Discard the correction, restore the AI original (body: cluster_id, case_ids)
GET	/api/v2/correlation/clusters/<id>/stix	Download STIX 2.1 bundle for a cluster (enriched with the AI narrative, or your edit, if cached)
POST	/api/v2/correlation/clusters/<id>/misp-push	Publish the cluster to MISP as a campaign event. Returns 409 if already published — pass force: true to republish. See MISP Cluster Publishing

Query parameters for report and clusters/<id>/stix:

• start_date, end_date — inclusive date bounds (YYYY-MM-DD)
• min_shared — minimum shared IOC count (default 2)