IOC Correlation

IOC Correlation

Overview

The Correlation tab on /dashboard surfaces IOCs that appear in multiple cases, groups them into clusters, and lets analysts apply campaign tags and generate AI narratives.

No new database tables — correlation is computed on-the-fly from the existing Ioc table using (ioc_value, ioc_type_id) equality.

How it works

1. Query Ioc for values appearing in ≥ N cases (HAVING COUNT(DISTINCT case_id) >= min_shared)
2. Build clusters via union-find; cluster_id = MD5[:8] of sorted case IDs
3. Compute per-cluster decay score and IOC confidence score
4. Render cluster cards + D3 force-directed graph

TLP filter: only TLP:GREEN (id=3) and TLP:CLEAR (id=4) IOCs participate in correlation surfaces. TLP:RED / AMBER / AMBER+STRICT IOCs are excluded to prevent cross-case information leakage on the shared dashboard.

Controls

• Quick-range buttons — 30d / 60d / 90d / 180d / all time
• Custom date range — From / To inclusive pickers
• Min shared IOCs — default 2; raise to focus on stronger overlaps

Cluster cards

Each cluster card shows:

• Case list with links
• Shared IOC chips
• Decay pill (● Active · 65%, color-coded green/amber/grey) — see below
• IOC confidence pill (⬡ 64% conf) — see below
• Tagged badge (green) if all cases in the cluster already carry the campaign tag
• Filter IOCs button — filters the Shared IOCs table to this cluster
• ✨ Analyze cluster button — generates or shows the AI cluster narrative

Decay score

Exponential half-life per IOC type:

Type	Half-life
ip-dst	14 days
domain	45 days
hash variants	180 days
default	60 days

Multiplied by tag weights (admiralty-scale reliability/credibility, CIRCL incident classification, MISP galaxy threat-actor/ransomware/malpedia/tool, TLP, noise flags). Age is anchored to Cases.open_date.

Labels: Active (high score) / Aging / Stale. All-closed clusters are capped at Aging.

IOC confidence

1 - 1/(1 + log2(count/min_shared + 1))

Shared IOC count	Approximate confidence
= min_shared (2)	~40%
5	~64%
10	~80%
36	~95%

D3 force-directed graph

• Cases = nodes (coloured by cluster)
• Shared IOCs = edges (thicker = more shared)
• Click a node to navigate to the case
• Drag to pan, scroll to zoom (0.2×–4×)

The graph only appears when the Correlation tab is active — hidden tab panes have offsetWidth = 0, causing D3 to draw all nodes at (0, 0). The graph defers its draw to the shown.bs.tab event.

D3 v7 is vendored locally at ui/public/assets/js/plugin/d3.v7.min.js — the script-src 'self' CSP blocks CDN loads.

Shared IOC click-through drawer

Clicking any row in the Shared IOCs table opens a slide-in drawer with per-case context:

• Case name / client / classification / severity / open-closed status
• IOC tags and case-level tags (violet chips)
• IOC description (ioc_description on the Ioc row)
• Linked notes with rendered markdown snippets

Ioc.ioc_tags is comma-separated (e.g. persistence,lateral-movement,T1053.005). Split on /[,|]/, not '|' alone.

The drawer is implemented in vanilla JS (document.createElement for escaping, fetch() for the API call) — jQuery ($) may not be loaded when inline scripts in content blocks run.

Note snippets are server-rendered via mistletoe (_render_md()). When the matched line is inside a markdown table, _extract_snippet() expands upward/downward to the full contiguous table block before rendering — a mid-table snippet produces a <table>, not <p>.

Per-IOC cross-case panel

On the edit-IOC modal (existing IOCs only), Check other cases → GET /api/v2/correlation/ioc-context renders the same enriched per-case cards. Note references use:

1. Formal IocNoteLink rows (labeled sourced from)
2. Substring fallback (labeled mentioned in)

Note chips show title only (per-IOC modal). The Shared IOC drawer shows title + snippet.

Applying a campaign tag

Apply campaign tag button on a cluster card:

POST /api/v2/correlation/apply-campaign-tag
{
  "tag": "campaign:cluster-28a0f9e1",
  "case_ids": [3, 7, 24],
  "shared_ioc_pairs": [{"ioc_value": "...", "ioc_type_id": 5}]
}

• Applies the tag to all cases in the cluster
• Also appends the tag to ioc.ioc_tags for each matching IOC in those cases
• Response includes cases_tagged and iocs_tagged counts

The cluster card re-renders with a green Tagged badge. The "Untagged clusters" summary card derives from the same per-item predicate — it counts clusters where NOT all cases already carry the suggested_campaign_tag.

AI cluster narrative

POST /api/v2/correlation/cluster-narrative

{
  "cluster": { ... },
  "case_meta": { ... },
  "force": false
}

Server-side cache — case_ai_artifact anchored to min(cluster.case_ids), kind = 'cluster_narrative:<cluster_id>', input_hash = MD5(payload+prompt+model). The cache invalidates naturally when cluster composition or the active model changes. force: true bypasses it.

Client-side cache — CORR._narrativeCache[cluster_id] for toggle-without-API-call within the page session.

Footer shows prompt_id · model · cached · generated_at · #<djb2-hash> · Re-run.

The suggested_name from the narrative is injected inline next to the cluster ID.

Entity-name prohibition (v2 prompt, load-bearing for STIX safety): the prompt explicitly forbids echoing any specific organization name, client name, or case identifier in its output. Victims are described by sector role only (e.g. "a water utility", "two energy-sector organizations"). This makes cached narratives safe to embed in STIX bundles shared with third parties.

When editing the prompt, bump both the # ClusterNarrativeSystemPrompt-<N> header in cluster_narrative.md AND the PROMPT_ID constant in cluster_narrative.py — existing v1 cache entries miss automatically on the next access (input_hash includes the full prompt text), so no manual cache clearing is needed.

Correcting a narrative by hand

An ✎ Edit button in the narrative header opens the campaign title and the prose for editing. This is for fixing a narrative that is mostly right rather than re-rolling the model and hoping for a better result.

• Saving marks the narrative as analyst-edited (pen icon, "edited by <user>", and an edited timestamp in the footer), and adds View AI original — a client-side toggle, no extra request — and Revert to AI.
• The corrected title also replaces the campaign name shown beside the cluster ID.
• confidence is not editable. It grades the underlying correlation data rather than the wording, and an analyst-authored value there is hard to interpret later. The edit badge carries the human-correction signal instead.
• The model's original output is never destroyed, so revert is always available.

Re-run is guarded. Because each generation inserts a new artifact row and reads take the latest, re-running over an edited narrative would silently discard the corrections. POST /cluster-narrative returns HTTP 409 (reason: manual_edit_present) unless the body carries discard_edit: true. The guard is server-side, so API clients get it too; the UI turns it into a confirm dialog, and declining restores the panel.

Shared mechanics and how to extend this to other AI surfaces: AI Features → Manual override.

Edits reach the STIX bundle — see the note in the next section.

STIX 2.1 export

Export STIX button on each cluster card downloads a self-contained STIX 2.1 bundle:

• identity — iris-ng identity object
• marking-definition — TLP:GREEN well-known object (34098fce-...)
• campaign — cluster summary (N cases, M shared IOCs)
• indicator per shared IOC — full pairs list, not the 20-value display truncation
• relationship — each indicator --indicates--> the campaign

AI narrative enrichment: if "Analyze cluster" has been run, the STIX endpoint looks up the most recent CaseAiArtifact (kind='cluster_narrative:<cluster_id>', anchored to min(cluster.case_ids)) before building the bundle. If found:

• suggested_name → campaign.name (the machine slug moves to campaign.aliases)
• Narrative prose → appended to campaign.description

The endpoint falls through silently when no narrative is cached — the bundle is always generated, enrichment is best-effort.

Manual edits are exported. The lookup reads display_content, so if the narrative has been corrected by hand your title becomes campaign.name and your prose becomes campaign.description. Partners receive the analyst's version rather than superseded model text. Note that the v2 prompt's entity-name prohibition constrains what the model writes; text you type by hand is your own responsibility, so keep organization and client names out of a narrative you intend to share. See AI Features → Manual override.

CaseAiArtifact field names (load-bearing): the model text is in .content (not .artifact_content) and the timestamp column is .generated_at (not .created_at). Read rendered text through .display_content, which returns the analyst edit when one exists and .content otherwise. Wrong field names cause AttributeError → HTTP 500 → Chrome shows "Site wasn't available" in the download history.

GET /api/v2/correlation/clusters/<cluster_id>/stix
  ?min_shared=2&start_date=YYYY-MM-DD&end_date=YYYY-MM-DD

The same filter params as /report are required so the cluster is reproducible. Returns 404 when cluster_id is not found under the current params.

IOC type → STIX pattern mapping is in source/app/iris_engine/stix_export.py::_ioc_pattern() (30+ type slugs; unknown types fall back to a valid x-iris-ng-indicator custom SCO). IDs are deterministic UUID v5 (namespace 00abedb4-...) — the same IOC always maps to the same indicator ID across exports. ioc_confidence (float 0–1) maps to STIX confidence (int 0–100). valid_from = earliest open_date from cluster's cases.

Endpoints

Method	Path	Description
GET	/api/v2/correlation/report	Full correlation report (clusters + graph data + IOC table)
GET	/api/v2/correlation/ioc-context	Cross-case context for a single IOC (per-IOC modal)
POST	/api/v2/correlation/apply-campaign-tag	Tag all cases in a cluster + their shared IOCs
POST	/api/v2/correlation/cluster-narrative	Generate (or return cached) AI narrative for a cluster. Returns 409 if the stored narrative was manually edited — pass discard_edit: true to override
PUT	/api/v2/correlation/cluster-narrative/edit	Save an analyst correction (body: cluster_id, case_ids, suggested_name, narrative)
DELETE	/api/v2/correlation/cluster-narrative/edit	Discard the correction, restore the AI original (body: cluster_id, case_ids)
GET	/api/v2/correlation/clusters/<id>/stix	Download STIX 2.1 bundle for a cluster (enriched with the AI narrative, or your edit, if cached)

Query parameters for report and clusters/<id>/stix:

• start_date, end_date — inclusive date bounds (YYYY-MM-DD)
• min_shared — minimum shared IOC count (default 2)