Skip to content

Dual Timeline

Jump to bottom
zach115th edited this page Jul 19, 2026 · 3 revisions

Dual Timeline

Overview

IRIS-NG adds a working timeline panel alongside the existing master timeline. The working timeline is a staging area: import bulk forensic tool output, review events individually, and promote the ones worth keeping to the master timeline.

Architecture

┌─────────────────────────────────────────────────┐
│  Case timeline page                             │
│                                                 │
│  [Master timeline]  │  [AI analysis panel]      │
│  (permanent record) │  (always-visible prose)   │
│                     │                           │
│  ─────────────────────────────────────────────  │
│                                                 │
│  [Working timeline panel] (right side)          │
│  import → review → promote / reject             │
└─────────────────────────────────────────────────┘

Master timeline events are permanent. Working timeline events are ephemeral staging entries in case_working_event.

Importing events

Hayabusa (JSONL)

POST /api/v2/cases/<cid>/working-timeline/import/hayabusa
Content-Type: multipart/form-data

file=<hayabusa-results.jsonl>
[begin_date=YYYY-MM-DD]   # optional, inclusive
[end_date=YYYY-MM-DD]     # optional, inclusive
csrf_token=<token>

Hayabusa's sigma-rule fan-out (1 EVTX event → N matched rules) is collapsed on (Timestamp, Computer, Channel, EventID, RecordID) so the panel shows one reviewable event per source event, not one per matched rule.

The EVTX filename in each card shows the basename only (full path preserved in event_raw.evtx_file). Collection paths from Windows tools use backslashes; always split on [\\/] — not os.path.basename() — in parsers running inside Linux containers.

EZ Tools / KAPE CSV (11 sub-formats)

POST /api/v2/cases/<cid>/working-timeline/import/eztools
Content-Type: multipart/form-data

file=<EZTools-output.csv>
[begin_date=YYYY-MM-DD]
[end_date=YYYY-MM-DD]
csrf_token=<token>

Sub-formats are auto-detected by column-header signature:

Sub-format Source tool
evtxecmd EvtxECmd
mft-usn MFTECmd (USN journal)
mft-full MFTECmd (full MFT)
prefetch-timeline PECmd Timeline CSV
prefetch-full PECmd Output CSV
appcompat AppCompatCacheParser
amcache-unassoc AmcacheParser (unassociated)
amcache-prog AmcacheParser (programs)
recycle-bin RBCmd
jumplist JLECmd
lnk LECmd

JumpList must be detected before LNK — the JumpList CSV is a superset of LECmd columns and would be mis-classified if the order were reversed.

csv.field_size_limit is set to sys.maxsize at module load — PECmd's Files / Directories cells routinely exceed Python's 128 KB default limit.

AppCompatCache forensic note: shimcache indicates the file existed, not that it executed. LastModifiedTimeUTC is MFT $STANDARD_INFORMATION ModifiedTimeUTC, not a last-run time. The parser emits <basename> present on disk titles (no "executed" suffix), severity stays None. Use Prefetch / Security 4688 / Sysmon 1 for execution confirmation.

PECmd prefetch folding: prefetch-full (PECmd_Output.csv) emits one event per prefetch keyed on LastRun, with previous run timestamps listed in the body. This avoids flooding the panel with 9 events per prefetch entry. Use prefetch-timeline if you want one event per run.

Master-timeline CSV → working timeline

The "Upload CSV of events" modal now offers a Master timeline / Working timeline radio. Choosing Working timeline routes to:

POST /api/v2/cases/<cid>/working-timeline/import/master-csv
Content-Type: multipart/form-data

file=<iris-export.csv>
[begin_date=YYYY-MM-DD]
[end_date=YYYY-MM-DD]
csrf_token=<token>

The canonical 10-column export format is accepted: event_date,event_tz,event_title,event_category,event_content,event_raw,event_source,event_assets,event_iocs,event_tags

Assets and IOCs are not auto-linked at import — promote individually to trigger resolution.

Date-range filter

All three import endpoints accept optional begin_date / end_date (YYYY-MM-DD). Filtering is server-side with inclusive UTC day bounds (begin → 00:00:00, end → 23:59:59.999999).

Rows with unparseable timestamps (event_date is None) are always kept — a filter must never silently swallow unplaceable data.

Reviewing events

Each working-timeline card shows:

  • Event timestamp, title, sigma rule match (Hayabusa), channel/EventID
  • ✨ Explain pill — LLM 3-paragraph explanation (what it detects / what happened / triage hint)
  • Promote / Reject / Reset buttons

Explain pill 3-state toggle:

  1. First click — fetches cached explanation (or generates if not cached)
  2. Second click — hides (artifact stays cached)
  3. Third+ click — re-renders from in-memory cache, no API call

The list endpoint bulk-joins cached explanations so they remain visible after promote/reject without N+1 follow-up GETs.

Promoting events

POST /api/v2/cases/<cid>/working-timeline/events/<eid>/promote

At promote time (not import time):

  1. Asset materialization — asset_resolver.py auto-creates CaseAssets rows for the host computer and subject/target users
    • Windows Computer/Server/DC heuristic by hostname
    • Windows Account AD/Local heuristic by domain presence
    • asset_domain populated from FQDN (bare hostnames and IPv4-prefixed strings → None)
  2. AI IOC extraction — ioc_resolver.py calls the IOC extractor (confidence ≥ 0.70, noise-flagged candidates dropped), find-or-creates on (case_id, type_id, value)
  3. Cross-links — update_event_assets(..., sync_iocs_assets=True) creates case_events_assets and IocAssetLink rows

Promoted events arrive on the master timeline flagged (event_is_flagged=True) — the flag is a "needs review" marker. Unflag once reviewed.

ISO timestamp serialization

Working timeline events store event_date as a naive datetime representing UTC.

The serializer appends Z (_iso_utc() in working_timeline.py) so browsers parse it as UTC rather than local time. Without the Z, new Date(iso) interprets the string as local time, and .getUTCHours() adds the offset on display.

This same latent bug exists on master-timeline cases_events.event_date — watch for it if you add new ingest sources that store naive non-UTC timestamps.

Master-timeline flag-by-default

New master-timeline events (manual add or promote from working) arrive with event_is_flagged = True. The flag means "needs review"; analysts uncheck it once verified. The running AI analysis panel treats:

  • is_flagged: false → HIGH confidence (reviewed, treat as fact)
  • is_flagged: true → MEDIUM confidence (provisional, hedge in narrative)

Event history

A History item in the event modal's ⋮ dropdown shows the event's full lifecycle from CasesEvent.modification_history JSONB. Lifecycle points recorded:

  • created (manual)
  • promoted from working timeline (<source>)
  • flagged / unflagged

Date sort toggle

Both timelines have a violet sort pill that toggles between newest-first and oldest-first display. The sort is client-side only — the list endpoint always returns events in database order; the UI re-renders on click.

Master timeline

A violet Newest first ▼ / Oldest first ▲ pill appears immediately to the right of the first grey date badge in the master timeline. Clicking it reverses the display order and re-injects the pill next to the new first date badge.

The sort state is preserved across build_timeline() calls (e.g. when a new event is added or filtered) via the module-scoped _tm_sort_dir variable.

Working timeline

Each date group row in the working timeline panel shows:

  • Left — violet sort pill (Oldest first ▲ / Newest first ▼)
  • Right — grey date badge

Clicking the pill calls irisWorkingTimeline.toggleSort(), which reverses _wt_sort_dir and re-renders the panel via irisWorkingTimeline.refresh().

Both sort toggles default to newest-first (▼) on page load.

Adding a new ingest source

  1. Create source/app/iris_engine/working_timeline/<source>_parser.py
    • Return a list of CaseWorkingEvent objects
    • Split Windows paths on [\\/] not os.path.basename()
    • Fold repeated-execution records (one card + secondaries in body, not N cards)
  2. Add an IMPORT_SOURCES JS entry and a dropdown item in the import modal
  3. Add a new /import/<source> endpoint in working_timeline.py
    • Reuse _read_date_window / _filter_rows_by_window for the date-range filter
    • Set working_event.source to a descriptive discriminator string

The case_working_event.source column already carries the discriminator for filtering and display.

Deduplication

On-demand duplicate detection on both timelines. Access via:

  • Master timeline — "Find duplicates" item in the ⋮ dropdown
  • Working timeline — clone icon on the panel header

Exact duplicates

Auto-resolved in one click. Duplicates are identified by a normalized key:

Timeline Key
Master (event_date, norm(event_title), norm(event_content))
Working (event_date, norm(event_title), norm(event_content), norm(event_source_host))

Normalization = lowercase, trim, collapse whitespace.

The earliest event_added / created_at row is kept; the rest are deleted.

POST /api/v2/cases/<cid>/timeline/dedup/scan
→ { exact_groups: [...], near_pairs: [...] }

POST /api/v2/cases/<cid>/timeline/dedup/auto-exact
→ { resolved_groups: N, deleted_events: M }

Near duplicates

Pairs where difflib.SequenceMatcher token-sort ratio ≥ 0.75 and event_date is within 5 minutes are surfaced in a side-by-side modal.

Actions per pair:

  • Keep left — delete the right event
  • Keep right — delete the left event
  • Merge — edit the merged content in the pre-populated editor, then choose which event ID to keep (the other is deleted)
  • Base on left / Base on right — swaps the pre-populated content in the merge editor
POST /api/v2/cases/<cid>/timeline/dedup/resolve
Body: { keep_id, delete_ids[], merged_fields?, target: "master"|"working" }

Load-bearing: event deletion requires FK cleanup

The upstream schema has no ON DELETE CASCADE on four join tables. Always use _delete_master_event() in dedup.py (or an equivalent) before deleting any CasesEvent row — bare db.session.delete(ev) raises IntegrityError:

Table FK column
case_events_assets event_id
case_events_ioc event_id
case_events_category event_id
event_comments comment_event_id

No migration, no rebuild

All dedup files are volume-mounted Python + Jinja templates. Deploy with:

docker restart iriswebapp_app iriswebapp_worker

patreon.com/zach115th
buymeacoffee.com/zach115th

Clone this wiki locally