Skip to content

Alert Intake and Correlation

Krishnendu Bhattacherjee edited this page Nov 10, 2026 · 2 revisions

Alert Intake and Correlation

JSON, CSV and EVTX alerts are normalized with source provenance. The timeline builder groups shared identifiers and splits sessions; default gap is 30 minutes and the event cap is 100.

EVTX intake

The EVTX adapter uses evtx==0.15.0 (MIT, Rust-backed; Windows x64 Python 3.10+ wheels). Keep large inputs outside the repository. This example parses a local file without enrichment or LLM calls:

from dataclasses import asdict
from src.engine.evtx import EvtxIntakeEngine

intake = EvtxIntakeEngine()
for batch in intake.ingest_evtx(r"C:\soc-data\events.evtx", batch_size=100):
    # Persist/process each batch here; do not collect the entire iterator.
    print("Alerts in batch:", len(batch))
print(asdict(intake.stats))

AlertIntakeEngine.ingest_directory() also accepts .evtx (case insensitive); its EVTX counters are available as engine.evtx.stats. Normalized alerts carry EventID as event_type, UTC timestamp, and original_record fields EventID, Channel, ComputerName, User, Details, and local-only raw_xml. Details flattens EventData/UserData with dotted paths, bracket indices for repeated siblings, and backslash escaping of literal dots/brackets/backslashes. User selection prefers EventData User, TargetUserName, SubjectUserName, then the System Security SID. Missing values remain unknown; naive times assume UTC, invalid/missing times are counted and retained as None; precision is microseconds. Source paths stay in source_file provenance, excluded from LLM prompts along with raw XML. Dataset labels and indicators are not inferred by this adapter.

Limits: unknown channels are retained. Parser errors count failed operations, which may represent whole chunks; they cannot measure the number of lost records. Normalization failures count rejected records. File errors and abandonment after 100 consecutive parser errors increment files_failed; readable files containing recoverable errors can have files_failed=0. Counters accumulate per adapter. The adapter uses one native parser thread with checksums enabled and recovery disabled, buffers one native chunk plus bounded alert batches, and rejects XML over 1,048,576 characters or event-data nesting over 32 levels. Full CLI correlation still collects alerts in memory; use the adapter directly for large-file intake. Offline tests cover synthetic files and parser-output fixtures. A separate 128 MiB synthetic native-parser check emitted 2,048 events with 0.50 MiB observed working-set growth after warm-up (batch 25, one parser thread). This does not establish real-data accuracy, throughput, or memory ceilings for unusual templates.

Wiki home · Project README

Clone this wiki locally