AiSOC v21.0.0 — the eval-honesty major

[beenuar]

AiSOC v21.0.0 is out — the eval-honesty major. Release notes · Technical Guide PDF, v1.2

The headline: the platform's numbers now carry their own provenance

• Every verdict scorecard names the model that actually served it, with an explicit provenance status (gateway-attested, self-reported, or unattributed). A run with missing attribution, zero real calls, or a deterministic fallback can no longer be advertised as a named model's performance — and an autonomy measurement that cannot be bound to an attributed call cannot authorize closures.
• Denominators are published, and the uncomfortable direction wins. Full-cohort recall keeps abstentions and failures in the malicious denominator; closure claims ship with their contamination rate; request reliability travels as requested/attempted/completed/failed/unattempted so a failure cannot vanish from a headline.
• Alert reduction now has three labelled figures instead of one flattering one: 75.3% (legacy grouping — not what production runs), 33.3% (the production key), and 24.5% on a detection-shaped cohort after this release's correlation fix, where a missing identifier narrows the group instead of widening it. The drop is the honest direction: the difference was unrelated alerts counted as grouped.
• Replay is time-frozen and re-hunting is labelled: evidence cutoffs are enforced in the retrieval queries (never by prompt), the model-knowledge limit is disclosed on historical claims, replay cannot execute real actions, and retro-hunt gained a rehearsal mode whose mutation boundary is the write adapter.
• Model suitability is per task, measured against a deterministic baseline floor (balanced accuracy exactly 0.250). Unfunded rows say not measured — with one-command reproduction against any local OpenAI-compatible endpoint.

Also in this release

• Business context reaches the triage model — tenant-authored rules are named in the prompt, and an impact statement survives only when it cites the rules that actually fired. Asset criticality now orders the investigation queue, weighs fusion confidence (unknown renders unknown, never zero-as-evidence), and tighten-only raises approval impact on critical assets — a deployment with no CMDB loses nothing.
• AI coding-agent telemetry as an input: a bundled Agentmetry ingest template (community request in AI coding-agent activity (Claude Code, Cursor, Codex) as an input to AiSOC #1577 — thanks @blitzcrieg1), with per-event-kind OCSF routing and the five-tier severity ladder preserved end to end.
• An ack-clock escalation bridge (externally contributed): unacknowledged CRITICAL cases get a Teams card with capped re-nags; HIGH arrives as a daily digest.
• The Investigation Rail shows which business rules fired and the cited impact; the reflex path reports nine named stages and a structured decision contract.

Honesty ledger

3 defects fixed in this release, 15 across it and the two before (3 here, 2 in v20.4.0, 10 in v20.3.0 — distinct product or CI defects named in Fixed/Security sections, dependency advisories excluded). The behavior change behind the major: correlation grouping narrows on missing identifiers (future alerts only; no API change), and pre-v21 scorecards are not comparable to v21 ones. Neither stable channel moves — v21 is an application major, and both channel guards hold until a deliberate promotion.

Upgrade: git pull && make up, or pull the v21.0.0 image tags. The README has the new evaluation/reflex/replay section, and the benchmark page carries the methodology.