detkit — unit-testing Sigma rules against sample events (would love feedback)

[ELSATOAH]

Something I kept hitting with detection-as-code: I can validate that a rule is well-formed Sigma, but that doesn't tell me whether it actually fires on the thing it's meant to catch, or whether it stays quiet on the benign logs sitting right next to it. That check usually happens by hand, or after the rule is already live in the SIEM.

So I built detkit to cover that gap. You keep a small test file next to each rule with sample events and what you expect to happen:

# whoami_execution.test.yml
tests:
  - name: fires on whoami
    event: { EventID: 4688, CommandLine: "cmd /c whoami /all", User: "alice" }
    expect: match
  - name: quiet for SYSTEM
    event: { EventID: 4688, CommandLine: "whoami", User: "SYSTEM" }
    expect: no_match

detkit test ./rules evaluates each rule's condition against its events and exits non-zero on a mismatch, so it drops into CI or pre-commit and blocks a PR that breaks a detection. Basically dbt test, but for detections.

A few notes:

• It runs the rule logic locally.
• It covers the common Sigma subset (contains/startswith/endswith/re, wildcards, cidr, nested fields, X of conditions). I ran it against this repo's ruleset and about 91% use only features it handles today.
• When it hits something it can't evaluate yet (base64 modifiers, arrays of objects), it prints a WARN naming the feature instead of returning a maybe-wrong answer. A detection tool that's quietly wrong felt worse than none.
• It's the testing half of the picture. It pairs with the Sigma quality linters people are building, it doesn't replace them.

Repo: https://github.com/ELSATOAH/detkit (MIT, pipx install detkit-cli)

Mostly I'd like to know how you all test detections today. Is a local, event-based test runner something that would fit your workflow, and what would it need to handle to earn a place in your CI? Very happy to hear that I've reinvented something or missed an obvious approach.