Skip to content

Lab SIEM EDR and Hardware Verification Matrix

github-actions[bot] edited this page Nov 15, 2026 · 1 revision

Opened: November 11, 2026 ยท This is an evidence ledger and reproducible acceptance plan, not proof that these deployments work in a current environment.

Interpretation & Evidence Tiers

Static only means syntax, documentation or a specific isolated fixture may be checked; lab blocked means no real device/SIEM/provider outcome has been provided; external provider required means an authorized API tenant, version and credentials are required. Do not copy a successful source parser status onto network coverage, a physical radio, a live endpoint agent or a customer device.

This matrix inventories high-use canonical guides and separates observed, supported evidence from still-required experiments. Dates below refer to inventory review, not runtime verification. For a live test to count, use the record template below and link the sanitized evidence only if it contains no private deployment details. Keep client names, internal IPs, raw alerts and credentials in secure engagement records.

Repeatable Positive/Negative Assessment Tool

The Offline Canary Detection Evidence Validator now provides a JSONL input contract, positive/benign-negative fixtures, and machine-readable true-positive/true-negative/false-positive/false-negative/unassessable outcomes. It checks case IDs, source, timezones, duplicate/missing telemetry, rule IDs and latency. The Python tool never contacts any SIEM, and its real_siem_validated flag remains false even when user-supplied exports are accepted. It is a building block for future authorized hardware/manager evidence, not a substitute for that test.

Observed Automated Fixture Evidence โ€” November 11, 2026

The Wireless IDS CI workflow and Guide Fixture CI workflow exercise offline Python unit tests, synthetic event normalization, a localhost-only HTTP decoy/webhook, and structural/regex-only Wazuh rule contracts on an ephemeral GitHub-hosted Ubuntu runner. The full PR #195 CI results are the record for the actual commit, runner and success/failure outcomes; do not transfer these claims to a different Wazuh version or device.

The audit identified and corrected malformed Wazuh XML: rule 100340 had been placed inside the grouping label of rule 100300, and 100402 inside the grouping label of 100400. The new offline tests assert root-level rule structure, unique IDs, local ancestry, and synthetic positive and benign-negative field patterns; they explicitly do not run /var/ossec/bin/wazuh-logtest or simulate Wazuh frequency/timeframe/grouping logic.

Scope Positive Synthetic Expectation Benign/Negative Expectation Actual Evidence Available
Kismet normalizer + Wazuh candidate rules wids.integration=kismet, DEAUTHFLOOD => structural candidate 100301; correlation => 100340 CHANCHANGE must not match the deauth child; other integration must not match GitHub Python fixture result only โ€” no actual manager alert
nzyme webhook + candidate rules disconnect_anomaly => structural candidate 100352 system_event must not match 100352; instead 100354 Loopback webhook status/JSONL and regex fixture tests; actual nzyme provider API not exercised
DIY HTTP honeypot + candidate rules GET => 100400; POST => 100400 + 100401; ESP32 serial => 100402 GET must not match elevated write-method 100401 Localhost-only HTTP and schema fixtures; actual Pi/ESP32 and Wazuh ingestion not exercised

Observed detection score: Not available. A passing code fixture demonstrates test-contract consistency, not actual sensor visibility, analyst detection, low false-positive rates, or an approved deployment. Log the real manager's match/level and non-match results with wazuh-logtest on a separately authorized Wazuh 4.x environment; record the exact installed release and ruleset commit, then check event ingestion and dashboard alert correlation. Do not deploy candidate rules directly to production from CI.

Initial Procedure Inventory

Canonical Procedure Evidence Available Unverified Condition / Acceptance Experiment Current Status
OSINT Case Investigator Isolated shell fixtures in hardening tests cover named behavior Current registry/API network response, rate limits, actual report quality and consent Static fixture only
Kismet โ†’ Wazuh Wireless Alerts Source/configuration, synthetic bridge fixtures and XML group/regex contract tests exist Kismet release, capture chipset, firmware, monitor mode, real 802.11 frames, end-to-end Wazuh ingestion and rule firing Lab blocked
nzyme โ†’ Wazuh Guide and local adapter/config content nzyme version, webhook semantics, signature, TLS, observed source/destination timestamps, false-positive handling External provider / lab blocked
Suricata & Zeek Documentation and example detection configuration Verified TAP/SPAN visibility, sensor interface, IDS ruleset revision, PCAP replay and actual alert + Zeek log correlation Lab blocked
Wi-Fi Disconnect Detection Defensive detection guidance and filter examples 802.11 frame capture, PMF state, negative benign disconnect fixtures, severity fidelity in SIEM Lab blocked
Graylog Deployment Historical Docker example (Graylog 5.2; OpenSearch 2.12.0) Supported version/migration, TLS, secrets, inputs, index writes, alert-to-notification chain Historical; not deployment-verified
Elastic SIEM/ELK Example compose/health check procedure Current image compatibility, secure account setup, ingestion, query/rule results and recovery after restart Lab blocked
Snort/IDS Integration Architectural guides Actual mirrored traffic reaching monitored interface; rule and SIEM event chain with timestamped capture Lab blocked
DIY Sensor Honeypot Local-only demo, loopback HTTP fixture and repaired XML rule group with synthetic positive/negative contract tests Isolated canary health-check, synthetic hit, JSONL/event schema validation, Wazuh intake, cleanup Lab blocked
Raspberry Pi Honeypot Hardware setup procedure Specific Pi model, OS kernel, network segmentation, capture, watchdog and SIEM correlation Hardware blocked
NanoVNA Fixture test runner for lab files Actual model, calibration standards, firmware, connection, measured sweep against known reference and RF safe handling Fixtures only; hardware blocked
ESP32 Security Firmware Catalog/source metadata only Exact board revision, bootloader/flasher release, backup/recovery, passive receive test, approved RF scope Hardware blocked
M5StickC Plus2 Source and board guidance Actual display/IR/RF peripherals, battery, firmware compatibility and reset/recovery Hardware blocked
PowerShell Port Scanner Source parser checks; bounds corrected Owned loopback listener, refused/timeout paths, DNS errors, timeouts, outbound-scope control and observed PowerShell version Static syntax only
AnythingLLM CVE Lookup Reviewed reference and risk boundaries Actual NVD/CISA/FIRST provider responses, valid JSON parsing, stale/unknown values, no invented remediation External provider required
Data Engineering Fixtures CI calls offline runner External broker/store and real telemetry behavior, ingestion latency, security of credentials and rollback Static/fixture only

Repeatable Test Sequence

  1. Approve scope: record named operator, authorization window, written owner approval, isolated lab network, allowed simulated activity, stop conditions, and emergency contact. A fixture is not approval for external or production targeting.
  2. Baseline: record OS/kernel, device model and revision, firmware/build, SIEM/EDR release, ruleset ID/revision, time sync, listening interfaces, monitoring points, target host, and expected sensor visibility. Use artificial IDs and RFC 5737 documentation addresses in public examples.
  3. Positive test: feed an approved PCAP or synthetic event into the pipeline; document event/alert time, log fields, source component, collector, rule identifier, rule version, and exact expected/observed detection. Do not automatically execute LLM-generated exploit code.
  4. Negative/benign test: supply similar harmless activity and empty/invalid input; measure false positives, dropped events, parse errors, crash/retry behavior and missing telemetry. A positive alert with zero negatives is insufficient to rate the rule effective.
  5. Score: score separately for sensor visibility, telemetry completeness, rule trigger, correlation/context, latency, and false-positive behavior. Use observed/failed/not-tested for every dimension instead of manufacturing a composite detection percentage.
  6. Cleanup & sign-off: record final state, disable test accounts, stop temporary listeners, delete synthetic artifacts per retention policy, roll back agent/rule changes, and have a human approve deployment or mark it blocked.

Copyable Per-Test Record (Fill Privately For Client Systems)

Field Operator Entry
Lab/engagement authorization and scope Required before executing; store sensitive details privately
Guide path and exact Git commit Not recorded โ€” supply commit under test
Test ID, operator and UTC date Not tested โ€” fill on execution
Device/VM model, OS, kernel and firmware Not tested โ€” identify exact versions
SIEM/EDR and detection rule/version Not tested โ€” name product/build and rule ID
Positive synthetic fixture and checksum Not recorded โ€” use approved non-sensitive fixture
Benign/negative fixture and checksum Not recorded โ€” must be independently reviewed
Expected alert source, fields and severity Not recorded โ€” derive from tested rule, not a guess
Observed source/collector/alert timestamps Not tested โ€” preserve redacted evidence privately
Detection/false positive/false negative results Not tested โ€” specify sample sizes
Telemetry gaps and vendor/API limitations Not tested โ€” document failures too
Rollback, data retention and cleanup Not performed โ€” document exact completion
Approval and next action Awaiting authorized operator review

Evidence & Publication Policy

Only anonymized fixture outputs, SHA-256 checksums of synthetic data, public documentation references, scoped test commands and non-sensitive summarized outcomes belong in this repository. Do not publish client log streams, telemetry containing personal data, deployment-specific addresses, tokens, proprietary SIEM rules or detailed exploitation output.

This is the starting work product for issue #190. The issue is not closed until specific hardware/vendor/SIEM tests have been performed and reviewed by an authorized operator.


Documentation Index ยท Verification Protocol ยท Maintenance Operations


๐Ÿ“„ Source: Documentation/LAB-VERIFICATION-MATRIX.md | โœ๏ธ Suggest an edit | โฌ…๏ธ Back to Technical Guides & Cheat Sheets

Clone this wiki locally