Skip to content

v0.5.0

Choose a tag to compare

@supermhel supermhel released this 28 Jul 22:21
· 369 commits to main since this release
c3ec328

[0.5.0] - 2026-07-23

Added (M2 proof artifacts + M7 continuous tracks, 2026-07-22)

  • mypy blocking gate: re-measured the stale "20 findings" baseline before
    trusting it — a live re-run found 47 across all 8 workstreams, not 20.
    Fixed all 47 with narrow, behavior-preserving changes; CI's || true on
    the mypy step removed. Regressions now fail CI the same way ruff/coverage
    already do.
  • Mutation-testing gate (mutmut): first-ever run in this repo, scoped to
    services/shared/sessions.py (a directory-wide run would take hours per
    mutant re-run here). Measured baseline: 142 mutants, 50 covered, 36
    killed / 14 survived (72% kill rate) — informational in CI, same
    measure-first sequencing mypy went through before it was flipped blocking.
  • MITRE empirical firing check (eval/attack/fire_check.py): closes the
    gap between the existing declared-coverage scorecard and the dataset-gated
    real-world replay lane — every MITRE-tagged rule is replayed against its
    own real producer fixture through the actual detection engine. 26/26
    tagged rules fire. Found and fixed a real harness bug along the way
    (synthetic timestamps stepping into the future tripped the anti-poisoning
    clock-skew guard, falsely showing 2 real rules as silent).
  • Observability: Prometheus + Grafana: new /metrics/prom exposition-
    format route (hand-rolled, stdlib-only — no new runtime dependency),
    opt-in observability compose profile with auto-provisioned Grafana
    dashboard. Live-verified: all 5 scrape targets up, real pipeline counters
    queryable. OTel tracing stays explicitly out of scope (bigger lift, still
    an ADR-only aspiration).
  • Modbus/TCP protocol-anomaly detector (modbus_anomaly.py), the second
    OT source after OPC UA — deliberately scoped as an anomaly detector over
    the protocol's own public function-code table, not a vendor-log parser
    (Modbus has no audit-log format to parse). New rule
    ot_modbus_unauthorized_write.yml (ATT&CK-ICS T0855) ships with a real
    producer and passes both the anti-dormancy gate and the new firing check.

Added (post-merge CI hardening, PR#2 → main)

  • CodeQL's first-ever live scan (2026-07-18/19) found 5 HIGH alerts, all fixed at
    the design level, zero dismissed: tenant_id path-injection in rules_view.py
    closed by never letting request data into a path expression (trusted-dir glob +
    stem match); the RBAC first-boot admin password redesigned to operator-supplied
    via FENGARDE_ADMIN_PASSWORD (the service never generates/logs/stores plaintext;
    unset + empty store now fails closed with a loud warning). Open CodeQL alert count
    on main: 0.
  • Three new CI gates: pip-audit (CVE audit of every pinned requirements.txt,
    the deliberate replacement for disabled Dependabot), docker-build (all 8 images
    must build — nothing previously proved this in CI), actionlint (lints the
    workflows themselves; immediately caught a real dead-var in the mypy step).
  • Supply-chain pinning: every workflow uses: SHA-pinned, all 8 Docker base
    images digest-pinned, least-privilege permissions: on every workflow — Scorecard
    alerts 54→19. The remaining 19 are accepted, not open work (11 want full
    transitive pip hash-pinning, declined as maintenance-disproportionate given
    version-pins + the pip-audit gate; 8 are practice/settings-level signals that
    accrue with repo history or are an explicit choice — Dependabot's own removal
    below is one of them).
  • Dependabot removed (.github/dependabot.yml) in favor of pip-audit's
    CVE-driven gate — a deliberate choice, not an oversight; see the pinning note above.
  • Fixed since PR#2 landed red: ruff F401 in runner.py; ossf/scorecard-action@v2
    pinned to v2.4.3 (no such floating tag existed); quality job missing
    hypothesis; the coverage gate's hand-synced test list re-synced post-merge.

Added (v0.5: closed the five disclosed post-M6 gaps + full Track X backlog)

Live-verified on Docker Desktop where the gap required it; SSOT.md has the
full evidence trail per item.

  • ILM → ISM retention policies, live-verified: rewrote the four
    retention policies in real OpenSearch 2.13 ISM schema
    (contracts/opensearch-mappings/ism-*.json, replacing the Elasticsearch-
    syntax ilm-policies.json that never worked on this stack), rewired
    infra/provision.sh to real idempotent PUTs. Also found and fixed a
    second, older bug the same live run surfaced: assets/events-bank/
    events-dc/alerts templates each had a top-level _comment field
    OpenSearch's real _index_template PUT rejects outright — silently
    masked forever by curl -sf swallowing the error. Only events-common
    had ever actually installed on a live cluster before this fix.
  • Redis-backed RBAC sessions: services/shared/sessions.py gained
    RedisSessionStore + a make_session_store() factory
    (FENGARDE_SESSION_BACKEND=memory|redis, fails loud rather than
    silently falling back — a session store is a security boundary).
    Live-verified against Docker Desktop's real Redis.
  • Live migrate/CAS verification: tools/migrate_opensearch.py's
    plan/apply cycle now has a live test proving real template PUTs, zero
    drift on a second plan(), and mapping_version round-tripping through
    a real cluster — previously wire-format tested only.
  • Open-core section in README stating the free/paid split explicitly
    (was previously only implicit in SSOT.md).
  • B3 dual-backend test verified live (was already built by prior work,
    never live-confirmed): BUS_BACKEND=redis test_runner.py passed all six
    parametrized bodies against Docker Desktop's real Redis.
  • B4 rule hot-reload (opt-in, RULES_RELOAD_INTERVAL_S, default off):
    Detector.reload() atomically swaps in a freshly parsed rule set,
    fail-closed on a malformed edit.
  • C2 dashboard auto-refresh: polls every 10s when data is live and the
    tab is visible, skips the DOM rebuild when nothing changed (protects an
    in-progress triage-note edit).
  • C3 MITRE ATT&CK/ATT&CK-ICS/ATLAS coverage heatmap: optional, shape-
    validated mitre: {tactic, technique} block on rule YAML (24 of 25
    rules tagged), propagated onto every alert, rendered as a new dashboard
    "Coverage" tab. Surfaced and fixed two real, previously-undetected bugs
    while live-verifying this: rules_view.py/webhooks.py computed their
    contracts/ path with container-incompatible math (GET /rules and
    webhook config loading had returned nothing on every live deployment
    since they shipped), and the dashboard's getAlerts() never mapped
    rule_id through.
  • Four new parser packs (DNS query log, Kubernetes audit, CEF, AWS
    CloudTrail) closing the long-standing class-4002 (DNS/HTTP Activity) gap
    and adding the first Kubernetes and cloud-control-plane producers. Five
    new rules ship with real producers: common_dns_exfil,
    dc_privileged_container, cloud_root_console_login,
    bank_mass_card_read (one additive field on the existing db_audit.py),
    common_rapid_account_lifecycle.
  • Periodicity/beaconing primitive: hit_periodic() on both window
    backends (coefficient of variation of inter-arrival deltas, reusing
    existing window state — no new storage), wired into the rule grammar as
    siem.periodicity: {max_cv}, and common_beaconing.yml — the design
    item flagged "design-first" since the v0.3 plan.
  • S7/PROFINET decision gate re-investigated: found the original
    "proprietary-shaped" deferral reasoning was too broad (S7-1500 ships a
    real, public RFC 5424 syslog security-event feed) but the concrete event
    vocabulary needed to parse it honestly is access-gated, not
    undocumented — still deferred, now for an evidenced reason with a
    concrete unblock path recorded.
  • B5 HA design doc: Redis Sentinel + OpenSearch multi-node
    recommendation, decision only, no code — closes the last open Track X
    item.

Added (M3 remainder: dashboard session login + CSRF)

Closes two items the M3 milestone had left genuinely open (verified by grep before
starting, not assumed from the plan doc): the RBAC session API (/auth/login,
/auth/logout, /auth/me, M4.2) was real and tested at the HTTP level, but nothing
in services/ws7-dashboard/ actually called it, and CSRF protection didn't exist.

  • Dashboard login UI (services/ws7-dashboard/index.html) — a login form gates
    the app behind a real session when FENGARDE_RBAC_DB is set; a user badge
    (username + role + Sign out) replaces it once authenticated. Wired to a new nginx
    proxy path (/api/auth/ → ws3-indexer's /auth/*, services/ws7-dashboard/ templates/default.conf.template). RBAC off (the default, every existing
    deployment) is byte-for-byte unaffected
    : GET /auth/me 404s (no such route), the
    login gate is skipped entirely, and the app renders exactly as before — same
    "opt-in, zero behavior change" convention as every other auth layer in this
    project. Found and fixed two real bugs while browser-testing this (Playwright/
    Chromium, not just the static contract test): a CSS-specificity trap where
    #loginScreen's own ID-selector rule silently outranked the hidden attribute
    (toggling .hidden in JS did nothing), and clearing an inline style.display with
    "" fell back to a stylesheet rule that was still none instead of becoming
    visible — both fixed by always setting an explicit display value, documented
    inline where a future edit could easily reintroduce either trap.
  • CSRF protection (services/ws3-indexer/triage_api.py::_check_csrf,
    services/shared/sessions.py) — a second, independent layer on top of the session
    cookie's existing SameSite=Strict: login now also mints a csrf_token (returned
    in the login/. /auth/me response body, never a cookie), and every state-changing
    (POST) request made with an active session must echo it back as X-CSRF-Token or
    get a 403 — enforced centrally in do_POST, a true no-op when RBAC is off or the
    request carries no session cookie at all (pure API-key callers are unaffected).
    Verified end-to-end in a real browser: login → write → reload persists; a wrong
    token 403s; logout invalidates the session and re-locks the app.

Added (M5 — NIS2 public template layer)

  • NIS2/§32 BSIG incident-report generator (services/ws3-indexer/nis2_template.py,
    contracts/nis2-de-schema.json) — deterministic German/English draft generator for
    the Art. 23 / §32 BSIG notification, additive on the existing
    POST/GET /alerts/{id}/report route via ?template=nis2. Zero LLM, zero paid
    dependency — every draft states its own NIS2-vs-DORA scope caveat inline
    (financial entities are typically DORA-governed, not NIS2) and carries the same
    mandatory status: "draft" + disclaimer discipline as the generic report backend.
    The paid, legally-validated layer stays fengarde-sec's, via the unmodified
    REPORT_BACKEND=http seam. docs/nis2-report-generator.md has the full scope/limits.
    Dashboard gained a "Rapport (NIS2)" option alongside the existing generic report
    button. Live-verified: a real bank-DB privilege-escalation alert through to a real
    German NIS2 draft, zero infra, zero manual steps (see make nis2-demo).

Added (M4 — MSP-grade)

  • Multi-tenancy: tenant-scoped OpenSearch indices, per-tenant rule enablement
    (contracts/tenants/<id>.yml), siem.tenant (Envelope v1) as the isolation key
    end-to-end. tools/test_multi_tenant_isolation.py.
  • RBAC: opt-in (FENGARDE_RBAC_DB) — SQLite user store with scrypt password
    hashing, session cookies, role enforcement (services/shared/{users,sessions,rbac}.py).
    Off by default; every pre-existing deployment is byte-for-byte unaffected.
  • Versioned REST API: contracts/triage-api.yaml (OpenAPI 3.1) formalizes
    /api/v1/... alongside the unchanged bare paths — spec-vs-code drift is
    CI-tested (test_api_v1.py).
  • Outbound HMAC-signed webhooks: contracts/webhooks/,
    services/ws3-indexer/webhooks.py — per-tenant/per-score-threshold alert delivery
    with signature verification, retry policy, and its own consumer group (cg-webhook)
    so a slow/down receiver never delays indexing.
  • Entry-points parser/rule plugin interface: docs/plugin-development.md — an
    external pip package can register additional parsers/rule packs via
    fengarde.parsers/fengarde.rule_packs entry points, never overriding a built-in.
  • Ops lifecycle: services/shared/users.py schema migration via
    PRAGMA user_version, tools/backup.py/restore.py, tools/migrate_opensearch.py
    (versioned, diffable index-template migration), services/shared/diskguard.py
    (free-space guardrails for the spool/backup paths).
  • Disclosed while building this, not hidden: OpenSearch ILM/retention policies
    were never actually installable on a live cluster — contracts/opensearch-mappings/ ilm-policies.json is written in Elasticsearch ILM syntax, but this stack runs
    OpenSearch's ISM plugin, a different schema at a different endpoint. Everything
    else in M4 is real and tested; this pre-existing issue surfaced during the
    versioned-index-mappings work and was tracked, not silently worked around (later
    actually fixed — see the ISM rewrite entry earlier in this file).

Fixed (adversarial repo-wide bug hunt, post-M4/M5)

A repo-wide (not PR-only) reviewer/bug-hunter pass over the M4/M5 surface, each finding
adversarially verified against the real code path, each fix shipping a regression test
independently confirmed (via a revert/run/restore cycle on the fix's own diff) to fail
without the fix and pass with it restored. Full findings + severity ranking + discarded
false positives are in the review that produced this list; six real bugs were fixed:

  • F1 (HIGH) — WS-4's stateful window rule counter (engine.py) keyed sliding-window
    state only on f"{rule_id}:{group}", with no tenant component. Two tenants sharing a
    group_by value (e.g. overlapping RFC1918 IPs — the normal case for an MSP) had their
    event counts pooled in one shared window, letting one tenant's traffic trip another
    tenant's threshold and misattribute the resulting alert — a direct breach of the M4.1
    tenant-isolation guarantee. Fixed by namespacing the counter key on siem.tenant.
    Follow-up caught by a dedicated review of this fix: Rule.alert_key() — which
    computes the actual alert_id persisted to storage, separately from the counter key —
    was still unnamespaced, so two tenants firing in the same window bucket on a shared
    group_by value got the identical alert_id and WS-3's cross-index find_alert()
    lookup could return the wrong tenant's doc. Fixed the same way, with the same
    revert/run/restore-verified regression test discipline.
  • F6 (MEDIUM/HIGH)active_directory.py assigned raw, un-typechecked fields
    (IpAddress/MacAddress/TargetUserSid) straight into OCSF schema-constrained fields,
    unlike every sibling parser, which already goes through shared/ocsf.py's
    valid_ip/valid_mac/safe_str guards. A malformed upstream field silently
    dead-lettered the whole event instead of just dropping the bad field, which can blind
    common_bruteforce/common_password_spray/common_lateral_movement on real AD
    authentication events.
  • F2 (MEDIUM)GET /alerts/{id}/report only applied the tenant gate when the
    backing alert doc was still present. Once the alert aged out (reports have independent
    retention), the gate was skipped and any authenticated caller could read another
    tenant's incident report. Now fails closed (404) for non-admins when the alert doc is
    absent.
  • F5 (LOW/MEDIUM)LoginRateLimiter (rbac.py) grew its per-username dict without
    bound and had no lock despite being mutated from multiple ThreadingHTTPServer handler
    threads — a memory-DoS risk on /auth/login plus dropped failure records under
    concurrency. Added a lock around all three methods plus a periodic sweep, mirroring the
    pattern window.py already uses for its own counters.
  • F4 (MEDIUM)tools/restore.py extracted the archive before verifying checksums,
    and its traversal guard only checked each member's name — not enough to stop a
    symlink member written "through" by a later member (the CVE-2007-4559 class). Switched
    to tarfile.extractall(filter="data") (PEP 706), which rejects symlinks, absolute
    paths, and .. traversal before anything is written.
  • F3 (MEDIUM)tenant_id flowed unvalidated into an OpenSearch index name
    (router.py) and a contracts/tenants/<id>.yml path (tenants.py). An uppercase or
    space-containing tenant_id produced an OpenSearch-invalid index name that silently
    dead-lettered every event for that tenant; a path-traversal-shaped tenant_id could
    construct a config path outside contracts/tenants/. Added
    shared/envelope.py::valid_tenant_id() (DNS-label-style allowlist): router.py now
    rejects (never normalizes — normalizing "Acme"/"ACME" to the same slug would silently
    merge two tenants' data) an invalid tenant with a ValueError on both the alert and
    event branches; tenants.py fails open (no rules disabled, same as a missing config
    file) rather than ever constructing the unsafe path.

New/extended tests: services/ws3-indexer/test_router.py, services/ws4-detection/test_tenants.py,
services/ws2-normalization/parsers/test_active_directory.py (new files), plus extensions to
tools/test_multi_tenant_isolation.py, services/ws3-indexer/test_reporting.py,
services/shared/test_rbac.py, tools/test_backup_restore.py.

Added (v0.5 M1 — correctness gates)

  • Envelope v1: schema_version, trace_id, tenant_id (formalizes siem.tenant, declared since Phase 0 but never wired), documented event_time/ingest_time/dedup-key semantics. Additive bus-schema change to contracts/bus-topics.md + contracts/ocsf-event.schema.json, owner-authorized. services/shared/envelope.py; wired through all 10 parsers via base_event(meta=...) and all 4 live WS-1 collectors.
  • make chaos (tools/chaos_test.py): kills each of ws1-ws5 mid-replay across 40 independent brute-force scenarios, asserts zero lost/duplicate alerts. Live-verified 2026-07-18 on a fresh Docker stack: scenarios=40 lost=0 duplicated=0, all 5 pipeline services SIGKILLed mid-replay — a genuine passing gate, not just reviewed (took 4 real harness-bug fixes across 4 runs to get an honest verdict; see SSOT.md §2 for the full history).
  • docs/degradation-matrix.md: every dependency's down-behavior, sourced from the actual fail-open/fail-closed code paths.
  • Hypothesis property tests (parsers/test_property_hardening.py): 100 generated examples per parser, all 10 pass. Found and fixed a real bug: 6 structured-record parsers (db_audit, mcp_agent, n8n_audit, opcua_audit, vmware_vsphere, windows_eventlog) assigned unguarded JSON-field values into schema-constrained ip/mac/hostname/name fields; services/shared/ocsf.py gains valid_ip/valid_mac/safe_str to fix all six.
  • Log-injection defense (services/shared/sanitize.py): strips ANSI escapes (blocks terminal/OSC-52 injection when viewing raw event content) and C0/DEL control chars (blocks newline-based log forging), wired into normalize_one() as a new sanitize stage between parse and enrich.
  • atheris fuzz harnesses for the top 3 parsers by regex complexity (linux_ssh, cisco_asa, windows_eventlog) + nightly CI job (.github/workflows/fuzz.yml). Locally spot-verified (millions of executions, zero crashes); full nightly budget runs only once merged to main.

Added (v0.5 M2 — public proof artifacts)

  • tools/fengarde_bench.py: one-command load generator, published real numbers in README (~13,750 EPS / ~84 MB peak RSS, zero-infra baseline — explicitly not a live-stack throughput claim).
  • Code quality floor: pyproject.toml (ruff/black/mypy/coverage config), .pre-commit-config.yaml. ruff clean and CI-blocking; black configured but not force-applied (98/100 files would reformat against this codebase's established style — a deliberate, documented, separate decision); mypy informational-only (20 real findings, honest baseline, not a strict gate on a largely-unannotated codebase); coverage gate blocking at measured baseline (WS-2 90%, WS-3 71% — the latter below the ~85% target, documented as an open gap, not silently lowered).
  • ADR backfill (docs/adr/001-006): Redis Streams, OCSF, OpenSearch, microservice split, fail-closed rules, local-first LLM triage.
  • Supply chain: found and fixed every service's requirements.txt being decorative prose, never actually installed from (each Dockerfile hardcoded its own unpinned inline pip install). Rewrote all 6 as real pinned manifests, switched Dockerfiles to install from them, dropped two never-imported "extras" (pysnmp, scikit-learn). .github/dependabot.yml, tools/generate_sbom.py (CycloneDX, CI-blocking freshness check), .github/workflows/codeql.yml, .github/workflows/scorecard.yml, README badges.

Added (v0.5 M3 — product completeness)

  • Agent rule pack complete: R4 (agent_egress_non_allowlisted_domain.yml) and R5 (agent_destructive_command.yml) join R1-R3, reusing the engine's existing not_in/Allowlist mechanism (R4) and single-shot pattern-match (R5, not a burst threshold). All 5 rules proven firing on real mcp_agent parser output, including R1+R3 together on one session log (services/ws4-detection/test_v05_agent_rules.py).
  • tools/agent_log_shipper.py: the missing real ingestion path for MCP/agent JSONL logs (file, --follow, or stdin) into raw.events — found while writing the doc below that this didn't exist yet. Proven end-to-end (tools/test_agent_log_shipper.py): a JSONL file with one malformed line ships, normalizes, fires R1+R3, and both alerts reach the index.
  • docs/agent-monitoring.md, docs/deployment.md (reverse-proxy TLS via Caddy, documented not built), docs/vs.md (honest FENGARDE vs Wazuh/Elastic Security/Security Onion comparison), new-rule issue template.

Changed — product rebrand to FENGARDE

Second rename in two months (see the ARGUS rename below). Adopted while merging
the M1-M6 roadmap branch into main; branch-local env vars/identifiers
(FENGARDE_API_KEY, FENGARDE_RBAC_DB) already used the new name before the merge,
main did not — this merge is what makes the rename repo-wide. No functional change.

Fixed / Added (deep-hardening pass — P0+P1+P2, 2026-07-16)

Full audit-and-fix pass across parser logic, the detection engine, and pipeline
robustness, ahead of public launch. Details and evidence in SSOT.md §1.

  • P0 (detection-integrity, 9e2745b): window-poisoning (far-future/non-numeric
    time) now fails closed instead of collapsing sliding-window counts or crashing
    the daemon; memory-vs-Redis window counters now agree on redelivery dedup; idle
    window keys are evicted (bounded memory); parser registry routing rewritten
    (source_type-authoritative — fixes a bank DB privileged op being silently
    mis-parsed as a vSphere read, meaning bank_db_priv_esc could never fire);
    vmware port-crash guard + any raising parser now dead-letters one record instead
    of aborting the batch; out-of-range IP octets are dropped, not dead-lettered;
    status_from_outcome() shared helper (vmware/db/n8n no longer hardcode
    "Success", masking failed logins from the brute-force rules); alert_id no
    longer collapses distinct no-ingest-id events onto one alert.
  • P1 (robustness, c84c8f6/625f4f8/7e354bb): /health probes the bus and
    returns 503 when degraded; compose healthchecks on ws1-5; tools/dlq_peek.py
    DLQ inspector/requeue tool; OpenSearch index() retries transient errors with
    backoff, surfaces 4xx immediately; HTTP servers get read timeouts (slowloris
    guard); dashboard (8080) + inventory (8000) bind to 127.0.0.1 by default
    (syslog 5514 stays open, must receive remote logs); shared to_epoch_ms()
    replaces 9 copy-pasted, FILETIME/ISO-mishandling time heuristics; rule tuning
    (service-account allowlist on after-hours-admin, documented brute-force/
    impossible-travel tradeoffs).
  • P2 (enhancements, b645b13..2f807ec): in/contains grammar operators;
    cross-source severity rubric (SEV_BY_CATEGORY) so the same action (e.g.
    delete) gets the same severity regardless of source; siem.sector override now
    validated + honored consistently across all 10 parsers; /metrics endpoint
    (per-topic acked/failed/deadlettered + ws1 ingest-edge counters); backpressure
    depth-watchdog on internal topics; IPv6 capture in ssh/ASA parsers + full 0-7
    ASA severity map; opt-in live Redis/OpenSearch test lane (make test-live);
    anti-dormancy CI gate now checks per-event satisfiability, not just "producible
    somewhere across all fixtures" (catches a rule that's dormant in practice
    because its group_by field is only ever produced by a different class).

Changed — ARGUS rebrand (717f4ed, 2026-07-15)

Renamed the project after a trademark/domain collision check on the prior name.
239 occurrences across 42 files, GitHub repo renamed (old clone URLs redirect),
ARGUS_API_KEYFENGARDE_API_KEY. No functional change.