Skip to content

v2.0.0

Choose a tag to compare

View all tags
@thomaspatzke thomaspatzke released this 04 Oct 16:30
· 19 commits to main since this release
v2.0.0
50708ba

pySigma 2.0 Introduction

Breaking Changes: SigmaPolicy as Central Security Control Point

pySigma 2.0 introduces SigmaPolicy — a unified security configuration framework that replaces scattered method parameters across the codebase. This is a breaking change but brings significant security and usability improvements.

What Changed:

  • Security-related settings (allow_template_vars, allow_external_sources, vars_allowed_paths, regex_engine) are no longer individual method parameters
  • All security settings now flow through the centralized SigmaPolicy dataclass
  • Pipeline loading methods (ProcessingPipeline.from_yaml(), from_dict()) now accept a policy parameter instead of individual flags
  • YAML and dictionary keys for security settings are explicitly stripped before instantiation — this prevents runtime opt-ins through untrusted pipeline definitions

Why This Matters:
pySigma operates in two fundamentally different trust models:

  • Private CI/CD pipelines: You control both rules and conversions — enable powerful features like template vars and external sources
  • Public web applications: Users upload rules — features that could be exploited are disabled by default

SigmaPolicy declares your trust model once, and those settings propagate through the entire pipeline automatically.

Predefined profiles available:

  • SafePolicy — RE2 engine, all security features disabled
  • TrustedPolicy — Python engine, for fully trusted rule sources

Check out the documentation for further details.

Hardened Defaults: DoS-Safe Regex Engine

Default regex engine is now RE2RegexEngine (backed by Google RE2):

  • ✅ Prevents catastrophic backtracking — no exponential time complexity from regex patterns
  • ✅ Eliminates ReDoS vulnerabilities from untrusted Sigma rules
  • ⚠️ Trade-off: RE2 doesn't support all advanced regex features that Python's re module does

If you need full regex compatibility, explicitly use PythonRegexEngine:

from sigma.policy import SigmaPolicy
from sigma.policy.regex_engine import PythonRegexEngine

policy = SigmaPolicy(regex_engine=PythonRegexEngine())  # Use with trusted rules only

Breaking API Changes

Method signatures have been updated across the codebase:

Before (v1.5.1):

pipeline = ProcessingPipeline.from_yaml(
    yaml_string,
    allow_template_vars=True,
    allow_external_sources=False,
    vars_allowed_paths=["/trusted/dir"]
)

After (v2.0.0):

from sigma.policy import SigmaPolicy

policy = SigmaPolicy(
    allow_template_vars=True,
    allow_external_sources=False,
    vars_allowed_paths=("/trusted/dir",)
)

pipeline = ProcessingPipeline.from_yaml(yaml_string, policy=policy)

Security Hardening Details

  • Top-level pipeline validation is strict: Unknown keys (including security settings) in YAML are rejected rather than silently passed through
  • Nested transformation sanitization: Security-related keys in transformation YAML are explicitly stripped before instantiation
  • Environment variable fallback: Legacy PYSIGMA_ALLOW_VARS_EXECUTION and PYSIGMA_ALLOW_EXTERNAL_SOURCES environment variables are respected as final fallback
  • Template sandboxing: Methods of arbitrary objects in templates are blocked; only safe built-in types (str, int, dict, list, etc.) can have methods called

Migration Guide

If upgrading from v1.5.1:

  1. Audit security settings in your pipeline definitions
  2. Replace method parameters with SigmaPolicy instances
  3. Explicit opt-in required for previously-enabled features (they now default to disabled)
  4. Update YAML pipelines — ensure security settings are passed as policy parameter, not as top-level keys
  5. Test thoroughly — the defaults are more restrictive for security

What's Changed

  • fix: correctly handle escaped backslash before placeholder in expand modifier by @thomaspatzke with @Copilot in #553
  • Order rules topologically when resolving references by @cpoder in #552
  • [high] Stop @pipeline from aliasing every decorated pipeline to one singleton by @elhoim in #559
  • [high] Fix utf16 BOM bytes so utf16|base64/base64offset rules can match by @elhoim in #555
  • [high] Copy filter detections per rule instead of sharing them across rules by @elhoim in #565
  • [high] Keep wildcards after backslashes in replace_string transformation by @elhoim in #566
  • Mark network-dependent plugin tests as online by @x0Lazarus in #571
  • [low] Add regression tests for base64offset with non-ASCII values by @elhoim in #558
  • [high] Keep deferred expressions correct under NOT and in all-deferred AND by @elhoim in #561
  • [medium] Group regex alternation and respect escaped anchors in re|contains/startswith/endswith by @elhoim in #560
  • [medium] Parse not/and/or as keywords so identifiers like notsel are not split by @elhoim in #554
  • Allow fractional correlation condition thresholds by @defensivedepth in #577
  • Load nested query post-processing from YAML by @x0Lazarus in #576
  • Track applied template post-processing steps by @x0Lazarus in #575
  • [medium] Don't silently drop rules with an OR of only deferred expressions by @elhoim in #567
  • [medium] Add missing and fix misnamed Windows service→Channel mappings by @elhoim in #556
  • fix: avoid UnboundLocalError in SigmaCorrelationRule.from_dict with collect_errors=True by @cristianchiriac in #545
  • [low] Escape backslash in SigmaString.to_plain so to_dict/from_dict round-trips by @elhoim in #562
  • [medium] Add SECURITY.md with a private vulnerability reporting policy by @elhoim in #564
  • [medium] Encode escaped wildcard literals correctly in base64/base64offset by @elhoim in #557
  • [medium] Fix over-broad IPv6 cidr wildcard expansion by @elhoim in #563
  • Honor explicit mappings before calling field-name functions by @x0Lazarus in #570
  • Check backend compatibility for YAML pipelines by @x0Lazarus in #573
  • [medium] Raise an error for 1/all of selectors that match no detection instead of dropping them by @elhoim in #574
  • [low] Release skill: always pass release notes via --notes-file by @elhoim in #578
  • [medium] Escape rule names in referenced rules expressions of correlations by @elhoim in #580
  • [low] Validate unmapped placeholder names in query expression placeholders by @elhoim in #579
  • Fix regular expression denial of service vulnerability by @thomaspatzke in #581
  • Implement SigmaPolicy for enhanced security and flexibility in template processing by @thomaspatzke in #582

New Contributors

Full Changelog: v1.5.1...v2.0.0

Contributors

elhoim, pipeline, and 5 other contributors
Assets 2
Loading