feat: detach dynamic source declarations from pipeline files#135
Merged
Conversation
Promote parse_sources/parse_dynamic_source to pub and add parse_sources_file()/parse_sources_dir() for loading standalone source YAML files. Extend validate_source_refs to accept an optional set of external source IDs so pipeline references can resolve against a daemon-level registry. Introduce DaemonSourceRegistry in rsigma-runtime with collision-error semantics: any source ID declared in two different sites is a hard startup error. This is the foundation for the --source CLI flag that decouples source declarations from pipeline files.
Add --source <file_or_dir> (repeatable) to the daemon CLI args. At startup the daemon loads external sources, builds a DaemonSourceRegistry with collision-error semantics, emits a deprecation warning for pipeline-embedded sources:, and resolves both external and pipeline sources through the unified registry. Update /api/v1/sources to list all sources with their declaration origin (external:<path> or pipeline:<name>). Add --source-file to pipeline resolve and --source to rule validate so offline tooling can validate pipelines referencing external sources. Update the LookupEnricher cache-miss error to mention --source as the primary way to declare sources.
…tion Add unit tests for parse_sources_file, parse_sources_dir, validate_source_refs with external IDs, and the DaemonSourceRegistry collision semantics.
New subcommand that extracts pipeline-embedded sources: blocks into standalone source files. Supports --strategy single (default, one consolidated file) or per-pipeline (one file per pipeline). Detects source ID collisions across pipelines, rewrites pipeline files with the sources: block removed, and supports --dry-run for preview. Includes unit tests for the YAML manipulation helpers and integration tests for the single-strategy, per-pipeline, dry-run, and no-sources cases.
Update docs/reference/dynamic-sources.md with a new "External source files" section (recommended path), deprecation notice for pipeline- embedded sources, collision semantics, and the migration tool reference. Update docs/guide/processing-pipelines.md with a deprecation warning for inline source declarations. Update docs/guide/enrichers.md to mention --source as the primary way to configure dynamic sources. Add docs/cli/engine/daemon.md entry for --source. Add new docs/cli/rule/migrate-sources.md documenting the migration subcommand with synopsis, flags, and examples.
This was referenced May 21, 2026
Add --source flag, external source files, migrate-sources subcommand, and DaemonSourceRegistry to CLI, eval, runtime, and root READMEs. Mark pipeline-embedded sources as deprecated.
mostafa
added a commit
that referenced
this pull request
May 21, 2026
The CLI command groups (#124) and alias hiding (#125/#139) now ship in the same upcoming release, so the two-stage deprecation lifecycle the original notes described (visible -> hidden -> removed) collapses into one (hidden -> removed). Updates the affected docs to read coherently with this single-release reality: - CHANGELOG: drop the standalone "Deprecated CLI aliases hidden from --help" subsection added a moment ago. Merge its content into the existing "CLI command groups" section, retitled "(PRs #124, #139)". The merged entry describes the final shipped state (aliases hidden but functional), uses the new warning text, and replaces the three-stage timeline list with a two-stage one (this release: hidden; v1.0: removed via #126). The sample --help output no longer lists deprecated rows and drops the stale `attack` group placeholder. - CHANGELOG: in the Detached dynamic sources entry, drop the misleading "same three-release cycle as the CLI command groups (#125, #126)" reference. #135's own deprecation runs over three releases on its own track; the comparison to CLI groups was true at draft time but is no longer accurate. - crates/rsigma-cli/README.md: the migration timeline previously claimed aliases "were visible in the previous release" - false, since the groups land in the same release that hides them. Rewritten to describe the current state without inventing a prior release. - docs/cli/index.md: the "Migration from flat subcommands (pre-0.11)" section was anchored to a wrong version number, claimed five groups (it's four; `attack` was dropped on main), and presented alias hiding as a future intermediate step under #125. Rewritten as a flat description of the current state with the v1.0 removal pointer retained. - docs/getting-started/concepts.md: one-liner about flat aliases updated to mention they are hidden and forward with a warning, with the v1.0 removal note inline. No source code changes. All 17 cli_deprecation tests still pass.
mostafa
added a commit
that referenced
this pull request
May 21, 2026
…verbatim v0.12.0 shipped on 2026-05-20 with PR #124's flat aliases visible- deprecated, including a CHANGELOG entry for `### CLI command groups (PR #124)`. The previous follow-up commit (ba4dc01) mistakenly rewrote that entry in place to describe the post-#139 state (hidden aliases, "PRs #124, #139", collapsed timeline), modifying tagged release notes that should be a historical record. This commit: - Restores the v0.12.0 `### CLI command groups (PR #124)` subsection byte-for-byte from origin/main. - Keeps the legitimate edit to the [Unreleased] `### Detached dynamic sources (#135)` entry: drop the "same three-release cycle as the CLI command groups (#125, #126)" reference that's no longer accurate. - Adds a new [Unreleased] subsection `### Deprecated CLI aliases hidden from --help (#125)` describing PR #139 as a standalone follow-up to v0.12.0's #124, with the new warning text and pointer to #126 for v1.0 removal. Net diff against origin/main is now scoped entirely to the [Unreleased] section. v0.12.0 and every earlier tagged release are untouched.
mostafa
added a commit
that referenced
this pull request
May 21, 2026
Adds `#[command(hide = true)]` to every deprecated flat top-level variant in `enum Commands` (`Eval`, `Daemon`, `Parse`, `Validate`, `Lint`, `Fields`, `Condition`, `Stdin`, `Convert`, `ListTargets`, `ListFormats`, `Resolve`). The dispatch arms and the forwarding behavior are unchanged, so: - Every alias still runs successfully and still prints the migration warning on stderr. - `rsigma <alias> --help` is still routable and renders the same flag list as the new grouped form, so scripts that introspect a subcommand keep working through the deprecation window. - `rsigma --help` now lists only the four noun-led groups (`engine`, `rule`, `backend`, `pipeline`) plus `help`. The `deprecation_warn` helper text is updated to match the new lifecycle stage: "This alias is hidden from `--help` and will be removed in v1.0." Tests: - `root_help_lists_all_groups_and_deprecated_aliases` rewritten as `root_help_hides_deprecated_aliases`; asserts `[deprecated]` is no longer rendered and each alias token is absent from the Commands block of `rsigma --help` (scoped to the Commands block so substrings like `engine daemon` or `stdin` flag values don't false-positive). - `deprecated_daemon_help_lists_new_path` renamed to `deprecated_daemon_help_still_works`; drops the obsolete root-help assertion but keeps the alias-specific `--help` reachability check. - Per-alias smoke tests (success / warning / stdout parity) are unchanged; all 17 deprecation tests still pass. Docs: - `crates/rsigma-cli/README.md` migration section rewritten. Aliases described as "hidden, undocumented forwarders" with the note that `rsigma <alias> --help` is still routable so existing scripts keep working. Timeline collapsed to hidden -> removed in v1.0. - `docs/cli/index.md` migration section rewritten: drops the wrong pre-0.11 version anchor, the obsolete "five groups" + `attack` placeholder, and the now-completed intermediate hide step (#125). - `docs/getting-started/concepts.md` one-liner updated to mention aliases are hidden, forward with a warning, and are removed at v1.0. - CHANGELOG gets a new `### Deprecated CLI aliases hidden from --help (#125)` entry under `## [Unreleased]`. The v0.12.0 release notes for the original `### CLI command groups (PR #124)` work are untouched (tagged release; historical record). The unrelated `### Detached dynamic sources (#135)` entry drops a stale "same three-release cycle as the CLI command groups (#125, #126)" comparison that's no longer accurate. Verification: - `cargo build -p rsigma --all-features`: clean - `cargo test -p rsigma --features daemon --test cli_deprecation`: 17 passed - `cargo test -p rsigma --features daemon` (full suite): 199 passed, 0 failed - `cargo clippy -p rsigma --all-features --all-targets -- -D warnings`: clean - `cargo fmt --all -- --check`: clean - `cargo build -p rsigma --no-default-features`: unchanged from baseline (10 pre-existing errors in commands/resolve.rs and commands/validate.rs unrelated to this PR) Removal at v1.0 tracked in #126. Closes #125.
5 tasks
mostafa
added a commit
that referenced
this pull request
May 24, 2026
The README ASCII diagram and assets/architecture.mmd predated three unreleased features already documented in CHANGELOG.md: post-evaluation enrichment (#134), server-side TLS for the daemon API listener (#128), and detached dynamic sources (#135). Update both surfaces so they match what the next release ships. README prose and ASCII diagram: - Add an Enrichment bullet to the streaming-runtime section listing the four primitives (template, lookup, http, command), the kind-aware ${detection.*} / ${correlation.*} namespaces, scope filter, HTTP response cache, and the on_error policies. - Insert an enrichment/ block in the rsigma-runtime ASCII diagram between RuntimeEngine and io/, and route the flow through it. - Mention TLS termination (mTLS, cert hot-reload) on the shared API listener in the io/ block. - Mention DaemonSourceRegistry (external --source + pipeline-embedded deprecated, collision-error) in the sources/ block. - Note the enrichments map in the EvaluationResult output box and the Output bullet of the prose. Mermaid diagram (assets/architecture.mmd): - Add 'enrichments' to the RuleHeader line in the OUTPUT node. - Add a new RENRICH node and wire RENG --> RENRICH --> RIO. - Mention TLS termination in the RIO node. - Mention DaemonSourceRegistry, --source, and the collision-error semantics in the RSRC node. Verified with mmdc: assets/architecture.mmd renders cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--source <file_or_dir>(repeatable) onrsigma engine daemonto declare dynamic sources independently of pipeline YAML files. Sources become a first-class daemon-level concept with standalone YAML files containing a top-levelsources:block.DaemonSourceRegistryin rsigma-runtime with collision-error semantics: any source ID declared in two different sites (external files, pipeline files, or both) is a hard startup error with the offending paths in the message.sources:blocks with atracing::warn!at parse time pointing operators at--sourceandrsigma rule migrate-sources.rsigma rule migrate-sourcessubcommand that extracts pipeline-embeddedsources:into standalone files, with--strategy single|per-pipelineand--dry-runsupport./api/v1/sourcesendpoint to list all sources from both scopes, marking each entry with its declaration origin (external:<path>orpipeline:<name>).--source-filetopipeline resolveand--sourcetorule validateso offline tooling can validate pipelines referencing external sources.parse_sources/parse_dynamic_sourcetopuband addparse_sources_file()/parse_sources_dir()entry points. Extendvalidate_source_refsto accept an optional set of external source IDs.Existing pipeline files that declare
sources:continue to work with a deprecation warning.Test plan
parse_sources_file,parse_sources_dir, directory-only YAML loadingvalidate_source_refswith external IDs (both valid and undeclared)DaemonSourceRegistrycollision semantics (within-external, external-vs-pipeline, empty)remove_sources_block,build_sources_yaml,extract_source_yamlrsigma rule migrate-sources(single strategy, per-pipeline, dry-run, no-sources)cargo clippy --workspace --all-targets --features daemon -- -D warningspasses with zero findingscargo fmt --all -- --checkpasses