feat!: multi-source observability and guardrails (v2.0.0 part 2, #34)#37
Merged
fxthiry merged 2 commits intorelease/2.0.0from Apr 16, 2026
Merged
Conversation
added 2 commits
April 16, 2026 14:36
Adds the `vl_source` Prometheus label to every per-rule counter, replaces
the legacy `valerter_victorialogs_up{rule_name}` gauge with a per-source
`valerter_vl_source_up{vl_source}`, introduces a `defaults.max_streams`
cap with load-time enforcement, applies +/-10% uniform jitter on reconnect
backoff per (rule, source) task to break thundering-herd alignment, and
ships a `/metrics` snapshot integration test that catches accidental
relabel or rename in future PRs.
Multi-source operators can now attribute every alert and error rate to a
specific source. The new gauge makes partial-source outages visible
without requiring per-rule reachability inference. The cap prevents an
accidental (rule x source) fan-out from DoS'ing a backend at startup.
Post-review hardening folded in: dated v1.2.1 in CHANGELOG, added
PromQL migration snippet for dashboards, extended snapshot coverage
to include lines_discarded / query_duration / last_query_timestamp /
per-notifier sentinel counters, made the jitter floor test exercise the
actual clamp branch, demoted backoff_delay_default to pub(crate),
extended ReconnectCallback::on_reconnect to receive vl_source, and
rejected max_streams=0 at load with a clear error.
Cargo.toml not bumped here; version bump lives in the release PR.
…dy_html Documents a pre-existing concern made wider by the v1.2.0 #26 fix: the example config now actively pipes `{{ _msg }}` into `body`, and operators who mirror this pattern in `email_body_html` may inadvertently render unescaped HTML/script content from untrusted log fields. Hardening of the email path is a follow-up; the advisory makes the limit visible at the v2.0.0 release boundary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes #34 — multi-source VictoriaLogs observability + guardrails. Builds on PR #36 (multi-source core). Last PR before the v2.0.0 release tag.
What ships
vl_sourcelabel. Multi-source operators can attribute every alert, error, throttle hit, and reconnect to a specific source.valerter_queue_size(shared, not per-source) keeps unlabeled.valerter_victorialogs_up{rule_name}removed and replaced byvalerter_vl_source_up{vl_source}. Per-source reachability gauge: 1 on connect success, 0 on permanent failure or stream error. Initialized to 0 for every configured source at startup.defaults.max_streams: 50(configurable). Total streams = sum of(rule, source)pairs spawned for enabled rules. Breach fails the config at load with both the actual count and the cap value.max_streams: 0is explicitly rejected.(rule, source)task. Sources behind a flapping load balancer no longer reconnect in lock-step.tests/metrics_snapshot.rsintegration test that scrapes/metricsafter a 2-source 1-rule warmup and asserts the set of metric names + label keys against an inline expected list. Catches accidental relabel/rename in future PRs.Validation
cargo test— 540+ tests pass (479 lib + integration suite, including newmetrics_snapshot)cargo clippy --all-targets --all-features -- -D warnings— cleancargo fmt --check— cleancargo run --bin valerter -- --validate -c config/config.example.yaml— exit 0max_streams: 2override (4 streams required > cap of 2). Withmax_streams: 50:/metricsscrape showsvalerter_vl_source_up{vl_source}for both sources at 1, all 8 per-rule counters carry bothrule_nameandvl_sourcelabels, andvalerter_victorialogs_upseries is absent.Breaking changes summary (full text in CHANGELOG v2.0.0)
rule_namenow also carriesvl_source(additive).valerter_victorialogs_up{rule_name}removed; replaced byvalerter_vl_source_up{vl_source}(per-source semantics, not per-rule). PromQL migration examples in CHANGELOG.defaults.max_streamscap introduced (default 50). Configs whose enabled-rule fan-out exceeds the cap fail at load.Out of scope (deferred)
valerter_active_streamsgauge (mentioned in earlier scoping; not needed for v2.0.0 acceptance, follow-up if requested).