cmd/bosun: adding "unknownIsNormal" flag to alerts to convert unknown events into normal ones. #1620

Merged
merged 1 commit into from Feb 24, 2016

Projects

None yet

2 participants

@captncraig
Contributor

Simply replaces the event status in run history. Should help with things like #1160 where sparse data is expected.

It will end up creating a normal event for all alert keys every 2 check cycles once they disappear. Not sure how to prevent that. Maybe it would be better to suppress unknown generation at all for these alerts, but then they are unclosable, so never-mind.

@kylebrandt kylebrandt commented on an outdated diff Feb 24, 2016
cmd/bosun/conf/conf.go
@@ -253,6 +253,7 @@ type Alert struct {
Unknown time.Duration
MaxLogFrequency time.Duration
IgnoreUnknown bool
+ UnkownsNormal bool
@kylebrandt kylebrandt commented on the diff Feb 24, 2016
cmd/bosun/sched/sched_test.go
@@ -338,3 +338,21 @@ func TestRename(t *testing.T) {
},
})
}
+
+func TestUnknownsAreNormal(t *testing.T) {
+ defer setup()()
+ testSched(t, &schedTest{
+ conf: `alert a {
+ unknownIsNormal = true
@kylebrandt kylebrandt and 1 other commented on an outdated diff Feb 24, 2016
docs/configuration.md
@@ -249,6 +249,7 @@ An alert is an evaluated expression which can trigger actions like emailing or l
* critNotification: comma-separated list of notifications to trigger on critical. This line may appear multiple times and duplicate notifications, which will be merged so only one of each notification is triggered. Lookup tables may be used when `lookup("table", "key")` is an entire `critNotification` value. See example below.
* depends: expression that this alert depends on. If the expression is non-zero, this alert is unevaluated. Unevaluated alerts do not change state or become unknown.
* ignoreUnknown: if present, will prevent alert from becoming unknown
+* unknownIsNormal: will convert unkown events into normal events. Useful in cases like elastic search where no data can be a good thing.
@kylebrandt
kylebrandt Feb 24, 2016 Member

For example, if you are alerting for the existence of error log messages, when there are none that means things are normal.

@kylebrandt
kylebrandt Feb 24, 2016 Member

Oh, also, does this get used with ignoreUnknown?

@captncraig
captncraig Feb 24, 2016 Contributor

no, ignoreUnknown would never do anything, because there would be no unknowns.

@kylebrandt
kylebrandt Feb 24, 2016 Member

Add that to the docs? Bet it will be a common question. Or, we can throw an error on the config if both exist.

@captncraig captncraig cmd/bosun: adding "unknownIsNormal" flag to alerts to convert unknown…
… events into normal ones.
c5a5316
@kylebrandt
Member

LGTM

@captncraig captncraig merged commit c5a5316 into master Feb 24, 2016

2 checks passed

bosun All checks Passed!
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@captncraig captncraig deleted the unkownToNormal branch Feb 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment