-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Persist silences to a local JSON file. #10
Conversation
Load silences at startup from a local JSON file, write them out every 10 seconds. It's not perfect (writes may possibly be interrupted/inconsistent if the program is terminated while writing), but this is a temporary solution to keep people from going crazy about lost silences until we have a proper storage management. If the JSON file gets corrupted, the alert manager simply starts up without any silences loaded.
if err != nil { | ||
log.Println("Couldn't load silences, starting up with empty silence list:", err) | ||
} | ||
saveSilencesTicker := time.NewTicker(10 * time.Second) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not save per mutation to silences? :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Laziness and a bit less code mainly :) I.e. we'd need to add a hook to each Del/Add/Update method to save afterwards. Also, need to store the filename in the Silencer, not only passed in from main. The current way also decouples writing from mutations, so they don't influence/block them. Worth it?
Generally 👍. |
Persist silences to a local JSON file.
Use ISO 8601 in the Receivers API
Addresses: Scanning your code and 410 packages across 83 dependent modules for known vulnerabilities... === Symbol Results === Vulnerability prometheus#1: GO-2024-2687 HTTP/2 CONTINUATION flood in net/http More info: https://pkg.go.dev/vuln/GO-2024-2687 Module: golang.org/x/net Found in: golang.org/x/net@v0.20.0 Fixed in: golang.org/x/net@v0.23.0 Example traces found: prometheus#1: cli/root.go:122:52: cli.NewAlertmanagerClient calls config.NewClientFromConfig, which eventually calls http2.ConfigureTransports prometheus#2: types/types.go:290:28: types.MultiError.Error calls http2.ConnectionError.Error prometheus#3: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.ErrCode.String prometheus#4: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameHeader.String prometheus#5: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameType.String prometheus#6: types/types.go:290:28: types.MultiError.Error calls http2.GoAwayError.Error prometheus#7: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.Setting.String prometheus#8: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.SettingID.String prometheus#9: types/types.go:290:28: types.MultiError.Error calls http2.StreamError.Error prometheus#10: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.NewClientConn prometheus#11: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.RoundTrip prometheus#12: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.chunkWriter.Write prometheus#13: types/types.go:290:28: types.MultiError.Error calls http2.connError.Error prometheus#14: types/types.go:290:28: types.MultiError.Error calls http2.duplicatePseudoHeaderError.Error prometheus#15: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.gzipReader.Close prometheus#16: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.gzipReader.Read prometheus#17: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldNameError.Error prometheus#18: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldValueError.Error prometheus#19: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.noDialH2RoundTripper.RoundTrip prometheus#20: types/types.go:290:28: types.MultiError.Error calls http2.pseudoHeaderError.Error prometheus#21: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.stickyErrWriter.Write prometheus#22: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.transportResponseBody.Close prometheus#23: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.transportResponseBody.Read prometheus#24: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.writeData.String Your code is affected by 1 vulnerability from 1 module. This scan also found 0 vulnerabilities in packages you import and 2 vulnerabilities in modules you require, but your code doesn't appear to call these vulnerabilities. Use '-show verbose' for more details. Signed-off-by: Holger Hans Peter Freyther <holger@freyther.de>
Addresses: Scanning your code and 410 packages across 83 dependent modules for known vulnerabilities... === Symbol Results === Vulnerability prometheus#1: GO-2024-2687 HTTP/2 CONTINUATION flood in net/http More info: https://pkg.go.dev/vuln/GO-2024-2687 Module: golang.org/x/net Found in: golang.org/x/net@v0.20.0 Fixed in: golang.org/x/net@v0.23.0 Example traces found: prometheus#1: cli/root.go:122:52: cli.NewAlertmanagerClient calls config.NewClientFromConfig, which eventually calls http2.ConfigureTransports prometheus#2: types/types.go:290:28: types.MultiError.Error calls http2.ConnectionError.Error prometheus#3: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.ErrCode.String prometheus#4: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameHeader.String prometheus#5: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameType.String prometheus#6: types/types.go:290:28: types.MultiError.Error calls http2.GoAwayError.Error prometheus#7: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.Setting.String prometheus#8: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.SettingID.String prometheus#9: types/types.go:290:28: types.MultiError.Error calls http2.StreamError.Error prometheus#10: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.NewClientConn prometheus#11: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.RoundTrip prometheus#12: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.chunkWriter.Write prometheus#13: types/types.go:290:28: types.MultiError.Error calls http2.connError.Error prometheus#14: types/types.go:290:28: types.MultiError.Error calls http2.duplicatePseudoHeaderError.Error prometheus#15: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.gzipReader.Close prometheus#16: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.gzipReader.Read prometheus#17: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldNameError.Error prometheus#18: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldValueError.Error prometheus#19: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.noDialH2RoundTripper.RoundTrip prometheus#20: types/types.go:290:28: types.MultiError.Error calls http2.pseudoHeaderError.Error prometheus#21: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.stickyErrWriter.Write prometheus#22: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.transportResponseBody.Close prometheus#23: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.transportResponseBody.Read prometheus#24: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.writeData.String Your code is affected by 1 vulnerability from 1 module. This scan also found 0 vulnerabilities in packages you import and 2 vulnerabilities in modules you require, but your code doesn't appear to call these vulnerabilities. Use '-show verbose' for more details. Signed-off-by: Holger Hans Peter Freyther <holger@freyther.de>
Addresses: Scanning your code and 410 packages across 83 dependent modules for known vulnerabilities... === Symbol Results === Vulnerability prometheus#1: GO-2024-2687 HTTP/2 CONTINUATION flood in net/http More info: https://pkg.go.dev/vuln/GO-2024-2687 Module: golang.org/x/net Found in: golang.org/x/net@v0.20.0 Fixed in: golang.org/x/net@v0.23.0 Example traces found: prometheus#1: cli/root.go:122:52: cli.NewAlertmanagerClient calls config.NewClientFromConfig, which eventually calls http2.ConfigureTransports prometheus#2: types/types.go:290:28: types.MultiError.Error calls http2.ConnectionError.Error prometheus#3: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.ErrCode.String prometheus#4: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameHeader.String prometheus#5: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameType.String prometheus#6: types/types.go:290:28: types.MultiError.Error calls http2.GoAwayError.Error prometheus#7: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.Setting.String prometheus#8: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.SettingID.String prometheus#9: types/types.go:290:28: types.MultiError.Error calls http2.StreamError.Error prometheus#10: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.NewClientConn prometheus#11: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.RoundTrip prometheus#12: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.chunkWriter.Write prometheus#13: types/types.go:290:28: types.MultiError.Error calls http2.connError.Error prometheus#14: types/types.go:290:28: types.MultiError.Error calls http2.duplicatePseudoHeaderError.Error prometheus#15: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.gzipReader.Close prometheus#16: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.gzipReader.Read prometheus#17: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldNameError.Error prometheus#18: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldValueError.Error prometheus#19: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.noDialH2RoundTripper.RoundTrip prometheus#20: types/types.go:290:28: types.MultiError.Error calls http2.pseudoHeaderError.Error prometheus#21: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.stickyErrWriter.Write prometheus#22: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.transportResponseBody.Close prometheus#23: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.transportResponseBody.Read prometheus#24: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.writeData.String Your code is affected by 1 vulnerability from 1 module. This scan also found 0 vulnerabilities in packages you import and 2 vulnerabilities in modules you require, but your code doesn't appear to call these vulnerabilities. Use '-show verbose' for more details. Signed-off-by: Holger Hans Peter Freyther <holger@freyther.de>
Addresses: Scanning your code and 410 packages across 83 dependent modules for known vulnerabilities... === Symbol Results === Vulnerability prometheus#1: GO-2024-2687 HTTP/2 CONTINUATION flood in net/http More info: https://pkg.go.dev/vuln/GO-2024-2687 Module: golang.org/x/net Found in: golang.org/x/net@v0.20.0 Fixed in: golang.org/x/net@v0.23.0 Example traces found: prometheus#1: cli/root.go:122:52: cli.NewAlertmanagerClient calls config.NewClientFromConfig, which eventually calls http2.ConfigureTransports prometheus#2: types/types.go:290:28: types.MultiError.Error calls http2.ConnectionError.Error prometheus#3: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.ErrCode.String prometheus#4: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameHeader.String prometheus#5: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameType.String prometheus#6: types/types.go:290:28: types.MultiError.Error calls http2.GoAwayError.Error prometheus#7: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.Setting.String prometheus#8: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.SettingID.String prometheus#9: types/types.go:290:28: types.MultiError.Error calls http2.StreamError.Error prometheus#10: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.NewClientConn prometheus#11: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.RoundTrip prometheus#12: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.chunkWriter.Write prometheus#13: types/types.go:290:28: types.MultiError.Error calls http2.connError.Error prometheus#14: types/types.go:290:28: types.MultiError.Error calls http2.duplicatePseudoHeaderError.Error prometheus#15: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.gzipReader.Close prometheus#16: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.gzipReader.Read prometheus#17: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldNameError.Error prometheus#18: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldValueError.Error prometheus#19: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.noDialH2RoundTripper.RoundTrip prometheus#20: types/types.go:290:28: types.MultiError.Error calls http2.pseudoHeaderError.Error prometheus#21: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.stickyErrWriter.Write prometheus#22: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.transportResponseBody.Close prometheus#23: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.transportResponseBody.Read prometheus#24: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.writeData.String Your code is affected by 1 vulnerability from 1 module. This scan also found 0 vulnerabilities in packages you import and 2 vulnerabilities in modules you require, but your code doesn't appear to call these vulnerabilities. Use '-show verbose' for more details. Signed-off-by: Holger Hans Peter Freyther <holger@freyther.de>
Addresses: Scanning your code and 410 packages across 83 dependent modules for known vulnerabilities... === Symbol Results === Vulnerability #1: GO-2024-2687 HTTP/2 CONTINUATION flood in net/http More info: https://pkg.go.dev/vuln/GO-2024-2687 Module: golang.org/x/net Found in: golang.org/x/net@v0.20.0 Fixed in: golang.org/x/net@v0.23.0 Example traces found: #1: cli/root.go:122:52: cli.NewAlertmanagerClient calls config.NewClientFromConfig, which eventually calls http2.ConfigureTransports #2: types/types.go:290:28: types.MultiError.Error calls http2.ConnectionError.Error #3: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.ErrCode.String #4: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameHeader.String #5: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.FrameType.String #6: types/types.go:290:28: types.MultiError.Error calls http2.GoAwayError.Error #7: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.Setting.String #8: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.SettingID.String #9: types/types.go:290:28: types.MultiError.Error calls http2.StreamError.Error #10: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.NewClientConn #11: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.Transport.RoundTrip #12: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.chunkWriter.Write #13: types/types.go:290:28: types.MultiError.Error calls http2.connError.Error #14: types/types.go:290:28: types.MultiError.Error calls http2.duplicatePseudoHeaderError.Error #15: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.gzipReader.Close #16: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.gzipReader.Read #17: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldNameError.Error #18: types/types.go:290:28: types.MultiError.Error calls http2.headerFieldValueError.Error #19: api/v2/client/silence/silence_client.go:196:35: silence.Client.PostSilences calls client.Runtime.Submit, which eventually calls http2.noDialH2RoundTripper.RoundTrip #20: types/types.go:290:28: types.MultiError.Error calls http2.pseudoHeaderError.Error #21: notify/email/email.go:253:14: email.Email.Notify calls fmt.Fprintf, which eventually calls http2.stickyErrWriter.Write #22: test/cli/acceptance.go:362:3: cli.Alertmanager.Start calls http2.transportResponseBody.Close #23: test/cli/acceptance.go:366:22: cli.Alertmanager.Start calls io.ReadAll, which calls http2.transportResponseBody.Read #24: notify/notify.go:998:21: notify.TimeActiveStage.Exec calls log.jsonLogger.Log, which eventually calls http2.writeData.String Your code is affected by 1 vulnerability from 1 module. This scan also found 0 vulnerabilities in packages you import and 2 vulnerabilities in modules you require, but your code doesn't appear to call these vulnerabilities. Use '-show verbose' for more details. Signed-off-by: Holger Hans Peter Freyther <holger@freyther.de>
Signed-off-by: Marco <llan@redhat.com>
Load silences at startup from a local JSON file, write them out every 10
seconds.
It's not perfect (writes may possibly be interrupted/inconsistent if the
program is terminated while writing), but this is a temporary solution
to keep people from going crazy about lost silences until we have a
proper storage management. If the JSON file gets corrupted, the alert
manager simply starts up without any silences loaded.