Skip to content

Conversation

@juzhao
Copy link
Contributor

@juzhao juzhao commented Jan 8, 2026

AlertmanagerClusterFailedToSendAlerts critical alert is defined in
https://github.com/prometheus/alertmanager/blob/release-0.29/doc/alertmanager-mixin/alerts.libsonnet#L61-L78
configured wrong smtp port for email_configs

receivers:
  - name: 'email.hook'
    email_configs:
    - to: **@**.com
      from: **@**.com
      smarthost: 'smtp.**.com:259' => wrong smtp port
      require_tls: false

saw the erorrs in alertmanager logs

time=2026-01-06T06:40:36.220Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=1 err="email.hook/email[0]: notify retry canceled after 3 attempts: establish connection to server: dial tcp [**:**:**:**]:259: connect: connection timed out"

expected the AlertmanagerClusterFailedToSendAlerts to be fired, but not, checked locally, the reason label has several possible values and some of them report 0 so min(...) is always 0, exclude the value 0 can trigger the alert
Image
same for the AlertmanagerClusterFailedToSendAlerts warning alert

dependabot bot and others added 2 commits January 14, 2026 16:57
…4888)

Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.30.0 to 0.32.0.
- [Commits](golang/mod@v0.30.0...v0.32.0)

---
updated-dependencies:
- dependency-name: golang.org/x/mod
  dependency-version: 0.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Junqi Zhao <juzhao@redhat.com>
Signed-off-by: juzhao <juzhao@redhat.com>
Signed-off-by: Junqi Zhao <juzhao@redhat.com>
Signed-off-by: juzhao <juzhao@redhat.com>
@juzhao juzhao force-pushed the update_AlertmanagerClusterFailedToSendAlerts branch from 58066ee to 43a89f4 Compare January 14, 2026 08:58
@juzhao
Copy link
Contributor Author

juzhao commented Jan 20, 2026

needs approval from a maintainer in: https://github.com/prometheus/alertmanager/blob/main/MAINTAINERS.md to run the workflows

@juzhao
Copy link
Contributor Author

juzhao commented Jan 21, 2026

@simonpasquier can you help to review and approve it?

@jan--f
Copy link
Contributor

jan--f commented Jan 22, 2026

@juzhao can you please rebase you PR?

@juzhao
Copy link
Contributor Author

juzhao commented Jan 23, 2026

@juzhao can you please rebase you PR?

done

Copy link
Contributor

@jan--f jan--f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough description.

@jan--f jan--f merged commit 17c791d into prometheus:main Jan 26, 2026
8 checks passed
@juzhao juzhao deleted the update_AlertmanagerClusterFailedToSendAlerts branch January 26, 2026 13:51
@SoloJacobs SoloJacobs mentioned this pull request Jan 29, 2026
SuperQ added a commit that referenced this pull request Feb 2, 2026
* [ENHANCEMENT] docs(opsgenie): Fix description of `api_url` field. #4908
* [ENHANCEMENT] docs(slack): Document missing app configs. #4871
* [ENHANCEMENT] docs: Fix `max-silence-size-bytes`. #4805
* [ENHANCEMENT] docs: Update expr for `AlertmanagerClusterFailedToSendAlerts` to exclude value 0. #4872
* [ENHANCEMENT] docs: Use matchers for inhibit rules examples. #4131
* [ENHANCEMENT] docs: add notification integrations. #4901
* [ENHANCEMENT] docs: update `slack_config` attachments documentation links. #4802
* [ENHANCEMENT] docs: update description of filter query params in openapi doc. #4810
* [ENHANCEMENT] provider: Reduce lock contention. #4809
* [FEATURE] slack: Add support for top-level text field in slack notification. #4867
* [FEATURE] smtp: Add support for authsecret from file. #3087
* [FEATURE] smtp: Customize the ssl/tls port support (#4757). #4818
* [FEATURE] smtp: Enhance email notifier configuration validation. #4826
* [FEATURE] telegram: Add `chat_id_file` configuration parameter. #4909
* [FEATURE] telegram: Support global bot token. #4823
* [FEATURE] webhook: Support templating in url fields. #4798
* [FEATURE] wechat: Add config directive to pass api secret via file. #4734
* [FEATURE] provider: Implement per alert limits. #4819
* [BUGFIX] Allow empty `group_by` to override parent route. #4825
* [BUGFIX] Set `spellcheck=false` attribute on silence filter input. #4811
* [BUGFIX] jira: Fix for handling api v3 with ADF. #4756
* [BUGFIX] jira: Prevent hostname corruption in cloud api url replacement. #4892
---------

Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Ben Kochie <superq@gmail.com>
Co-authored-by: Ben Kochie <superq@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants