[Bug] smtp.icoremail.net:465 fail to send mail alert due to 'does not advertise the STARTTLS extension'

[regardfs]

Here is my config:

global:
  resolve_timeout: 15s

route:
  receiver: 'slack-notifications'
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 4h
  group_by: ["alertname", "instance"]
  routes:
    - receiver: 'email'
      group_by: ['alertname', 'cluster', 'instance']

receivers:
  - name: slack-notifications
    slack_configs:
    - send_resolved: true
      api_url: https://hooks.slack.com/services/adadadad123123xxxxxxxx
      channel: '#alert'
      text: '{{ template "slack.text" . }}'
  - name: 'email'
    email_configs:
    - to: '123@it.com'
      send_resolved: true
      require_tls: true
      from: 123@it.com
      smarthost: smtp.icoremail.net:465
      auth_username: 123@it.com
      auth_password: xxxx
      html: '{{ template "email.test.html" . }}'
      headers: { Subject: "[Alert] Prometheus Alert Email" }

templates:
  - '/etc/alertmanager/templates/alertText.tmpl'
  - '/etc/alertmanager/templates/alertEmail.tmpl'

Then start service It could not send alert mail with following error:

time="2017-09-08T08:46:47Z" level=error msg="Error on notify: require_tls: true (default), but "smtp.icoremail.net:465" does not advertise the STARTTLS extension" source="notify.go:283"
time="2017-09-08T08:46:47Z" level=error msg="Notify for 1 alerts failed: require_tls: true (default), but "smtp.icoremail.net:465" does not advertise the STARTTLS extension" source="dispatch.go:262"

So, i modify require_tls to false, so I receive message error like

time="2017-09-08T08:29:52Z" level=error msg="Error on notify: *smtp.plainAuth failed: wrong host name" source="notify.go:283"
time="2017-09-08T08:29:52Z" level=error msg="Notify for 1 alerts failed: *smtp.plainAuth failed: wrong host name" source="dispatch.go:262"

When changed to gmail , no error , no exception happened. so , I wander does it not supporting icoremail ?

[stuartnelson3]

Using port 465 enters a hardcoded path that attempts to use TLS:

https://github.com/prometheus/alertmanager/blob/master/notify/impl.go#L275-L285

For historical reasons, this port is supported with SSL but port 587 should be used instead if possible.

See the post here: http://blog.mailgun.com/25-465-587-what-port-should-i-use/

[regardfs]

@stuartnelson3 , hi, icoremail just provide 465 port with ssl protocol, not TLS!...

[stuartnelson3]

SSL was superceded by TLS back in 1999 (name was changed). If your provider is nearly 20 years behind on protecting connections, it might be time to change.

[regardfs]

@stuartnelson3 but in china most providers still use ssl，WTF

[simonpasquier]

If someone is interested by this issue, he/she may test #1591 (build artifacts are available here).

[chanjarster]

@simonpasquier Thanks, that works. I have tested gmail(port 587) and another mail provider (port 465 with ssl), everything goes fine.

[linux-xiayun]

@chanjarster Hi，我想问一下你这个邮箱是什么，我用阿里的邮箱465不能发送「err="*smtp.plainAuth failed: wrong host name"」

[chanjarster]

@linux-xiayun

smtp_smarthost: 'smtp.qiye.aliyun.com:465'
smtp_hello: 'company.com'
smtp_from: 'username@company.com'
smtp_auth_username: 'username@company.com'
smtp_auth_password: password
smtp_require_tls: false

Replace the username, company and password to your mail account info.

[linux-xiayun]

@chanjarster tks, I have tried but it didn't work. I used tencent email port 587

[mpx]

RFC 8314 requires that SMTP connections to port 465 use TLS (not STARTTLS). STARTTLS on 587 should still be supported but is deprecated.

PLAIN auth was fixed in 352b2ae (released in v0.16.0) which allows forced TLS via port 465 to work. It should be possible to use port 465 now.

require_tls (or the global smtp_require_tls) must be set to false to avoid alertmanager attempting STARTTLS. This configuration interaction with forced TLS (port 465) probably should be cleaned up since it's counter intuitive.

[karthikeayan]

Facing similar issue, AWS SES giving below error,

level=error ts=2019-07-04T13:08:51.42960258Z caller=dispatch.go:264 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="require_tls: true (default), but \"email-smtp.us-east-1.amazonaws.com:465\" does not advertise the STARTTLS extension"

Tried, required_tls: flase, then getting below error,

level=error ts=2019-07-04T13:33:10.9007873Z caller=email.go:147 msg="failed to close SMTP connection" err="write tcp 100.96.2.214:59836->54.235.154.163:587: use of closed network connection"

Working fine with Gmail SMTP.

[simonpasquier]

@karthikeayan

Looking at the SES documentation, I would try either:

• email-smtp.us-east-1.amazonaws.com:587 with required_tls: true for STARTTLS.
• email-smtp.us-east-1.amazonaws.com:465 with required_tls: false for SMTPS.

[karthikeayan]

@simonpasquier I have tried both, facing same issue,

level=error ts=2019-07-05T14:41:15.200309877Z caller=email.go:147 msg="failed to close SMTP connection" err="tls: use of closed connection"

[simonpasquier]

Hmm, email.go:147 is when AlertManager tries to close the connection to the server. The server should have received the email though, no?
I'm not sure what would cause this on SES as when I try with Gmail (both with STARTTLS and SMTPS), it doesn't yield any error message and I receive my emails as expected. Googling for the error message, I came across this old issue.

[simonpasquier]

@karthikeayan I've done some tests with the email notifier recently and a few things can be improved (see #1953). In particular the failed to close connection message can happen when the client failed to authenticate to the SMTP server (eg bad credentials) as the remote server may have closed the connection already.

[karthikeayan]

@simonpasquier thank you, will test it once available in next release

[ilyache]

@karthikeayan - just wanted to share I was experiencing the same thing with AWS SES and it ended up being because of the credentials being provisioned for the wrong region.

[karthikeayan]

@ilyache thank you! I spent another 2 hours to find this, turns out to be AWS access keys will not work here. We need to create SMTP credentials from AWS Console which is similar to AWS access keys.

"global": 
  "resolve_timeout": "5m"
"receivers": 
- "name": "org-prometheus-notify"
  "email_configs":
  - "to": "ksundararajan@organization.com"
    "from": "noreply@organization.com"
    "smarthost": "email-smtp.us-east-1.amazonaws.com:587"
# The catch here is aws access key and secret won't work here, we should create SMTP credentials from AWS Console
    "auth_username": "<SMTP_USERNAME>"
    "auth_password": "<SMTP_PASSWORD>"
    "require_tls": true
"route": 
  "group_by": 
  - "job"
  "group_interval": "5m"
  "group_wait": "30s"
  "receiver": "org-prometheus-notify"
  "repeat_interval": "12h"
  "routes": 
  - "match": 
      "alertname": "Watchdog"
    "receiver": "org-prometheus-notify"

[stringang]

Tencent mail can work normally using port 587

[simonpasquier]

Closing the issue because there's no evidence that it's a problem with Alertmanager. If you struggle to setup the email integration, please use our user mailing list or IRC channel (#prometheus).

[pythonzm]

@linux-xiayun

smtp_smarthost: 'smtp.qiye.aliyun.com:465'
smtp_hello: 'company.com'
smtp_from: 'username@company.com'
smtp_auth_username: 'username@company.com'
smtp_auth_password: password
smtp_require_tls: false

Replace the username, company and password to your mail account info.

this work for me. i use *.qiye.163.com