[Bug] Don't connect to STMP server using TLS unless configured.

[bergquist]

It should be possible to send emails to an SMTP server that doesn't require TLS connections.

ref #7116

[vanduijnjordi]

I would like to let you know that we are patiently waiting for this bug to be fixed. Given our infrastructure, we will only be able to send e-mails without TLS authentication. We know you are working hard, so this is just to let you know that it would really help us if this is implemented and that we are hoping that it will make the 4.3.0 milestone! 👍

[fredm1]

Just like @vanduijnjordi we are too waiting for this as we are setup not to use TLS auth. This is preventing us from setting up alerts that send emails.

[torkelo]

we are unable to replicate this with any SMTP server we are testing with.

Tested port 25 (no TLS setup) with
https://github.com/namshi/docker-smtp

[systems1]

I am getting related error for test send email
default name was alias
t=2017-05-10T10:13:05-0400 lvl=eror msg="Failed to send alert notifications" logger=context userId=1 orgId=1 uname=admin error="x509: certificate is valid for Alias.xxx.com , not post.xxxx.com"

updated to server name and got below

t=2017-05-10T10:32:13-0400 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="x509: certificate signed by unknown authority"
t=2017-05-10T10:32:13-0400 lvl=eror msg="Failed to send alert notifications" logger=context userId=1 orgId=1 uname=admin error="x509: certificate signed by unknown authority"
t=2017-05-10T10:32:13-0400 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/alert-notifications/test status=500 remote_addr=10.159.246.136 time_ms=59ns size=48

[daniellee]

@systems1 This sounds like you are using a self signed certificate. If this is being used internally with no risk of a MITM (Man in the middle) attack then try this:

• In your SMTP settings in the ini file, set skip_verify to true
• restart the Grafana Server

[ghost]

I am also experiencing this issue. We are using an internal smtp relay which does not require authentication. We specify the HOST and usually provide some non-existent email address for the FROM_ADDRESS. We have SKIP_VERIFY=true.

The error from Grafana is "TLS not available due to local problem". On the mail relay server, the error is "lost connection after STARTTLS from unknown".

We are using the Grafana 4.2.0 docker image.

[sgerodes]

Yes, we have the same problem on grafana 4.3.1
We get the Error status 500 and the message in grafana "Failed to send alert notification"
We do not use TLS and its very hard to change.
Port 25 is working. But the test still do not work.
switching the skip_verify does not affect something.
Is there any way to do it without TLS? If I get TLS working would it fix the problem?

grafana.log:
t=2017-06-06T17:22:08+0200 lvl=info msg="Sending notification" logger=alerting.notifier type=email id=0 isDefault=false
t=2017-06-06T17:22:08+0200 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses=[sg@loyaltypartner.com]
t=2017-06-06T17:22:08+0200 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error=EOF
t=2017-06-06T17:22:08+0200 lvl=eror msg="Failed to send alert notifications" logger=context userId=1 orgId=3 uname=admin error=EOF
t=2017-06-06T17:22:08+0200 lvl=eror msg="Request Completed" logger=context userId=1 orgId=3 uname=admin method=POST path=/api/alert-notifications/test status=500 remote_addr=[::1] time_ms=5 size=48

SMTP config:
#################################### SMTP / Emailing ##########################
[smtp]
enabled = true
host = 127.0.0.1:25
;user =
;password =
;cert_file =
;key_file =
skip_verify = true
from_address = admin@grafana.localhost
from_name = Grafana

[watmin]

I just chased this down, my server is having the same issue with my one LAN using a mail forwarder to my primary server. TLS appears to be enabled always following this implementation, if I'm reading this correctly you can never use SMTP without TLS:

https://github.com/grafana/grafana/blob/5bf5ed68e17dabb8c3aa957c6a0b6a9e3e117817/vendor/gopkg.in/gomail.v2/smtp.go

So, choosing to enable skip_verify doesn't help us because that still requires the use of TLS cert, but just allows one to use a self signed rather than one from a CA.

I just resolved this on the LAN's postfix forwarder with the following.

Update grafana to send an email via a mail server with a self signed cert:

/etc/grafana/grafana.ini

...
[smtp]
enabled = true
host = mail-forwarder.domain.tld:587
;user =
;password =
;cert_file =
;key_file =
skip_verify = true
from_address = grafana@domain.tld

[emails]
welcome_email_on_sign_up = true
...

Update the postfix install on the mail-forwarder box with a self signed cert:

cd /etc/postfix/
mkdir tls
chmod 0700 tls
cd tls/
openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -keyout server.key -out server.crt
chmod 400 server.*

Next update the postfix configs:

/etc/postfix/master.cf
Uncomment this line:

submission inet n       -       y       -       -       smtpd

/etc/postfix/main.cf
Add these lines:

smtpd_tls_security_level = may
smtpd_tls_key_file = /etc/postfix/tls/server.key
smtpd_tls_cert_file = /etc/postfix/tls/server.crt

Restart grafana-server and postfix and then send yourself a test message.

Cheers.

I really hope this gets fixed, the default settings makes it appear that we are able to use SMTP over tcp/25 without encryption.

[torkelo]

Strange, we tested against port 25 without tls, works fine., issue is I think if the SMTP server cannot handle the test to upgrade the connection

[wookiesh]

Still have the same problem with last version, should be reopened ?

t=2017-09-15T16:59:34+0200 lvl=info msg="Sending notification" logger=alerting.notifier type=email id=0 isDefault=false
t=2017-09-15T16:59:34+0200 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses=[joseph_piron@me.com]
t=2017-09-15T16:59:34+0200 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="tls: server selected unsupported protocol version 300"

Smtp config:

[smtp]
enabled = true
host = relay:25
;user =
;password =
;cert_file =
;key_file =
skip_verify = true
from_address = monitoring
from_name = Monitoring
```

[sergeylanzman]

Bug in gomail fix here go-gomail/gomail#107

[bergquist]

@sergeylanzman thank you for letting us know. Unfortunately I don't think we have time to fix this before 5.0

[bergquist]

seems to be fixed in the new fork https://github.com/go-mail/mail/blob/v2/smtp.go#L87

[alexeyYakovlev1982]

Hello!
Is it actually fixed and tested by someone?

Checked on Grafana 5.1.3 - used rpm on CentOS 6, grafana still sends STARTLS.
Log is:
t=2018-06-01T19:39:28+0200 lvl=eror msg="Async sent email 0 succeed, not send emails: someone@somewhere.com err: x509: cannot validate certificate for 10.1.17.120 because it doesn't contain any IP SANs"

Config is:

[smtp]
enabled = true
host = 10.1.17.120:25
;user =
;password =
;cert_file =
;key_file =
;skip_verify = false
from_address = dont.reply@*****.com
from_name = Grafana

[ArnaudLevaufre]

Hi,

I have the same issue here using the 5.2.1 docker build. Settings set via env variable should result in the following configuration:

[smtp]
enabled = true
host = postfix.prod.svc.cluster.local:25
from_address = grafana@something.com
from_name = Grafana

grafana log the following errors:

t=2018-07-23T14:06:39+0000 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses=[whatever@something.com]
t=2018-07-23T14:06:39+0000 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="454 4.7.0 TLS not available due to local problem"
t=2018-07-23T14:06:39+0000 lvl=eror msg="Failed to send alert notifications" logger=context userId=2 orgId=1 uname=uname@something.com error="454 4.7.0 TLS not available due to local problem"

And postfix log the following errors:

2018-07-23T14:09:55.897250+00:00 postfix-7489b56bf6-qww46 postfix/smtpd[23661]: connect from unknown[10.48.170.15]
2018-07-23T14:09:55.898035+00:00 postfix-7489b56bf6-qww46 postfix/smtpd[23661]: lost connection after STARTTLS from unknown[10.48.170.15]

[gdcrocx]

This configuration worked fine for me!

[smtp]
enabled = true
host = 10.1.1.40:25
;user =
password =
;cert_file =
;key_file =
skip_verify = true
from_address = grafana@grafana.local
from_name = Grafana Dashboard
ehlo_identity = server.grafana.local

Grafana Server version:

grafana-server -v
Version 6.0.2 (commit: 3f4c2e7, branch: HEAD)

[ghost]

Grafana seems to lack these settings for email alerts:
Encryption method: SSL/TLS or STARTTLS
Authentication type: PLAIN or LOGIN.

We need these settings because Grafana cannot send any emails when using MS Exchange mailer.

[ipalopezhentsev]

@gdcrocx thanks for your suggestion, it has helped!

[thameezb]

Hi, still seem to be hitting this bug
Grafana Version v6.5.2

Config (helm chart):

  smtp:
    enabled: true
    host: "10.10.120.18:25"
    skip_verify: true
    from_address: "grafana@grafana.co.za"
    from_name: "Grafana"
    ehlo_identity: "server.grafana.local"

Log Error:
t=2020-03-24T14:02:29+0000 lvl=eror msg="failed to send notification" logger=alerting.notifier uid= error="Failed to send notification to email address: <EMAILADD>: 454 4.7.0 TLS not available due to local problem"

[maurya-m]

same issue here with version grafana-server -v
Version 6.6.0 (commit: 5c11bbd, branch: HEAD)

[maurya-m]

This configuration worked fine for me!

[smtp]
enabled = true
host = 10.1.1.40:25
;user =
password =
;cert_file =
;key_file =
skip_verify = true
from_address = grafana@grafana.local
from_name = Grafana Dashboard
ehlo_identity = server.grafana.local

Grafana Server version:

grafana-server -v
Version 6.0.2 (commit: 3f4c2e7, branch: HEAD)

I tried to downgrade to the version this version 6.0.2 , still the same error:

t=2020-04-01T10:55:46+0000 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses=[sasa.asa@sassss.com]
t=2020-04-01T10:55:46+0000 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="454 4.7.0 TLS not available due to local problem"
t=2020-04-01T10:55:46+0000 lvl=eror msg="failed to send notification" logger=alerting.notifier uid= error="454 4.7.0 TLS not available due to local problem"

[lafrech]

I hope I'm not adding to the noise but I just stumbled upon this so I figured I'd share my issue and solution for anyone in the same situation.

I'm using Exim4 in smarthost mode, outbound only, no local mail. I didn't generate a certificate and I could send a test mail using the mailx command.

However, when sending a mail from Grafana, I got this error in exim logs:

TLS error on connection from localhost (grafana.domain.tld) [::1] (cert/key setup: cert=/etc/exi4/exim.crt key=/etc/exim4/exim.key): Error while reading file.

Grafana accepts a self-signed certificate thanks to the skip_verify config parameter, but it still requires a certificate.

I generated a self-signed certificate using the command found in the Debian docs:

/usr/share/doc/exim4-base/examples/exim-gencert

(Since it is only local, I left all the fields blank.)

And now it works.

So unless I'm misunderstanding something and my exim config is wrong, this should be the way to go.

Edit

I do have another VM with a similar exim config with no cert. I don't even need the skip_verify parameter in Grafana and it works. I can't tell why it works there while I had to do that on this instance. I might not investigate any further since it works but the config I described above may be unnecessary.