Support self signed certs for alert notifications #9548

laughland · 2017-10-16T19:23:05Z

What Grafana version are you using? grafana/grafana:master
What datasource are you using? Prometheus
What OS are you running grafana on? CentOS 7 on virtualBox
What did you do? Tried to add a notification alert channel via the grafana UI
What was the expected result? A notification would be sent
What happened instead?

grafana_1        | t=2017-10-16T18:06:24+0000 lvl=eror msg="Failed to send alert notifications" logger=context userId=1 orgId=1 uname=admin error="Post https://mattermost.XXXXXXXXXX.com/hooks/<hook id>: x509: certificate signed by unknown authority"
grafana_1        | t=2017-10-16T18:06:24+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/alert-notifications/test status=500 remote_addr=X.X.X.X time_ms=26 size=48 referer=http://localhost:3000/alerting/notification/1/edit

To troubleshoot we manually added our certificates with Grafana running in a Docker container:

docker run --rm --entrypoint="/bin/bash" -p 3000:3000 -it grafana/grafana

Our signed certificates were added to /etc/ssl/certs and then we ran c_rehash /etc/ssl/certs

We curled our mattermost instance curl https://mattermost.ourdomain.com successfully:

root@4b3ababf117c:/# curl https://mattermost.ourdomain.com
<!DOCTYPE html> <html> <head> <meta http-equiv=X-UA-Compatible content="IE=edge"> <meta name=viewport content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0"> ....

Is there some way to use our self signed certificates with Grafana to add an alerting channel?

The text was updated successfully, but these errors were encountered:

bergquist · 2017-10-17T06:31:52Z

There are currently no way of using self signed certs for alert notifications.

https://github.com/grafana/grafana/blob/master/pkg/services/notifications/webhook.go#L27

If we decide to add support for it. Alert notifications should not use services/notifications/webhook.go but instead create a new abstractiong in https://github.com/grafana/grafana/tree/master/pkg/services/alerting/notifiers for dealing with http settings.

bergquist · 2017-10-17T06:49:02Z

@laughland Btw. Im not sure your using the correct path for certs on Centos. https://unix.stackexchange.com/questions/363308/how-do-i-list-all-available-ssl-ca-certificates-on-centos-6/363309#363309

laughland · 2017-10-31T20:07:50Z

We were successful in getting notifications working in Grafana with our CA signed cert.

Here is a snippet of our docker-compose.yml:

grafana:
        image: grafana/grafana:latest
        volumes:
          - grafana_data:/var/lib/grafana
          - /home/user/grafana/certs/:/etc/ssl/certs/:z
        expose:
          - 3000
        ports:
          - "3000:3000"
        env_file:
          - ./grafana/config.env

bergquist · 2017-11-01T15:59:59Z

I think you closed this by mistake. re opening.

laughland · 2017-11-01T16:06:01Z

Yes, thanks!

andrewlarioza · 2019-04-04T23:24:54Z

I'm having the same issue with our Grafana Alerts channel using webhook. The webhook url is also using a self-signed ssl certificate. We ran grafana on container based on the official image pulled form the repo. Is there any workaround for this?

valdemarpavesi · 2019-09-23T17:13:39Z

I had same problem using official docker grafana/grafana.

there is no permission to install new certification.

here is the solution:

install cert to HOST.

case ubuntu:

test by curl

# curl https://mattermost.com/hooks/cqcdo196cidnix5hrf17g6kbdh

copy cert file to:

# cd /usr/share/ca-certificates/grafana/

mattermost.crt  

format must be CRT.

copy using chrome:

Steps
1. Open Google Chrome. ...
2. Select Show Advanced Settings > Manage Certificates.
3. Select the certificate you wanted to export then click "Export" button then next.
4. Now, you will get a "Certificate Export Wizard" box. ...
5. Select "No, do not export the private key" then click next.
6. Select "DER encoded binary x.509(.cer) then click next.


format is CER and must be converted to CRT

# openssl x509 -inform DER -in  mattermost.cer -out  mattermost.crt 

on mattermost.crt  check for

"Certificate Authority Information Access"

my case was :  DigiCertBaltimoreCA-2G2.crt

install certificates:

# dpkg-reconfigure ca-certificates

all alias installed to
# ls /etc/ssl/certs


found out by strace that had to install DigiCertBaltimoreCA-2G2.crt: 

# strace  curl https://mattermost.com/hooks/cqcdo196cidnix5hrf17g6kbdh

 stat("/etc/ssl/certs/8ad296c9.0", 0x7ffcdee57550) = -1 ENOENT (No such file or directory)

could not find the alias-hash.


openssl based on cert required ,will build a hash to find  installed cert

looking for 8ad296c9, but,  from mattermost.crt   hash is:  cb318169

# openssl x509 -hash -fingerprint -noout -in /usr/share/ca-certificates/grafana/mattermost.crt
cb318169
SHA1 Fingerprint=04:25:3C:72:0C:95:9D:A2:9C:1E:4F:AA:55:C6:6F:94:B9:8F:25:07

+++

it was missing the root certification.


#  openssl x509 -hash -fingerprint -noout -in /usr/share/ca-certificates/grafana/DigiCertBaltimoreCA-2G2.crt
8ad296c9
SHA1 Fingerprint=A9:D5:30:02:E9:7E:00:E0:43:24:4F:3D:17:0D:6F:4C:41:41:04:FD

and ROOT certification have   hash 8ad296c9.

had two install both: mattermost.crt   + DigiCertBaltimoreCA-2G2.crt



now, cert is working

# curl https://mattermost.com/hooks/cqcdo196cidnix5hrf17g6kbdh
<!DOCTYPE html> <html> <head> <meta http-equiv=X-UA-Compatible content="IE=edge"> <meta name=viewport content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0"> <meta name=robots content="noindex, nofollow"> <meta name=referrer 


now, have to inform docker to use HOST CERT from /etc/ssl/certs

mount host dir /etc/ssl/certs:

# docker run -d --name=grafana --hostname=grafana  -p 3000:3000  -v /etc/ssl/certs:/etc/ssl/certs -e "GF_INSTALL_PLUGINS=grafana-piechart-panel"   grafana/grafana

test inside the docker:

# docker exec -it grafana bash
bash-5.0$   curl https://mattermost.com/hooks/cqcdo196cidnix5hrf17g6kbdh

<!DOCTYPE html> <html> <head> <meta http-equiv=X-UA-Compatible content="IE=edge"> <meta name=viewport content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0"> <meta name=robots content="noindex, nofollow"> <meta name=referrer 

++++
working.

stale · 2019-12-01T23:19:09Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

dmpe · 2020-03-27T16:52:54Z

Facing same problem. Could you please at least add an option to disable TLS /SSL check - I am also using self-signed certs and this is very hard to deal with.

bs444 · 2020-06-23T19:51:28Z

Bump.

switchgl · 2020-07-02T16:08:58Z

Bump, preferably an option within the notification channel options to tinker with cert details.

bapBardas · 2020-12-04T14:15:14Z

Same issue faced here using the docker image grafana/grafana when trying to configure a notification channel via webhook.

bapBardas · 2020-12-07T10:32:42Z

I similarly had the issue where my webhook notification channel was targetting a https exposed REST server using a certificate signed by an untrusted root certificate.

I managed to have this issue fixed by mounting a self written ca-certificates.crt file in the grafana container (which is based on debian FYI).

Based on those inputs :

I mounted a configmap file as a volume in /etc/ssl/certs/ in my kuberntes pod so that the ca-certificates.crt file contains the root certificate used to sign the https certificate.

mdegat01 · 2021-04-01T15:13:14Z

I think I'm seeing this? It seems related but not sure if its a separate issue so figured I'd ask. My alert is non-functional because I see this error every time it runs its query to see if it needs to alert me:

2021/04/01 11:03:31 https://39bd2704-loki:3100/loki/api/v1/query_range?direction=BACKWARD&end=1617289411880222719&interval=1&limit=1000&query=count_over_time%28%28%7Bcontainer_name%3D%22hassio_supervisor%22%7D+%7C~+%22Watchdog+found+a+problem+with%22%29%5B1m%5D%29&start=1617289351880222719&step=1
t=2021-04-01T11:03:31-0400 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=1 name="Addon crashed" error="tsdb.HandleRequest() error Get \"https://39bd2704-loki:3100/loki/api/v1/query_range?direction=BACKWARD&end=1617289411880222719&interval=1&limit=1000&query=count_over_time%28%28%7Bcontainer_name%3D%22hassio_supervisor%22%7D+%7C~+%22Watchdog+found+a+problem+with%22%29%5B1m%5D%29&start=1617289351880222719&step=1\": x509: certificate signed by unknown authority" changing state to=alerting

But this only happens when the query is run by the alert. If I go to explore and run this same query it runs without issue:

[01/Apr/2021:11:09:09 -0400] 200 192.168.1.11, 192.168.1.5, 172.30.32.1(172.30.32.2) GET /api/datasources/proxy/3/loki/api/v1/query_range?direction=BACKWARD&limit=1000&query=%7Bcontainer_name%3D%22hassio_supervisor%22%7D+%7C~+%22Watchdog+found+a+problem+with%22&start=1617286149000000000&end=1617289750000000000&step=1 HTTP/1.1 (Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:86.0) Gecko/20100101 Firefox/86.0)

My source is configured correctly, I can save and test and use explore. It just doesn't seem to work from alerts.

large64 · 2021-12-06T20:22:03Z

@mdegat01 I've just managed to solve this problem for myself. It had nothing to do with my Grafana setup. My source of data is an influxdb Docker container which happened to use the wrong cert file. I'm using certbot to get my certs ready on my host. I've been using cert.pem file as INFLUXDB_HTTP_HTTPS_CERTIFICATE env variable in my container. Then I've read README (/etc/letsencrypt/live/README) which told me that

`[cert name]/privkey.pem`  : the private key for your certificate.
`[cert name]/fullchain.pem`: the certificate file used in most server software.
`[cert name]/chain.pem`    : used for OCSP stapling in Nginx >=1.3.7.
`[cert name]/cert.pem`     : will break many server configurations, and should not be used
                 without reading further documentation (see link below).

So I started using fullchain.pem instead of cert.pem and querying is working like a charm! I'm sooo happy! :)

duchymor · 2022-03-09T08:27:29Z

This worked for me on Alpine based Docker image:

USER root
COPY cacert.crt /usr/share/ca-certificates/custom/
WORKDIR /usr/share/ca-certificates/
RUN ls -1R custom/* >> /etc/ca-certificates.conf;
RUN update-ca-certificates
USER grafana

relishcolouredhat · 2022-12-16T12:53:32Z

Yeah, this being missing is a bit frustrating - we want to direct a webhook towards a service behind our k8s ingress; but get an error that the x509 cert is for the ingress and not the service hostname...

mirkobrankovic · 2023-04-04T13:29:41Z

It would be nice to have Skip TLS Verify option on Alert channels just like on Data sources :)

gregbown · 2023-04-14T19:43:16Z

Had this same issue running Grafana 9.4.7-1 on Rocky Linux release 8.7, trying to send an alert to a. NodeJS alert management tool. Both the Grafana app and the NodeJS app were using the self signed root via a cert trust chain. My development environment is set up to replicate production.

The only documentation I found that had marginal mention of this TLS verification were a few threads in community.

The solution was to update-ca-trust adding my root CA cert to ca-bundle in my Rocky Linux instance

sudo cp rootCAcert.pem /etc/pki/ca-trust/source/anchors/rootCAcert.pem
sudo update-ca-trust extract

It would ease prototyping or development if the TLS verification was configurable

mailinglists35 · 2023-07-10T14:18:26Z

this worked for me for smtp notifications for grafana preinstalled in oracle linux virtualization manager (olvm) engine host

edit /etc/grafana/grafana.ini

change skip_verify from false to true

[smtp]
[...]
skip_verify = true

[root@olvmm ~]# rpm -qf /etc/grafana/grafana.ini
grafana-7.5.15-4.el8.x86_64

came here via https://community.grafana.com/t/grafana-webhook-failed-to-send-alert-notifications-x509-certificate-signed-by-unknown-authority/5701 which is the third google result for "Failed to send notification to email addresses x509: certificate is not valid for any names, but wanted to match XXXXX"

sbengo · 2023-09-21T06:51:44Z

Hi, faced this on the new grafana unified alert feature.

The webhook contact point doesn't allow to configure the skip verify.
Are there any short plans to add this?
Should I create a new issue as it is related with the new unified alerting @armandgrillet ?

Thanks,
Regards

fapeliberty · 2023-11-07T15:09:29Z

Hello,

facing the same thing, we need to be able to skip-verify the webhook tls endpoint,

Thanks

BICH0 · 2023-11-10T21:48:53Z

Hi, faced this on the new grafana unified alert feature.

The webhook contact point doesn't allow to configure the skip verify. Are there any short plans to add this? Should I create a new issue as it is related with the new unified alerting @armandgrillet ?

Thanks, Regards

I totally agree with this but, since this issue is 7 years old, it seems like it wont be happening soon 😢

bergquist changed the title ~~[Bug] Alerting Notification Error: x509: certificate signed by unknown authority~~ [Feature Request] Support self signed certs for alert notifications Oct 17, 2017

bergquist added the area/alerting/notifications Issues when sending alert notifications label Oct 17, 2017

laughland closed this as completed Nov 1, 2017

bergquist reopened this Nov 1, 2017

stale bot added the stale Issue with no recent activity label Dec 1, 2019

stale bot removed the stale Issue with no recent activity label Mar 27, 2020

bergquist added the type/feature-request label Jun 29, 2020

bergquist changed the title ~~[Feature Request] Support self signed certs for alert notifications~~ Support self signed certs for alert notifications Jun 29, 2020

bergquist added the area/backend label Jun 29, 2020

R-Studio mentioned this issue Apr 11, 2023

Grafana external Alertmanager datasource Skip TLS Verify or CA Cert options not working #66238

Open

armandgrillet added the area/alerting Grafana Alerting label Sep 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support self signed certs for alert notifications #9548

Support self signed certs for alert notifications #9548

laughland commented Oct 16, 2017 •

edited

bergquist commented Oct 17, 2017 •

edited

bergquist commented Oct 17, 2017

laughland commented Oct 31, 2017 •

edited

bergquist commented Nov 1, 2017

laughland commented Nov 1, 2017

andrewlarioza commented Apr 4, 2019

valdemarpavesi commented Sep 23, 2019 •

edited

stale bot commented Dec 1, 2019

dmpe commented Mar 27, 2020

bs444 commented Jun 23, 2020

switchgl commented Jul 2, 2020

bapBardas commented Dec 4, 2020

bapBardas commented Dec 7, 2020

mdegat01 commented Apr 1, 2021 •

edited

large64 commented Dec 6, 2021

duchymor commented Mar 9, 2022 •

edited

relishcolouredhat commented Dec 16, 2022

mirkobrankovic commented Apr 4, 2023

gregbown commented Apr 14, 2023 •

edited

mailinglists35 commented Jul 10, 2023 •

edited

sbengo commented Sep 21, 2023

fapeliberty commented Nov 7, 2023

BICH0 commented Nov 10, 2023

Support self signed certs for alert notifications #9548

Support self signed certs for alert notifications #9548

Comments

laughland commented Oct 16, 2017 • edited

bergquist commented Oct 17, 2017 • edited

bergquist commented Oct 17, 2017

laughland commented Oct 31, 2017 • edited

bergquist commented Nov 1, 2017

laughland commented Nov 1, 2017

andrewlarioza commented Apr 4, 2019

valdemarpavesi commented Sep 23, 2019 • edited

stale bot commented Dec 1, 2019

dmpe commented Mar 27, 2020

bs444 commented Jun 23, 2020

switchgl commented Jul 2, 2020

bapBardas commented Dec 4, 2020

bapBardas commented Dec 7, 2020

mdegat01 commented Apr 1, 2021 • edited

large64 commented Dec 6, 2021

duchymor commented Mar 9, 2022 • edited

relishcolouredhat commented Dec 16, 2022

mirkobrankovic commented Apr 4, 2023

gregbown commented Apr 14, 2023 • edited

mailinglists35 commented Jul 10, 2023 • edited

sbengo commented Sep 21, 2023

fapeliberty commented Nov 7, 2023

BICH0 commented Nov 10, 2023

laughland commented Oct 16, 2017 •

edited

bergquist commented Oct 17, 2017 •

edited

laughland commented Oct 31, 2017 •

edited

valdemarpavesi commented Sep 23, 2019 •

edited

mdegat01 commented Apr 1, 2021 •

edited

duchymor commented Mar 9, 2022 •

edited

gregbown commented Apr 14, 2023 •

edited

mailinglists35 commented Jul 10, 2023 •

edited