[Bug] Unable to download grafana.com dashboards - x509: certificate signed by unknown authority

[dan-j]

Describe the bug
Unable to use GrafanaDashboard with remote HTTPS URLs. This is the error from the logs:

2021-11-02T22:33:37.061Z    ERROR    dashboard-node-exporter-full    failed to request dashboard from grafana.com, falling back to config map; if specified    {"error": "failed to request dashboard url 'https://grafana.com/api/dashboards/
1860/revisions/23/download': Get \"https://grafana.com/api/dashboards/1860/revisions/23/download\": x509: certificate signed by unknown authority"}

The base docker image for the operator is registry.access.redhat.com/ubi8/ubi-micro:8.4, this image doesn't include the trusted CA bundle so all HTTPS requests which should be trusted aren't.

Discussed on slack, but creating this issue to make it more easily discoverable.

Version
v4.0.1

To Reproduce

Create a GrafanaDashboard like so:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: node-exporter-full
spec:
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "Prometheus"
  grafanaCom:
    id: 1860
    revision: 23
  json: ""

Expected behavior

The dashboard from grafana.com should be downloaded and no errors appear in the operator logs.

Suspect component/Location where the bug might be occuring
Docker image doesn't contain trusted CA bundles.

Runtime (please complete the following information):

• OS: Ubuntu and CoreOS
• Grafana Operator Version: 4.0.1
• Environment: Kubernetes v1.21.3-gke.2001
• Deployment type: GKE public cluster

[g-chcht]

Hello,

We're having the same issue

[dan-j]

So I've managed to get it to work by running the ubi-minimal image locally and copying the ca-bundle.crt as so:

docker run -itd --rm --name minimal registry.access.redhat.com/ubi8/ubi-minimal:8.4
docker cp minimal:/etc/pki/tls/certs/ca-bundle.crt .

Create a ConfigMap with ca-bundle.crt as a file entry.

Add a volume mount to the operator deployment something like so:

          volumeMounts:
            - mountPath: /etc/pki/tls/certs/ca-bundle.crt
              subPath: ca-bundle.crt
              name: ca-bundle
              readOnly: true
      volumes:
        - name: ca-bundle
          configMap:
            name: ca-bundle

[dan-j]

What would the maintainers prefer a PR do? Use ubi-minimal as the base image in the Dockerfile? Or use another stage in the Dockerfile to copy from ubi-minimal?

[NissesSenap]

First of all thanks for reporting the issue and taking such a deep look at it.
I would personally love to be able to keep on using ubi-micro, mainly to minimise the potential attack service as much as possible and lower startup times. Not that the operator starts up often but any way :)

So copying the certs from ubi-minimal sounds like a good idea to me. Sure it will increase the build time a bit but build only happens once.

[rdgacarvalho]

Any EAT regarding this?

[NissesSenap]

About now :D
The next question is when/how we will cut a new tag for it. I will work to get this done quickly.