Custom CA not recognized by pelorus pods (grafana and prometheus)

[kevin-anderson-redhat]

Attempts to login to prometheus and grafana fail with Error Page 500 Internal Error
Logs indicate that
oauth fails with "certificate signed by unknown authority"

[etsauer]

This is something we'll have to fix in our helm chart, but for now the proper workaround is to:

1. Create an empty ConfigMap in the pelorus namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    config.openshift.io/inject-trusted-cabundle: true
  name: cluster-ca-bundle
data:

2. Add a volume and volumeMount to the prometheus/pelorus-prometheus resource in the pelorus namespace:

  ...
  containers:
    name: prometheus-proxy
    ...
    volumeMounts:
    ...
    - mountPath: /etc/pki/ca-trust/extracted/pem/
      name: prometheus-trusted-ca-bundle
      readOnly: true
  ...
  volumes:
  ...
  - configMap:
      items:
      - key: ca-bundle.crt
        path: tls-ca-bundle.pem
      name: prometheus-trusted-ca-bundle-39man1pbaa8jq
      optional: true
    name: prometheus-trusted-ca-bundle

References: https://docs.openshift.com/container-platform/4.4/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

[etsauer]

This should fix the prometheus component, but since the thanos-sidecar container is added by the operator, we need to figure out how to control that

[kevin-anderson-redhat]

Does the issue at the following link mean that the grafana-operator is not easily resolved? grafana/grafana-operator#190

[garethahealy]

@robbbbh ; did you have to do this for our prev customer?

• Custom CA not recognized by pelorus pods (grafana and prometheus) #91 (comment)

[redhat-cop-ci-bot]

@kevin-anderson-redhat no, I think there are ways to make it happen with grafana, its just the oauth container that needs it, and that config gets pretty customizable.

If you can try the above and give some feedback, we can try to help troubleshoot.

[kevin-anderson-redhat]

@etsauer We were not able to get grafana to operate. Still getting an untrusted cert failure in the grafana-proxy container.