[release-5.7] LOG-5243: Use safe bearer token auth to scrape operator metrics

[periklis]

Description

In OpenShift clusters we have the option to scrape operator metrics either via cluster-monitoring (default case) or user-workload-monitoring (managed clusters, where users track operator metrics themselves). Until now the service monitor for scraping operator metrics was only compatible with cluster-monitoring that allows using bearerTokenFile and tlsConfig.caFile. Both are not allowed when scraping with user-workload-monitoring. The Prometheus Operator in user-workload-monitoring is configured with ArbitraryFSAccessThroughSMsConfig.Deny: true which in turn disallows the prometheus binary to access it's own serviceaccount token to scrape metrics.

The serviceaccount elasticsearch-operator-metrics-reader is introduced along with a secret that holds a long-lived API tokenand the service CA certificate. The token is referenced in the ServiceMonitor in authorization.credentials replacing bearerTokenFile. The certificate is referenced in the ServiceMonitor in tlsConfig.ca replacing tlsConfig.caFile. Also it is used by Prometheus to scrape metrics from the Elasticsearch Operator manager container only through the kube-rbac-proxy sidecar. This serviceaccount is assigned in a ClusterRoleBinding namely elasticsearch-operator-read-metrics to get access to the Non-Resoure-URL get/metrics. In addition the ClusterRole named metrics-reader is removed from the bundle and re-introduced as elasticsearch-operator-metrics-reader to omit naming collisions.

/cc @xperimental @JoaoBraveCoding

[openshift-ci-robot]

@periklis: This pull request references LOG-5243 which is a valid jira issue.

In response to this:

Description

In OpenShift clusters we have the option to scrape operator metrics either via cluster-monitoring (default case) or user-workload-monitoring (managed clusters, where users track operator metrics themselves). Until now the service monitor for scraping operator metrics was only compatible with cluster-monitoring that allows using bearerTokenFile and tlsConfig.caFile. Both are not allowed when scraping with user-workload-monitoring. The Prometheus Operator in user-workload-monitoring is configured with ArbitraryFSAccessThroughSMsConfig.Deny: true which in turn disallows the prometheus binary to access it's own serviceaccount token to scrape metrics.

The serviceaccount elasticsearch-operator-metrics-reader is introduced along with a secret that holds a long-lived API tokenand the service CA certificate. The token is referenced in the ServiceMonitor in authorization.credentials replacing bearerTokenFile. The certificate is referenced in the ServiceMonitor in tlsConfig.ca replacing tlsConfig.caFile. Also it is used by Prometheus to scrape metrics from the Elasticsearch Operator manager container only through the kube-rbac-proxy sidecar. This serviceaccount is assigned in a ClusterRoleBinding namely elasticsearch-operator-read-metrics to get access to the Non-Resoure-URL get/metrics. In addition the ClusterRole named metrics-reader is removed from the bundle and re-introduced as elasticsearch-operator-metrics-reader to omit naming collisions.

/cc @xperimental @JoaoBraveCoding

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[periklis]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: periklis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [periklis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[periklis]

/test e2e-upgrade

[periklis]

/test e2e-upgrade

[xperimental]

/lgtm

[periklis]

/override ci/prow/e2e-upgrade

[openshift-ci]

@periklis: Overrode contexts on behalf of periklis: ci/prow/e2e-upgrade

In response to this:

/override ci/prow/e2e-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@periklis: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-upgrade	be0a658	link	false	/test e2e-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.