MON-3229: Remove the dependency on the apiserver auth

[marioferh]

Description

When the control plane nodes are under pressure or the apiserver is just not available, no telemetry data is emitted by the monitoring stack

Solution

Remove the dependency on the apiserver would be to use mTLS communication between telemeter-client and the Prometheus pods.

Add /federate endpoint to rbac proxy and allow telemeter-client to authenticate via mTLS to reach Prometheus metrics.

Add mTLS auth to telemeter-client openshift/telemeter#455

Type of change

• Remove the dependency on the apiserver auth
• kube-rbac-proxy exposing the /federate endpoint
• Add tests
• No user facing changes, so no entry in CHANGELOG was needed.

[simonpasquier]

I think that we also need to update the telemeter client so it can authenticate against the Prometheus server using a client TLS certificate. Right now, the forwarder package (which reads the metrics from the /federate endpoint) only supports bearer token authentication.

https://github.com/openshift/telemeter/blob/ee1ba4699b82ecb2033f886af12b9e2e451d2296/pkg/forwarder/forwarder.go#L149-L182

[marioferh]

Testing TLS changes
openshift/telemeter#455

[marioferh]

depends on openshift/telemeter#457

[marioferh]

/retest-required

[marioferh]

/retest-required

[marioferh]

it seems as everything is working with last patch of openshift/telemeter#455
As example:
with cmd: curl --connect-timeout 5 -v -s --fail --cert /etc/tls/private/tls.crt --key /etc/tls/private/tls.key --cacert ./etc/serving-certs-ca-bundle/service-ca.crt "https://prometheus-k8s.openshift-monitoring.svc:9092/federate?" -G --data-urlencode 'match[]={job="telemeter-client"}'

# TYPE metricsclient_request_send untyped                                              
metricsclient_request_send{client="federate_to",container="kube-rbac-proxy",endpoint="https",instance="10.131.0.29:8443",job="telemeter-client",namespace="openshift-monitoring",pod="telemeter-client-858d8d9c5b-hwxq2",service="telemeter-client",status_code="200",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-1"} 2 1686586
998902          

After this PR is merged I can add some test in telemeter side. Also this tests in tests origin is checking always that cmo and telemeter federate scraping is working right: https://github.com/openshift/origin/blob/master/test/extended/prometheus/prometheus.go#L276-L305

[marioferh]

/retest

[marioferh]

/test e2e-aws-ovn-single-node

[raptorsun]

/retitle MON-3229: Remove the dependency on the apiserver auth

[openshift-ci-robot]

@marioferh: This pull request references MON-3229 which is a valid jira issue.

In response to this:

Description

When the control plane nodes are under pressure or the apiserver is just not available, no telemetry data is emitted by the monitoring stack

Solution

Remove the dependency on the apiserver would be to use mTLS communication between telemeter-client and the Prometheus pods.

Add /federate endpoint to rbac proxy and allow telemeter-client to authenticate via mTLS to reach Prometheus metrics.

Add mTLS auth to telemeter-client openshift/telemeter#455

Type of change

• Remove the dependency on the apiserver auth
• kube-rbac-proxy exposing the /federate endpoint
• Add tests
• No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sthaha]

nit: This seems to repeat at various place. Perhaps we should store this in a variable / config.

[raptorsun]

the other reference to this user is here https://github.com/openshift/cluster-monitoring-operator/blob/master/jsonnet/utils/generate-secret.libsonnet#L18

I think it not so repetitive as all the *rbac-secret.yaml files are generated from that function.

[marioferh]

if its ok for @sthaha we can moving forward with this PR and maybe do a refactor in the future with other variables configs or SA.

[sthaha]

@marioferh , nit can always be ignored 🤗

[marioferh]

/hold
wait for more reviews

[raptorsun]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: marioferh, raptorsun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [marioferh,raptorsun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@marioferh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/versions	a280a8a	link	false	/test versions

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[marioferh]

/unhold