Inabilility to create token for remote cluster

[jpraychev]

When configuring RemoteClusterAccess to collect metrics from remote clusters using the remoteClusterConfig field of the CRD I am not able to generate token request and 401 HTTP error is being returned to the operator.

Looking at the code, the k8s client used to create the token request need to somehow authenticate to the remote cluster which I am not able to see how is done.

func queryConfigFromClusterAccessConfig(ctx context.Context, cac *v1alpha1.ClusterAccessConfig, inClient client.Client, externalScheme *runtime.Scheme) (*orchestrator.QueryConfig, error) {
	clsData, errData := getCusterDataFromSecret(ctx, cac, inClient)
	if errData != nil {
		return nil, errData
	}

	saName := cac.ServiceAccountName
	saNamespace := cac.ServiceAccountNamespace

	token, errToken := getTokenWithAPI(ctx, inClient, saName, saNamespace, clsData.audience)
	if errToken != nil {
		return nil, errToken
	}
....

The k8s client from the above snippet seems to be authenticated to the local cluster where the metrics operator resides, thus is unable to generate a temporary token to the remote cluster.

[maximiliantech]

@moelsayed will work on this task.

[moelsayed]

I validated this feature using a simple scenario with two kind clusters: host-cluster and remote-cluster.

• The host-cluster hosts the metrics-operator.
• The remote-cluster is the target cluster where metrics would are collected from.

Instead of deploying and OIDC provider, I configured the API server of the host-cluster to act as a provider, and configured the remote-cluster to use it as an OIDC provider. This enables cross-authentication between the two clusters.

The detailed steps to create the setup are here.

On the remote-cluster, we need to create the required RBAC configuration to allow the remote use to get the metric values.

Next, I created the following resources on the host-cluster:

• A secret containing the remote cluster details: caData, host and audience:

kubectl  create secret -n default generic remote-cluster-secret \
   --from-file=caData=<REMOTE_CLUSTER_CA.crt>  \
   --from-literal=host="<REMOTE_CLUSTER_API_ENDPOINT>" \
   --from-literal=audience="mcp-operator"

• Then, we create the RemoteClusterAccess and metric resources. Note that all resources referenced here (secret, namespaces and service-account) are on the host-cluster:

apiVersion: metrics.openmcp.cloud/v1alpha1
kind: RemoteClusterAccess
metadata:
  name: remote-cluster
  namespace: metrics-operator-system
spec:
  remoteClusterConfig:
    clusterSecretRef:
      name: remote-cluster-secret
      namespace: default
    serviceAccountName: mcp-operator
    serviceAccountNamespace: mcp-operator-system
---
apiVersion: metrics.openmcp.cloud/v1alpha1
kind: Metric
metadata:
  name: remote-pods
spec:
  name: remote-pods-metric-total
  description: Pods
  target:
    kind: pod
    group: ""
    version: v1
  interval: 1m # in minutes
  remoteClusterAccessRef:
    name: remote-cluster
    namespace: metrics-operator-system

The metric then goes into ready state and reports metrics to the default datasink:

$ k get metric remote-pods -o yaml 
apiVersion: metrics.openmcp.cloud/v1alpha1
kind: Metric
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"metrics.openmcp.cloud/v1alpha1","kind":"Metric","metadata":{"annotations":{},"name":"remote-pods","namespace":"metrics-operator-system"},"spec":{"description":"Pods","interval":"1m","name":"remote-pods-metric-total","remoteClusterAccessRef":{"name":"remote-cluster","namespace":"metrics-operator-system"},"target":{"group":"","kind":"pod","version":"v1"}}}
  creationTimestamp: "2025-07-22T11:02:30Z"
  generation: 1
  name: remote-pods
  namespace: metrics-operator-system
  resourceVersion: "51332"
  uid: 60c3034e-b2ae-4587-bd8e-22c39456cf0a
spec:
  description: Pods
  interval: 1m
  name: remote-pods-metric-total
  remoteClusterAccessRef:
    name: remote-cluster
    namespace: metrics-operator-system
  target:
    group: ""
    kind: pod
    version: v1
status:
  conditions:
  - lastTransitionTime: "2025-07-22T11:04:07Z"
    message: Metric reconciled successfully
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-07-22T11:04:07Z"
    message: metric value recorded for resource '/v1, Kind=pod'
    reason: MonitoringActive
    status: "True"
    type: Available
  observation:
    latestValue: "9"
    timestamp: "2025-07-22T12:54:07Z"
  ready: "True"