Centralized in-cluster Headlamp installation for "managing" clusters in different hyperscalers

[pgmgb]

We have dev, qual and prod Kubernetes environments with multiple clusters in different hyperscalers (AKS & GKE).
We would like to offer one centralized in-cluster Headlamp installation in each environment for our developers.
Our goal would be that the developers are able to log in with their accounts and therefore their in-Kubernetes permissions are used in Headlamp. I was able to find documentation on how to configure OIDC for AKS and GKE, but it seems that this works only if the clusters belong to the same hyperscaler.

Is there any experience and documentation about a centralized in-cluster Headlamp installation for clusters in different hyperscalers?

[illume]

This has not been documented, but I recall others wanting to do this.

It might be worth joining the headlamp channel on the kubernetes slack and asking on there.

I think you should be able to use something like oauth2proxy in front to authenticate them via your idp. Can you sketch out the architecture a bit more?

[pgmgb]

Hi @illume ,
thank you for the quick response.
I tried to create an overview of how we currently access the Kubernetes clusters in Azure and GCP with the related information. My question would be if it's possible to offer one centralized in-cluster Headlamp for our developers so they could decide to use the command line or a web UI for accessing our Kubernetes infrastructure.

---

update:

I tried to understand the part of Use a non-default kube config file because we would like to offer a list of clusters to the developers. It would have been nice to be able to declare Kubernetes clusters as secrets like ArgoCD is doing it. In the end I created a ConfigMap with the kubeconfig as content and mounted it  to /home/headlamp/.config/Headlamp/kubeconfigs/config.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: headlamp-kubeconfig
  namespace: headlamp
  labels:
    app.kubernetes.io/component: configuration
    app.kubernetes.io/instance: headlamp
    app.kubernetes.io/managed-by: Helmfile
    app.kubernetes.io/name: headlamp
    app.kubernetes.io/part-of: headlamp
data:
  kubeconfig: |
    apiVersion: v1
    kind: Config
    clusters:
      - cluster:
          certificate-authority-data: LS0XXX
          server: https://aks-01-dev:443
        name: aks-01-dev
      - cluster:
          certificate-authority-data: LS0XXX
          server: https://gke-01-dev:443
        name: gke-01-dev
    contexts:
      - name: gke-01-dev
        context:
          cluster: gke-01-dev
          namespace: default # Dummy namespace
          user: default
      - name: aks-01-dev
        context:
          cluster: aks-01-dev
          namespace: default # Dummy namespace
          user: default
    users:
      - name: default
        user:
          name: default # Dummy user
          # password: XXX # Do not provide a password - Headlamp will ask for token

This will now show the two clusters.

But to be honest, I did not expect to also have to declare the contexts and users. Every user should, in the end, use their own credentials. So it does not make sense for me to declare an user inside the kubeconfig.

Do I understand/interpret something wrong?

[illume]

It would have been nice to be able to declare Kubernetes clusters as secrets like ArgoCD is doing it.

I think it’s good if we can borrow patterns like how others are doing things in a good ways. Perhaps this way is something to consider in the future.

The diagram and further descriptions does help understand what you want to achieve. Thanks.

[illume]

Would you mind bringing this to the headlamp channel on the kubernetes slack? I think it will get some more knowledgeable folks to have a look.