Eclipse Che with Azure Kubernetes Service OIDC

[maheshrajrp]

Summary

Hi All,

Currently I am trying to upgrade to latest eclipse-che, but blocked as Azure Kubernetes Service doesn't support External OIDC [1] and Azure Entra doesn't fit our use-case.
What are the other reliable approaches that can be used for production cases ?
One approach I was able to gather is to use vCluster (Thanks @monaka), do we have any production setup steps related to that other than [2] ?

[1] Azure/AKS#2861
[2] https://vrelevant.net/vcluster-with-oidc/

Thanks

Relevant information

No response

[monaka]

These configs are a digest.

As I'm working to release my Che distribution for commercial, my whole configurations are huge and too complex to share here.
But I think these configs will give some hints.

# values.yaml for vcluster-k8s.
api:
  image: registry.k8s.io/kube-apiserver:v1.27.1
  extraArgs:
    - --oidc-issuer-url=https://keycloak.c.pizzafactory.jp/auth/realms/che
    - --oidc-client-id=che-private
    - --oidc-username-claim=sub
    - --oidc-username-prefix=-
    - --oidc-groups-claim=groups

sync:
  ingresses:
    enabled: true

kind: CheCluster
apiVersion: org.eclipse.che/v2
spec:
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_USERNAME__CLAIM: sub

  networking:
    auth:
      oAuthClientName: "che-private"
      oAuthSecret: "{{hidden}}"
      identityProviderURL: "https://keycloak.c.pizzafactory.jp/auth/realms/che"
      externalIdentityProvider: true
      openShiftoAuth: false
      gateway:
        oAuthProxy:
          cookieExpireSeconds: 30 # see also eclipse/che#22765

Note that "Che on vcluster-oidc" approach can be used in almost all k8s clusters. Not only in AKS.

[maheshrajrp]

Hi @monaka ,
Thanks for responding. I think externalIdentityProvider: true, is not relevant anymore I'm getting error: checlusters.org.eclipse.che "eclipse-che" is invalid error.

[maheshrajrp]

These configs are a digest.

As I'm working to release my Che distribution for commercial, my whole configurations are huge and too complex to share here. But I think these configs will give some hints.

# values.yaml for vcluster-k8s.
api:
  image: registry.k8s.io/kube-apiserver:v1.27.1
  extraArgs:
    - --oidc-issuer-url=https://keycloak.c.pizzafactory.jp/auth/realms/che
    - --oidc-client-id=che-private
    - --oidc-username-claim=sub
    - --oidc-username-prefix=-
    - --oidc-groups-claim=groups

sync:
  ingresses:
    enabled: true

kind: CheCluster
apiVersion: org.eclipse.che/v2
spec:
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_USERNAME__CLAIM: sub

  networking:
    auth:
      oAuthClientName: "che-private"
      oAuthSecret: "{{hidden}}"
      identityProviderURL: "https://keycloak.c.pizzafactory.jp/auth/realms/che"
      externalIdentityProvider: true
      openShiftoAuth: false
      gateway:
        oAuthProxy:
          cookieExpireSeconds: 30 # see also eclipse/che#22765

Note that "Che on vcluster-oidc" approach can be used in almost all k8s clusters. Not only in AKS.

Also, I have one more doubt, you seem to be using same client-id for both CheCluster and vCluster, I am guessing you are using keycloak. So, in that sense, one should be private client (i.e. with client_id and client_secret for CheCluster) and vCluster is public client (i.e. with just client_id, username and password). I believe I am running into this issue, I am getting 401in Eclipse Che when I try to start workspace, not sure why. It would be great if you have some idea on that.

[maheshrajrp]

Here's my values.yaml of vCluster

api:
  extraArgs:
    - --oidc-issuer-url=https://private.keycloak.com/realm/sample
    - --oidc-client-id=client-kubernetes
    - --oidc-username-claim=email
    - --oidc-groups-claim=groups
init:
  manifestsTemplate: |-
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: oidc-cluster-admin
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: Group
      name: {{ .Values.ClusterAdminGroup }}
service:
  type: LoadBalancer
  loadBalancerAnnotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

CheCluster Patch YAML

kind: CheCluster
apiVersion: org.eclipse.che/v2
spec:
  networking:
    auth:
      oAuthClientName: some-other-kubernetes
      oAuthSecret: dsadsad
      identityProviderURL: "https://private.keycloak.com/realm/sample"
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_USERNAME__CLAIM: email

[monaka]

vCluster is public client

Ah, I should share more simplified configs.
Sorry for sharing limited resources.

Actually, in this case, vCluster is also using a private client.

In that sample, I used kubelogin for auth in kubectl.
So clent_secret is a client side.
(see: https://github.com/int128/kubelogin?tab=readme-ov-file#getting-started )

Try https://che-tavola.camino.pizzafactory.jp/ if you have free time.
This is evidence that my config works well. It runs on vcluster-k8s-oidc on AKS.

I am getting 401

As you might know, similar issues are shown.
My suggestion is to use "same realms", "same client", "same secret" between "OIDC configured kube-API" and "Che".

[akurinnoy]

@tolusha any ideas?

[maheshrajrp]

@monaka Thanks for help. I was able to get it working. Idk, what happened, when I redeployed the second time with same configurations it worked, seems second times a charm !!

Thanks for the help 😃

[maheshrajrp]

@akurinnoy Thanks for the comment.

Between, if there is any need to publish this, let me know I could pick this up (as my first contribution).

[maheshrajrp]

Closing this as I was able to make it work with vCluster.

[tolusha]

@maheshrajrp

Between, if there is any need to publish this, let me know I could pick this up (as my first contribution).

That would be really nice if you make a contribution into che-docs [1]. I would really appreciate.

[1] https://github.com/eclipse-che/che-docs

[maheshrajrp]

@tolusha , thanks for confirming would love that. Will be picking that up.