OIDC configuration not working on fresh install or update/upgrade

[Zuzuske]

What happened?

OIDC configuration is not applied to /etc/kubernetes/manifests/kube-apiserver.yaml on fresh install or second run/upgrade.

What did you expect to happen?

this:

    - --oidc-issuer-url=[REDACTED]
    - --oidc-client-id=kubernetes
    - --oidc-username-claim=email
    - '--oidc-username-prefix=oidc:'
    - --oidc-groups-claim=groups
    - '--oidc-groups-prefix=oidc:'

appear in /etc/kubernetes/manifests/kube-apiserver.yaml

when ~/kubespray/[env]/group_vars/k8s_cluster/k8s-cluster.yml is:

kube_oidc_url: "[REDACTED]"
kube_oidc_client_id: kubernetes
## Optional settings for OIDC
# kube_oidc_ca_file: "{{ kube_cert_dir }}/ca.pem"
kube_oidc_username_claim: email
kube_oidc_username_prefix: 'oidc:'
kube_oidc_groups_claim: groups
kube_oidc_groups_prefix: 'oidc:'

How can we reproduce it (as minimally and precisely as possible)?

install cluster with OIDC settings

OS

HOST: Darwin 24.3.0 x86_64

TARGET:

root@master1:~# printf "$(uname -srm)\n$(cat /etc/os-release)\n"
Linux 6.8.0-52-generic aarch64
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Version of Ansible

ansible [core 2.16.10]

Version of Python

Python 3.12.2

Version of Kubespray (commit)

release 2.27.0

Network plugin used

calico

Full inventory with variables

[all]
master1 ansible_host=live_master_1 etcd_member_name=etcd1
master2 ansible_host=live_master_2 etcd_member_name=etcd2
worker1 ansible_host=live_worker_1 etcd_member_name=etcd3

[kube_control_plane]
master1
master2

[kube_node]
master1
master2
worker1

[etcd]
master1
master2
worker1

[calico_rr]

[k8s_cluster:children]
kube_control_plane
kube_node
calico_rr

Command used to invoke ansible

ansible-playbook -i inventory/live/hosts.yaml --user root --become --become-user=root cluster.yml

Output of ansible run

sorry, no, only in private conversations.

Anything else we need to know

Its just a regular install on a fresh system. OIDC setting are not appearing in /etc/kubernetes/manifests/kube-apiserver.yaml

manually adding these:

    - --oidc-issuer-url=[REDACTED]
    - --oidc-client-id=kubernetes
    - --oidc-username-claim=email
    - '--oidc-username-prefix=oidc:'
    - --oidc-groups-claim=groups
    - '--oidc-groups-prefix=oidc:'

OIDC then starts to work

[trickyut]

Hello, I want to confirm the same problem.

We used 2.27.0 version to upgrade k8s from 1.30.4 to 1.31.4

After adjusting below parameters in inventory\k8s\group_vars\k8s_cluster\k8-cluster.yaml the /etc/kubernetes/manifests/kube-apiserver.yaml has been generated without having them enabled.

kubernetes_audit: true
kube_token_auth: true

After the upgrade I decided to run cluster.yaml to check if these parameters will be added to kube-apiserver.yaml and that failed as well.

I decided to remove all manifests from /etc/kubernetes/manifests folder on one of the control nodes and trigger cluster.yaml playbook. It crashed on waiting api-server to startup on that node. Manifest files were not recreated

My new config parameters were added by Kubespray to /etc/kubernetes/kubeadm-config.yaml.
and to get manifests recreated with the proper variables i had to run

kubeadm init phase control-plane apiserver --config=/etc/kubernetes/kubeadm-config.yaml

[razorxster]

Hi!

I tried with 2.26. Same :(

[VannTen]

It this still the case in master with #12015 in ?

[razorxster]

Would be nice to create a hotfix tag for 2.26 and 2.27 release too, like 2.26.1 and 2.27.1.

Thanks ina advance!

[VannTen]

So, yes, I presume ?

[CaptainKrby]

Hello, we upgraded to 1.31.7 from this commit and this time the oidc parameters remained. No authentication break. I have no explanation as to whether the bug has been fixed since then.

[VannTen]

I believe this is another case of #11552 (=> modification are not taken into account, but current settings remains).
Could anyone experiencing the bug confirm ? (that you didn't have those OIDC settings before).

[Zuzuske]

I believe this is another case of #11552 (=> modification are not taken into account, but current settings remains). Could anyone experiencing the bug confirm ? (that you didn't have those OIDC settings before).

Hey. I can confirm that I did not have OIDC settings before upgrade, but also on fresh install it also failed.
On my dev cluster I tried an upgrade-cluster.yaml and a cluster.yaml runs, both failed.
On my live cluster I did fresh install, same - on OIDC configuration.

all envs same versions.

[ryanhashemi-tmx]

We are facing the same issue - created the cluster with no OIDC configurations and now unable to set them with the following config in k8s-cluster.yml

kube_oidc_auth: true
kube_oidc_url: https://dex-k8s.dev.xxxx.com
kube_oidc_client_id: k8s01-kubelogin-dev
kube_oidc_username_claim: email
kube_oidc_groups_claim: groups

I don't get any error on upgrade-cluster.yaml or cluster.yaml , but the oidc parameters are not passed to the api-server pods though they are in the kubeconfig file located at /etc/kubernetes/kubeadm-config.yaml on the first master node.

root@k8s01-master001:~ # cat /etc/kubernetes/kubeadm-config.yaml | grep oidc -A 2
  - name: oidc-issuer-url
    value: "https://dex-k8s.dev.xxxx.com"
  - name: oidc-client-id
    value: "k8s01-kubelogin-dev"
  - name: oidc-username-claim
    value: "email"
  - name: oidc-groups-claim
    value: "groups"

[ruant]

Adding my issue with this to the list here.

[Zuzuske]

Fresh install with 2.28.0 still doesn't work.

[VannTen]

This should have been fixed by #12015, this is a bit strange.

[Zuzuske]

This should have been fixed by #12015, this is a bit strange.

Just to be sure, I did a second fresh install for my dev cluster. No OIDC config.

[stasicki]

see: #6004 (comment)

TL;DR - You have to use upgrade-cluster.yml to apply kube_kubeadm_apiserver_extra_args.