[Release-1.27] - Allow setting priorityClassName on ServiceLB daemonset.

[brandond]

Backport fix for Allow setting priorityClassName on ServiceLB daemonset.

• Allow setting priorityClassName on ServiceLB daemonset. #10033

[mdrahman-suse]

Validated with release-1.27 branch on commit 6d6cf86

Testing

• Steps: Allow setting priorityClassName on ServiceLB daemonset. #10033 (comment)

Validation

$ k3s -v
k3s version v1.27.14+k3s-6d6cf866 (6d6cf866)
go version go1.21.9

$ kubectl apply -f pc.yaml
priorityclass.scheduling.k8s.io/high-priority created

$ kubectl get priorityClass
NAME                      VALUE        GLOBAL-DEFAULT   AGE
high-priority             1000000      false            5s
system-cluster-critical   2000000000   false            4m32s
system-node-critical      2000001000   false            4m32s

$ kgp
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-77ccd57875-ch4rp                 1/1     Running     0          4m21s
kube-system   helm-install-traefik-crd-jhr9j           0/1     Completed   0          4m21s
kube-system   helm-install-traefik-wqmfs               0/1     Completed   1          4m21s
kube-system   local-path-provisioner-5c6c7f7cd-8pqjj   1/1     Running     0          4m21s
kube-system   metrics-server-c44988498-sz9jl           1/1     Running     0          4m21s
kube-system   svclb-traefik-0f99a4cb-vxz9z             2/2     Running     0          4m3s
kube-system   traefik-7d5c94d587-bp727                 1/1     Running     0          4m4s

$ kubectl get pods -n kube-system svclb-traefik-0f99a4cb-vxz9z -o yaml | grep priority
  priority: 2000001000
  priorityClassName: system-node-critical

$ kubectl annotate svc -n kube-system traefik svccontroller.k3s.cattle.io/priorityclassname=high-priority
service/traefik annotate

$ kubectl get svc -n kube-system traefik -o  yaml | grep priority
    svccontroller.k3s.cattle.io/priorityclassname: high-priority

$ kgp
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-77ccd57875-ch4rp                 1/1     Running     0          4m55s
kube-system   helm-install-traefik-crd-jhr9j           0/1     Completed   0          4m55s
kube-system   helm-install-traefik-wqmfs               0/1     Completed   1          4m55s
kube-system   local-path-provisioner-5c6c7f7cd-8pqjj   1/1     Running     0          4m55s
kube-system   metrics-server-c44988498-sz9jl           1/1     Running     0          4m55s
kube-system   svclb-traefik-0f99a4cb-p7dpb             2/2     Running     0          11s
kube-system   traefik-7d5c94d587-bp727                 1/1     Running     0          4m38s

$ kubectl get pods -n kube-system svclb-traefik-0f99a4cb-p7dpb -o yaml | grep priority
  priority: 1000000
  priorityClassName: high-priority