[Release-1.28] - Allow setting priorityClassName on ServiceLB daemonset.

[brandond]

Backport fix for Allow setting priorityClassName on ServiceLB daemonset.

• Allow setting priorityClassName on ServiceLB daemonset. #10033

[mdrahman-suse]

Validated with release-1.28 branch on commit 9d5becb

Testing

• Steps: Allow setting priorityClassName on ServiceLB daemonset. #10033 (comment)

Validation

$ k3s -v
k3s version v1.28.10+k3s-9d5becb2 (9d5becb2)
go version go1.21.9

$ kubectl apply -f pc.yaml
priorityclass.scheduling.k8s.io/high-priority created

$ kubectl get priorityClass
NAME                      VALUE        GLOBAL-DEFAULT   AGE
high-priority             1000000      false            5s
system-cluster-critical   2000000000   false            90s
system-node-critical      2000001000   false            90s

$ kgp
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-6799fbcd5-xcx7p                  1/1     Running     0          83s
kube-system   helm-install-traefik-crd-6c6x7           0/1     Completed   0          83s
kube-system   helm-install-traefik-mgghc               0/1     Completed   1          83s
kube-system   local-path-provisioner-6f5d79df6-wlv82   1/1     Running     0          83s
kube-system   metrics-server-54fd9b65b-lr2r8           1/1     Running     0          83s
kube-system   svclb-traefik-391ebb26-nf4zx             2/2     Running     0          66s
kube-system   traefik-7d5f6474df-58zkx                 1/1     Running     0          66s

$ kubectl get pods -n kube-system svclb-traefik-391ebb26-nf4zx -o yaml | grep priority
  priority: 2000001000
  priorityClassName: system-node-critical

$ kubectl annotate svc -n kube-system traefik svccontroller.k3s.cattle.io/priorityclassname=high-priority
service/traefik annotated

$ kubectl get svc -n kube-system traefik -o  yaml | grep priority
    svccontroller.k3s.cattle.io/priorityclassname: high-priority

$ kgp
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-6799fbcd5-xcx7p                  1/1     Running     0          2m24s
kube-system   helm-install-traefik-crd-6c6x7           0/1     Completed   0          2m24s
kube-system   helm-install-traefik-mgghc               0/1     Completed   1          2m24s
kube-system   local-path-provisioner-6f5d79df6-wlv82   1/1     Running     0          2m24s
kube-system   metrics-server-54fd9b65b-lr2r8           1/1     Running     0          2m24s
kube-system   svclb-traefik-391ebb26-rmbh9             2/2     Running     0          14s
kube-system   traefik-7d5f6474df-58zkx                 1/1     Running     0          2m7s

$ kubectl get pods -n kube-system svclb-traefik-391ebb26-rmbh9 -o yaml | grep priority
  priority: 1000000
  priorityClassName: high-priority