Add support for svclb pod PriorityClassName

[brandond]

Proposed Changes

Add support for svclb pod PriorityClassName

At this point I would prefer not to add another CLI flag to control the default priorityClassName, but you can override it on a per-service basis via an annotation. The annotation can be set to an empty value if svclb pods for that service should not have a priorityClassName set.

Types of Changes

enhancement

Verification

See linked issue

Testing

Linked Issues

• Allow setting priorityClassName on ServiceLB daemonset. #10033

User-Facing Change

ServiceLB now sets the priorityClassName on svclb pods to `system-node-critical` by default. This can be overridden on a per-service basis via the `svccontroller.k3s.cattle.io/priorityclassname` annotation.

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 88.88889% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 42.85%. Comparing base (94e29e2) to head (e5ff47c).
Report is 2 commits behind head on master.

Files	Patch %	Lines
pkg/cloudprovider/servicelb.go	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10045      +/-   ##
==========================================
- Coverage   50.07%   42.85%   -7.23%     
==========================================
  Files         158      158              
  Lines       14032    14041       +9     
==========================================
- Hits         7027     6017    -1010     
- Misses       5676     6888    +1212     
+ Partials     1329     1136     -193     

Flag	Coverage Δ
e2etests	36.37% <88.88%> (-10.21%)	⬇️
inttests	19.25% <33.33%> (-17.91%)	⬇️
unittests	16.58% <33.33%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dereknola]

We are gonna need to open a docs PR on this.

[brandond]

btrfs test flaked; merging