Add toleration support to ServiceLB DaemonSet

[aleskxyz]

Proposed Changes

This pull request introduces a new feature that allows ServiceLB to handle Kubernetes tolerations defined in a service's annotations. The changes include:

• Implementation of the getTolerations method, which retrieves and validates tolerations from a service's annotations in JSON or YAML string format.
• Updates to the newDaemonSet method to append the retrieved tolerations to the DaemonSet's pod tolerations.

These changes enable users to customize tolerations directly through service annotations, providing greater flexibility in pod scheduling.

Types of Changes

• New Feature

Verification

To verify the changes:

1. Create a Kubernetes Service with the LoadBalancer type and svccontroller.k3s.cattle.io/tolerations annotation containing valid JSON or YAML formatted tolerations list.
2. Deploy the service and ensure that the corresponding ServiceLB DaemonSet's pods include the specified tolerations.
3. Verify that the system handles invalid or malformed tolerations gracefully, emitting appropriate error messages.

yaml example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    svccontroller.k3s.cattle.io/tolerations: |
      - effect: NoSchedule
        key: public
        operator: Exists
      - effect: NoExecute
        key: disk
        operator: Equal
        value: ssd

  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: ingress-nginx

json example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    svccontroller.k3s.cattle.io/tolerations: |
      [
        {
          "effect": "NoSchedule",
          "key": "public",
          "operator": "Exists"
        },
        {
          "effect": "NoExecute",
          "key": "disk",
          "operator": "Equal",
          "value": "ssd"
        }
      ]

  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: ingress-nginx

Testing

Linked Issues

This PR addresses issue #1988.

User-Facing Change

- **New Feature**: Users can now define Kubernetes tolerations for ServiceLB DaemonSet directly in the `svccontroller.k3s.cattle.io/tolerations` annotation on services. 

Further Comments

[brandond]

This looks good, thanks! I have one nit, and a question about whether or not there is existing upstream validation code we can reuse.

Can you provide an example of both json and yaml annotation values?

[aleskxyz]

This looks good, thanks! I have one nit, and a question about whether or not there is existing upstream validation code we can reuse.

Can you provide an example of both json and yaml annotation values?

Thanks for review. I have updated the code and remove the redundant check.
I have also updated the description of the PR with example for both yaml and json

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 18 lines in your changes missing coverage. Please review.

Project coverage is 39.90%. Comparing base (0ee714d) to head (01d4714).
Report is 1 commits behind head on master.

Files	Patch %	Lines
pkg/cloudprovider/servicelb.go	25.00%	17 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (0ee714d) and HEAD (01d4714). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (0ee714d)	HEAD (01d4714)
unittests	1	0
e2etests	7	6

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10687      +/-   ##
==========================================
- Coverage   49.64%   39.90%   -9.75%     
==========================================
  Files         179      162      -17     
  Lines       14955    14333     -622     
==========================================
- Hits         7424     5719    -1705     
- Misses       6168     7437    +1269     
+ Partials     1363     1177     -186     

Flag	Coverage Δ
e2etests	35.91% <25.00%> (-10.44%)	⬇️
inttests	36.58% <25.00%> (-0.06%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[aghodus]

Ios17