Helm chart: Add priorityClass support for operator

[varet80]

With the following change I am enabled the ability for the operator to be scheduled before minio clusters (if desired)

In case of restarts and or cases where required a full restart. We need operator to be scheduled before minio pods.
This way Pods can find the operator and startup properly.

priorityClass allows this more (https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)

[pjuarezd]

the tests failing are not related to this PR changes, there is no harm in allow set a priorityClass for operator deployment in the helm chart, already tested it.

[allanrogerr]

Tests

Assign in Operator helm chart

operator:
  priorityClassName: first

Test deploy.

Ensure file /testing/deploy-tenant.sh is edited to install_operator helm. Once cluster is up, apply the priority class yaml below

./testing/deploy-tenant.sh

Observe tenant pods for priority

k -n minio-operator get pod/minio-operator-686b95f945-6sswx -o jsonpath="{.spec.priorityClassName}"

first

Validate operator and tenants work as expected

--

Create a priorty class

cat <<EOF | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: first
value: 1000000
globalDefault: false
description: "Test priority class"
EOF

[allanrogerr]

Works as expected @varet80

[pjuarezd]

Thank you @allanrogerr, could you approve the PR please?

[allanrogerr]

For uniformity, I would suggest:

Suggested change

	{{- with .Values.operator.priorityClassName }}
	priorityClassName: {{ . }}
	{{- end }}
	{{- with .Values.operator.priorityClassName }}
	priorityClassName:
	{{- toYaml . | nindent 8 }}
	{{- end }}

[allanrogerr]

Your thoughts?

[pjuarezd]

yeah!, same I though yesterday, but tried that and creates a carriage return that seems unecesary, ie:

priorityClassName:
        prioriryClass

The current code outpus as follows, that is more natural.

priorityClassName: className