toleration diff and nodeReadinessLabel merge with manifest matchExpressions

[FxKu]

When adding tolerations to the manifest it will simply be ignored since the comparison between running and desired stateful set does not yield a mismatch. This PR will introduce a rolling update trigger for toleration changes just like with nodeAffinity. Only slice length is compared for now.

With nodeAffinity we spotted another problem: If somebody specified nodeAffinity in the manifest and has configured node_readiness_label the resulting affinity on the pods will have two matchExpressions section in the nodeSelectorTerms.

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: manifestlabelkey
            ...
        - matchExpressions:
          - key: myreadinesslabelkey
            ...

Multiple matchExpressions are evaluated with OR condition. Thus, pods could be scheduled on nodes having either one of the specified expressions. This might be fine for some users. However, we want the readiness label to be evaluated always together with other specified affinities. Therefore, we will introduce another option to choose if the config affinity should get merged under one matchExpressions (AND) or stay in it's own (OR).

Finally, this PR also tries to re-enable the taint based eviction and node readiness label tests which use the features touched here.

[sdudoladov]

could you add 2 examples (for OR and AND`) to the admin docs showcasing the different behavior ?

[FxKu]

👍

[sdudoladov]

👍