Deploy default and support custom affinities in our operators

[soenkeliebau]

Geneal Tasks

Beta Give feedback

1. Add ADR 26 on affinities documentation#362
   +0
   
2. POC for two products (kafka and hdfs) done
3. We still need to support the old nodeSelector mechanism
4. Old nodeSelector mechanism is deprecated, Warning in operator is thrown
5. Concepts page written (documentation)

Products

Beta Give feedback

1. [Merged by Bors] - Add affinity structs operator-rs#556
   +0
   
2. [Merged by Bors] - Deploy default and support custom affinities airflow-operator#241
   release/2023-04 +1
   
3. [Merged by Bors] - Deploy default and support custom affinities druid-operator#406
   release/2023-04 +1
   
4. [Merged by Bors] - Deploy default and support custom affinities hbase-operator#322
   release/2023-04 +1
   
5. [Merged by Bors] - Deploy default and support custom affinities hdfs-operator#319
   release/2023-04 +1
   
6. [Merged by Bors] - Deploy default and support custom affinities hive-operator#315
   release/2023-04 +1
   
7. [Merged by Bors] - Deploy default and support custom affinities kafka-operator#557
   7 of 7
   release/2023-04 +1
   
8. [Merged by Bors] - Deploy default and support custom affinities nifi-operator#436
   release/2023-04 +1
   
9. opa-operator -> We always deploy a DaemonSet and don't see any large benefit of adding affinity as of now. Might change in the future, but requires custom logic (compared to the other operators)
10. [Merged by Bors] - Deploy default and support custom affinities spark-k8s-operator#217
    +0
11. [Merged by Bors] - Deploy default and support custom affinities superset-operator#337
    release/2023-04 +1
    
12. [Merged by Bors] - Deploy default and support custom affinities trino-operator#391
    release/2023-04 +1
    
13. [Merged by Bors] - Deploy default and support custom affinities zookeeper-operator#649
    release/2023-04 +1
    

[soenkeliebau]

@nightkr @lfrancke and @soenkeliebau briefly discussed this yesterday and decided on a way forward.

The main discussion was whether it would be sufficient to allow overriding this as part of a generic safety-valve type of functionality where the user is allowed to specify pod_templates and the operator merges these into the pods it would create.

We decided that affinities are sufficiently common that we want to expose this as a fully supported functionality, instead of relying on overrides. Overrides will also be implemented, but under a separate ticket.

To this effect we will look into adding Option<PodAffinitiy> and Option<PodAntiAffinity> to our structs.
These will need to be defineable at the role and rolegroup level.

If these are not defined the operators should add sensible default affinities, like the ones mentioned in the ticket. If these are defined they should override the default affinities. To suppress affinites completely these should be set to null.

[fhennig]

I didn't look into it much, but will be also be able to co-locate roles? I.e. for druid, the broker and router should ideally run on the same node (see here).

Can we also define that I want my Superset to always run preferably run next to the Druid Broker? Maybe that's already a bit advanced, just some ideas :)

[razvan]

This looks like a good chance to introduce first class support for pod templates. These would be merged with templates generated by the operators.. Then, not only affinities can be customized.

[sbernauer]

That was my initial thought as well. After some discussion I was convinced that a "generic" podOverwrite will still need some time as well as development until ready and we need this feature rather quick.
Also affinity is "important enough" to not rely on podOverwrites - at least for our POs.
Happy to have a short call and convince you as well :)

[razvan]

I understand the complexity argument and I agree. Still, we could design the manifest structure now in such a way that doesn't require breaking changes in the future.
For example, instead of adding a config.podAffinity now and painfully change it later, add a config.podTemplateOverwrite.affinity and be forward compatible with the future. Also be more in line with what people already know.

[lfrancke]

All done! Thank you very much!