Support KRaft for consensus building

[maltesander]

Description

This PR add a new role controller to configure KRaft instead of ZooKeeper. This is prep work for Kafka 4.0, where ZooKeeper is removed.

fixes #690

BREAKING Changes:

• renamed server.properties to broker.properties and controller.properties respectively (this affects e.g. config overrides)

What was done?

• CRD:
  • make zookeeperConfigmapName optional
  • split KafkaConfig into Broker and Controller config
  • split out common configs (e.g. resources)
  • added controller role
  • split roles to own module
• Product config:
  • clean out properties
  • remove zookeeper.connect, zookeeper.timeout (never used or overwritten in zk mode)
  • add controller to product config machinery
• Implementation
  • Added sanity check to have Kafka4.0 / Controller and zookeeper mutually exclusive
  • introduce AnyConfig to better handle different config types
  • rework container commands and split to own module
    • KRaft formatting
    • node.id handling
  • rework resources (statefulset, configmap, service) and split to own module
  • increased default broker memory to 2gb
  • removed obsolete errors
  • better constants and config handling
  • hashing for "random" but deterministic integers for node.id
  • cluster-id is set to KafkaCluster metadata.uid
  • Kafka 4.0 support
  • log4j2 support
• Tests:
  • Added use-kraft-controller dimension in almost all tests to split between Kraft / zk

What was not done?

• (Stackable) Listener
• Discovery

TODO

• 3.7.2 no dynamic quorum (bad for scaling) https://developers.redhat.com/articles/2024/11/27/dynamic-kafka-controller-quorum -> scaling flaky
• better liveness / readiness for controller (e.g. check if quorum joinend?)
• zk-kraft migration
• antiaffinities controller / broker?
• PDBs controller?

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[razvan]

LGTM!