fix(kafka): support combined mode with brokers.replicaCount=0#87
Merged
Conversation
Enable combined mode where each KRaft controller also acts as a broker (process.roles=broker,controller) by setting cluster.brokers.replicaCount=0. This allows 3-node HA deployments without separate broker StatefulSets. Fixes: - _helpers.tpl: internalReplicationFactor now uses controllers when brokers=0 - _helpers.tpl: validation allows brokers=0 or >=3 (not 1-2) - _helpers.tpl: minInSyncReplicas validation for both dedicated and combined modes - configmap-scripts.yaml: heredoc YAML indentation fixed with nindent - configmap-scripts.yaml: controller start.sh generates process.roles=broker,controller when brokers=0 - configmap-scripts.yaml: controller exposes CLIENT listener in combined mode - service-client.yaml: selector routes to controller when brokers=0 - statefulset-cluster.yaml: controller exposes port 9092 in combined mode - statefulset-cluster.yaml: broker StatefulSet skipped when brokers=0 Documentation: - README.md: added combined mode to architecture table and quick start - values.yaml: documented brokers.replicaCount=0 behavior - values.schema.json: documented combined mode in schema - docs/combined-mode.md: comprehensive architecture guide - examples/combined-mode/: values.yaml and README.md - ci/combined-mode.yaml: test scenario for combined mode Tested with helm lint --strict and helm template across all ci/ scenarios.
mberlofa
added a commit
that referenced
this pull request
Apr 14, 2026
… combined mode
When brokers.replicaCount=0 (combined mode), the controller headless service must
expose both ports 9093 (controller/quorum) and 9092 (client/inter-broker) to enable
proper inter-broker communication between controller pods acting as brokers.
Problem:
- In combined mode, controllers have process.roles=broker,controller
- Inter-broker replication uses inter.broker.listener.name=CLIENT (port 9092)
- Controller headless service only exposed port 9093
- Result: Inter-broker replication failed because pods couldn't reach each other
via kafka-controller-X.kafka-controller-headless:9092
Impact:
- CRITICAL: Breaks topic replication in combined mode
- Affects: Direct pod-to-pod broker connections via headless DNS
- Symptom: Connection refused when brokers try to replicate data
Solution:
- Conditionally expose port 9092 on controller headless service when brokers=0
- Skip rendering broker headless service entirely when brokers=0 (no broker pods exist)
- Update docs/combined-mode.md to document headless service behavior
Changes:
- templates/service-headless.yaml: Add conditional client port to controller headless
- templates/service-headless.yaml: Wrap broker headless in {{ if gt brokers 0 }}
- docs/combined-mode.md: Document headless service port exposure
Testing:
✓ helm lint --strict
✓ Combined mode: controller headless exposes 9093+9092, broker headless not rendered
✓ Cluster mode: controller headless exposes 9093, broker headless exposes 9092+9094
✓ Single-broker mode: unchanged (exposes both ports as before)
Fixes: Inter-broker communication in combined mode
Related: PR #87 (combined mode support)
mberlofa
added a commit
that referenced
this pull request
Apr 14, 2026
… combined mode (#89) ## Summary Fix critical bug in combined mode where the controller headless service only exposed port 9093 (controller/quorum), breaking inter-broker communication on port 9092 (client). ## Problem In combined mode (`brokers.replicaCount=0`), controller pods run with `process.roles=broker,controller` and handle both: - **Controller traffic** on port 9093 (KRaft quorum) - **Broker traffic** on port 9092 (client + inter-broker replication) The controller headless service (`kafka-controller-headless`) was only exposing port 9093, causing inter-broker replication to fail when brokers tried to connect via stable pod DNS like: ``` kafka-controller-0.kafka-controller-headless:9092 kafka-controller-1.kafka-controller-headless:9092 ``` ## Impact | Severity | Area | Effect | |----------|------|--------| | 🔴 **CRITICAL** | Inter-broker replication | Connection refused - topics cannot replicate | | 🔴 **CRITICAL** | Combined mode production | Completely broken for any multi-replica workload | | ✅ OK | Bootstrap via ClusterIP | Works (uses `kafka:9092` service) | | ✅ OK | Controller quorum | Works (uses port 9093 which was already exposed) | ## Root Cause ```yaml # BEFORE (broken) apiVersion: v1 kind: Service metadata: name: kafka-controller-headless spec: ports: - name: controller port: 9093 # ❌ Missing: port 9092 for inter-broker communication ``` When Kafka's inter-broker replication tries to connect: ```properties # server.properties (generated in combined mode) inter.broker.listener.name=CLIENT # Uses port 9092 ``` Result: `kafka-controller-1` tries to replicate from `kafka-controller-0.kafka-controller-headless:9092` → **Connection refused** ## Solution 1. **Expose port 9092 on controller headless service** when `brokers.replicaCount=0` 2. **Skip rendering broker headless service** when `brokers.replicaCount=0` (no broker pods exist) ```yaml # AFTER (fixed) apiVersion: v1 kind: Service metadata: name: kafka-controller-headless spec: ports: - name: controller port: 9093 {{- if eq (.Values.cluster.brokers.replicaCount | int) 0 }} - name: client port: 9092 # ✅ Now exposed in combined mode {{- end }} ``` ## Testing ### Combined Mode (brokers=0) ```bash $ helm template kafka charts/kafka -f charts/kafka/ci/combined-mode.yaml \ --show-only templates/service-headless.yaml # Result: # ✅ Controller headless service ports: 9093 (controller) + 9092 (client) # ✅ Broker headless service: NOT RENDERED (no broker pods) ``` ### Normal Cluster Mode (brokers=3) ```bash $ helm template kafka charts/kafka -f charts/kafka/ci/cluster.yaml \ --show-only templates/service-headless.yaml # Result: # ✅ Controller headless service ports: 9093 only # ✅ Broker headless service ports: 9092 (client) + 9094 (internal) ``` ### All CI Scenarios ``` ✓ helm lint --strict ✓ single-broker.yaml ✓ cluster.yaml ✓ combined-mode.yaml ✓ cluster-tuned.yaml ✓ metrics.yaml ``` ## Changes ```diff 2 files changed, 10 insertions(+), 1 deletion(-) charts/kafka/templates/service-headless.yaml: + Expose port 9092 on controller headless when brokers=0 + Wrap broker headless service in {{ if gt brokers 0 }} charts/kafka/docs/combined-mode.md: + Document headless service port exposure behavior ``` ## Verification in Production After this fix is deployed, you can verify: ```bash # 1. Check headless service endpoints kubectl get endpoints kafka-controller-headless -o yaml # Expected in combined mode: # ports: # - name: controller # port: 9093 # - name: client # port: 9092 # ✅ Now present # 2. Test inter-broker connectivity kubectl exec -it kafka-controller-1 -- \ /opt/kafka/bin/kafka-broker-api-versions.sh \ --bootstrap-server kafka-controller-0.kafka-controller-headless:9092 # Expected: Success (no connection refused) # 3. Check topic replication kubectl exec -it kafka-controller-0 -- \ /opt/kafka/bin/kafka-topics.sh --describe --topic __consumer_offsets \ --bootstrap-server localhost:9092 # Expected: All replicas in-sync ``` ## Related - Relates to: PR #87 (combined mode support) - **MERGED** - Fixes: Critical bug blocking production use of combined mode ## User Impact - **Breaking changes**: None - **Upgrade path**: Seamless - just upgrade to new chart version - **Backward compatibility**: Fully compatible with existing deployments - **Performance**: No impact (just exposes additional port) --- **This is a critical production bug fix for combined mode. Without this fix, combined mode cannot be used for any workload requiring topic replication.**
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable combined mode where each KRaft controller also acts as a broker (
process.roles=broker,controller) by settingcluster.brokers.replicaCount: 0.This allows 3-node HA deployments without separate broker StatefulSets, ideal for cost-optimized production environments with moderate throughput requirements.
Fixes
Template Corrections:
_helpers.tpl:internalReplicationFactornow usescontrollers.replicaCountwhenbrokers=0_helpers.tpl: Validation allowsbrokers=0or≥3(not 1-2)_helpers.tpl:minInSyncReplicasvalidation for both dedicated and combined modesconfigmap-scripts.yaml: Heredoc YAML indentation fixed withnindentconfigmap-scripts.yaml: Controller generatesprocess.roles=broker,controllerwhenbrokers=0configmap-scripts.yaml: Controller exposes CLIENT listener in combined modeservice-client.yaml: Selector routes to controller whenbrokers=0statefulset-cluster.yaml: Controller exposes port 9092 in combined modestatefulset-cluster.yaml: Broker StatefulSet skipped whenbrokers=0Testing
helm lint --strictpassedhelm templatewith default values passedhelm templatewith allci/*.yamlscenarios passedDocumentation
README.md: Added combined mode to architecture table and quick start examplesdocs/combined-mode.md: Comprehensive architecture guide with when to use, trade-offs, validation stepsexamples/combined-mode/: Production-ready values.yaml with resources, PDB, and detailed READMEci/combined-mode.yaml: Test scenario for CI pipelinevalues.yaml: Documentedbrokers.replicaCount=0behaviorvalues.schema.json: Schema updated with combined mode documentation andminimum: 0Configuration Example
When to Use Combined Mode
Ideal for:
Trade-offs:
Files Changed
Checklist
mainhelm lint --strictpasseshelm templatetested with all CI scenariosvalues.schema.jsonupdated