Skip to content

KAFKA-20890: Restore dynamic broker configs before startup - #23304

Open
1fanwang wants to merge 3 commits into
apache:trunkfrom
1fanwang:fix/kafka-20890-dynamic-config-restore
Open

KAFKA-20890: Restore dynamic broker configs before startup#23304
1fanwang wants to merge 3 commits into
apache:trunkfrom
1fanwang:fix/kafka-20890-dynamic-config-restore

Conversation

@1fanwang

@1fanwang 1fanwang commented Aug 29, 2026

Copy link
Copy Markdown

A broker config rolled out safely as 1, then 2, then 4 comes back as the static value 1 after a KRaft broker restarts without a current local metadata snapshot. The broker still reports 4: the dynamic cache and the cluster-default DescribeConfigs response both say 4 while the running broker uses 1. Re-applying 4 is then rejected as an invalid jump from 1, so the operator can neither see nor restore the value.

Startup builds the quota managers and other runtime components from the static config and installs the metadata publishers much later, so persisted broker configs arrive after everything that reads them was constructed. A current local snapshot hides the gap, so this surfaces only on a restart with a missing or stale one.

The broker now loads the cluster-default and per-broker configs together from the metadata loader's first complete image, before those components are built, through a short-lived publisher removed once startup continues. A restarted broker now runs with 4, reports 4, and validates a later update against 4. Live updates validate their reconfigurables before applying, and a rejected update restores the previous maps. A same-instance restart also clears values removed while the broker was offline.

Fixes https://issues.apache.org/jira/browse/KAFKA-20890

Testing

A parameterized restart regression covers all three snapshot states.

snapshot at restart trunk with this change
none fails, active value 1 passes
stale fails, active value 1 passes
current passes passes
Raw output
$ ./gradlew :core:test --tests 'kafka.server.DynamicBrokerReconfigurationTest.testRestoreDynamicConfigFromMetadataLog'

# trunk f5e01c7b0a, with only the new test applied
snapshot=none    FAILED  org.opentest4j.AssertionFailedError: expected: <4> but was: <1>
snapshot=stale   FAILED  org.opentest4j.AssertionFailedError: expected: <4> but was: <1>
snapshot=current PASSED
3 tests completed, 2 failed
BUILD FAILED in 1m 6s

# with this change
snapshot=none    PASSED
snapshot=stale   PASSED
snapshot=current PASSED
BUILD SUCCESSFUL in 42s

$ ./gradlew :core:test --tests 'kafka.server.DynamicBrokerConfigTest' \
    --tests 'kafka.server.metadata.BrokerMetadataPublisherTest'
DynamicBrokerConfigTest > testConfigUpdateWithReconfigurableValidationFailure() PASSED
BrokerMetadataPublisherTest > testInitialDynamicConfigFailureKeepsPublisherRemovable() PASSED
BUILD SUCCESSFUL in 2m 29s

@github-actions github-actions Bot added triage PRs from the community core Kafka Broker labels Aug 29, 2026
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Generated-by: Claude Opus 5 (claude-opus-5)
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Generated-by: Claude Opus 5 (claude-opus-5)
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Generated-by: Claude Opus 5 (claude-opus-5)
@1fanwang
1fanwang force-pushed the fix/kafka-20890-dynamic-config-restore branch from a7cbd91 to 050f997 Compare August 29, 2026 13:44
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Kafka Broker needs-attention triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant