KAFKA-13673: disable idempotence when config conflicts #11788

showuon · 2022-02-17T13:16:31Z

Disable idempotence when conflicting config values for acks, retries
and max.in.flight.requests.per.connection are set by the user. For the
former two configs, we log at info level when we disable idempotence
due to conflicting configs. For the latter, we log at warn level since
it's due to an implementation detail that is likely to be surprising.

This mitigates compatibility impact of enabling idempotence by default.

Added unit tests to verify the change in behavior.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

showuon · 2022-02-17T13:17:55Z

clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java

@@ -514,7 +514,7 @@ private TransactionManager configureTransactionState(ProducerConfig config,
                                                         LogContext logContext) {
        TransactionManager transactionManager = null;

-        if (config.idempotenceEnabled()) {
+        if (config.getBoolean(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG)) {


refactor: the idempotenceEnabled() method is only to get the config of ENABLE_IDEMPOTENCE_CONFIG now.

showuon · 2022-02-17T13:19:30Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

-    boolean idempotenceEnabled() {
-        boolean userConfiguredTransactions = this.originals().containsKey(TRANSACTIONAL_ID_CONFIG);
-        boolean idempotenceEnabled = this.getBoolean(ENABLE_IDEMPOTENCE_CONFIG);
-        if (!idempotenceEnabled && userConfiguredTransactions)
-            throw new ConfigException("Cannot set a " + ProducerConfig.TRANSACTIONAL_ID_CONFIG + " without also enabling idempotence.");
-
-        return idempotenceEnabled;
-    }


Move the transaction.id config validation into validation method, so that we don't need to validate it each time when checking idempotenceEnabled.

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

showuon · 2022-02-17T13:21:07Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

+                if (userConfiguredIdempotence) {
+                    throw new ConfigException("Must set " + RETRIES_CONFIG + " to non-zero when using the idempotent producer.");
+                }
+                log.warn("`enable.idempotence` will be disabled because {} config is set to 0.", RETRIES_CONFIG, retries);


log warning to users to let them know the enable.idempotence is disabled.

One question is what we want to do with this longer term. Do we want to upgrade this to an error in Apache Kafka 4.0?

I agree we directly throw exception after AK 4.0.

Thinking about this some more, not sure there's a lot of value in forcing users to set idempotence=false in cases where they're setting acks=1|0 or retries=0. So, I'd change the warning to info for these cases.

max.in.flight.requests.per.connection is different since it's an implementation constraint that idempotence doesn't work when it's > 5 vs inherent to the configuration. For this one, I'd have a warning and mention that it will become an error in Kafka 4.0.

Sounds good to me. I've updated the PR. Also appended a note to upgrade doc. Please take a look. Thank you.

showuon · 2022-02-17T13:23:03Z

@mimaison @hachikuji @vvcephei , call for review. Thanks.

mimaison

Thanks Luke for the PR! Should we also update the docs of these configs?

mimaison · 2022-02-21T17:01:11Z

docs/upgrade.html

@@ -73,7 +74,8 @@ <h5><a id="upgrade_311_notable" href="#upgrade_311_notable">Notable changes in 3
 <ul>
    <li>A bug prevented the producer idempotence default from being applied which meant that it remained disabled unless the user had explicitly set
 	<code>enable.idempotence</code> to true. See <a href="https://issues.apache.org/jira/browse/KAFKA-13598">KAFKA-13598</a>for more details.
-        This issue was fixed and the default is properly applied.</li>
+        This issue was fixed and the default is properly applied.
+        However, if <code>enable.idempotence</code> is unset and the other configs conflicts with it, idempotence will be disabled to avoid break existing producers.</li>


break -> breaking
Same below

mimaison · 2022-02-21T17:04:47Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

+                if (userConfiguredIdempotence) {
+                    throw new ConfigException("Must set " + RETRIES_CONFIG + " to non-zero when using the idempotent producer.");
+                }
+                log.info("`enable.idempotence` will be disabled because {} config is set to 0.", RETRIES_CONFIG, retries);


I think it would read better if we remove config from the sentence.

showuon · 2022-02-22T06:50:19Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

-                                                        + ACKS_CONFIG + "</code> must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible "
-                                                        + "values are set, a <code>ConfigException</code> will be thrown.";
+                                                        + ACKS_CONFIG + "</code> must be 'all'. If incompatible values are set, a <code>ConfigException</code> will be thrown. "
+                                                        + "The default value is `true`. But if incompatible values are set and this config is not set explicitly, idempotent producer will be disabled automatically.";


If these values are not explicitly set by the user, suitable values will be chosen.
-> remove this line because we already default all the values to suitable values.

Added:
The default value is 'true'. But if incompatible values are set and this config is not set explicitly, idempotent producer will be disabled automatically.

We should update the documentation for the 3 configs mentioned here too. I have a suggestion to try and make this particular one a bit clearer regarding the defaults, will post it a bit later.

Can we update MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_DOC, RETRIES_DOC and ACKS_DOC to mention the impact on idempotence?

Also, for this description, I was thinking something like:

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to be less than or equal to " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE
+ " (with message ordering preserved for any allowable value), " + RETRIES_CONFIG + " to be greater than 0, and " + ACKS_CONFIG + " must be 'all'.

Idempotence is enabled by default if no conflicting configurations are set. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled. If idempotence is explicitly enabled and conflicting configurations are set, a ConfigException is thrown.

Thoughts?

This is a difficult one. The update looks better than my current change. Thanks.

showuon · 2022-02-22T06:51:12Z

@mimaison , updated the PR in this commit: 29bb358. Thanks.

mimaison

LGTM

docs/upgrade.html

hachikuji · 2022-02-23T02:07:08Z

@showuon Thanks for the patch. The approach seems reasonable. At least I cannot think of a better alternative. If the user is explicitly opting into weaker semantics, then it seems fair to disable idempotence. The inflight limitation is vexing, but I hope we can do something about that separately.

ijuma

I did a full pass and left some more comments. This is a bit complex, so trying to make it as clear to users as possible.

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java

ijuma · 2022-02-23T15:15:26Z

docs/upgrade.html

@@ -21,7 +21,7 @@

 <h5><a id="upgrade_320_notable" href="#upgrade_320_notable">Notable changes in 3.2.0</a></h5>
    <ul>
-        <li>Idempotence for the producer is enabled by default. In 3.0.0 and 3.1.0, a bug prevented this default from being applied,
+        <li>Idempotence for the producer is enabled by default if no conflicting configurations are set explicitly. In 3.0.0 and 3.1.0, a bug prevented this default from being applied,


Sorry, after reading this again, I think we can simply say "if no conflicting configurations are set". The "explicitly" feels redundant. Same for the two other paragraphs below. Yes, I know I suggested this text. :)

No worries. I agree we should make it as clear as possible, and this update indeed makes it better!

ijuma · 2022-02-23T15:16:48Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

-                                                        + ACKS_CONFIG + "</code> must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible "
-                                                        + "values are set, a <code>ConfigException</code> will be thrown.";
+                                                        + ACKS_CONFIG + "</code> must be 'all'. If incompatible values are set, a <code>ConfigException</code> will be thrown. "
+                                                        + "The default value is `true`. But if incompatible values are set and this config is not set explicitly, idempotent producer will be disabled automatically.";


Can we update MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_DOC, RETRIES_DOC and ACKS_DOC to mention the impact on idempotence?

Also, for this description, I was thinking something like:

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to be less than or equal to " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE
+ " (with message ordering preserved for any allowable value), " + RETRIES_CONFIG + " to be greater than 0, and " + ACKS_CONFIG + " must be 'all'.

Idempotence is enabled by default if no conflicting configurations are set. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled. If idempotence is explicitly enabled and conflicting configurations are set, a ConfigException is thrown.

Thoughts?

showuon · 2022-02-25T03:11:42Z

@ijuma , please take a look again when available. Thanks.

ijuma

In addition to the one minor point inline, I think we should add a note to MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_DOC, RETRIES_DOC and ACKS_DOC to mention the impact on idempotence. Basically copy the part from the idempotence docs so that people can understand the implications of setting the configs within their own documentation.

ijuma · 2022-02-25T13:22:48Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

+        }
+
+        // validate `transaction.id` after validating idempotence dependant configs because `enable.idempotence` config might be overridden
+        idempotenceEnabled = idempotenceEnabled && !shouldDisableIdempotence;


Maybe we can set this to false in the shouldDisableIdempotence block? Seems a bit more natural.

Good suggestion! Updated.

showuon · 2022-02-26T03:25:01Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java


    /** <code>retries</code> */
    public static final String RETRIES_CONFIG = CommonClientConfigs.RETRIES_CONFIG;
    private static final String RETRIES_DOC = "Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error."
            + " Note that this retry is no different than if the client resent the record upon receiving the error."
-            + " Allowing retries without setting <code>" + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + "</code> to 1 will potentially change the"
+            + " Allowing retries and disabling <code>enable.idempotence</code> but without setting <code>" + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + "</code> to 1 will potentially change the"


side fix: currently, we said:

Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records

which is not quite right, because we didn't have the idempotence before.
Updated to:

Allowing retries and disabling enable.idempotence but without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records

Maybe we need to reword this a bit given the new defaults. Move the following nearer the top:

Users should generally"
+ " prefer to leave this config unset and instead use " + DELIVERY_TIMEOUT_MS_CONFIG + " to control

Move the paragraph regarding what happens if idempotence is disabled to the end.

Thoughts?

@ijuma Agree! Below is what I've updated. I think it looks better.

michalxo · 2022-02-28T14:32:41Z

I ran almost full kafka test suite. Got in total around 23 failures, which is great compared to trunk where fails almost everything.
I have executed it last Wednesday, so might not have all the changes since then.

showuon · 2022-03-01T06:56:52Z

Thanks for verifying it @michalxo !

mimaison · 2022-03-01T17:25:35Z

@ijuma Can you take another look? I'm just waiting on this before getting started on 3.0.1.

ijuma · 2022-03-01T17:37:55Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

-    // max.in.flight.requests.per.connection should be less than or equal to 5 when idempotence producer enabled to ensure message ordering
-    private static final int MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE = 5;
+                                                                            + " message re-ordering after a failed send due to retries (i.e., if retries are enabled)."
+                                                                            + " Note additionally that enabling idempotence requires this config value to be less than or equal to " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE + "."


Nit: remove "Note". "Additionally, enabling..." reads better.

ijuma · 2022-03-01T17:39:48Z

@mimaison @showuon I left a couple more minor comments that should hopefully be easy to address.

ijuma

Thanks for the updates and the patience. LGTM. I left a minor final comment below, but no re-review required after you address it.

ijuma · 2022-03-02T23:18:25Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

            + " <code>" + DELIVERY_TIMEOUT_MS_CONFIG + "</code> expires first before successful acknowledgement. Users should generally"
            + " prefer to leave this config unset and instead use <code>" + DELIVERY_TIMEOUT_MS_CONFIG + "</code> to control"
            + " retry behavior."
-            + " <p>"
+            + "<p>"
+            + "Allowing retries and disabling <code>enable.idempotence</code> but without setting <code>" + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + "</code> to 1 will potentially change the"


It reads weird to say

disabling enable.idempotence

I would write the sentence as:

Allowing retries while setting enable.idempotence to false and " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to 1 will potentially change the"

I would also move this paragraph below:

Enabling idempotence requires this config value to be greater than 0...

@ijuma , thanks for the comment. I've updated as below (though I'm not 100% sure I catch your point about this one: I would also move this paragraph below....

This is my update. I think we can merge this PR first to cut the RC for v3.0.1 first. And we can have follow-up PR if you have any other comments. Thank you for your review!

I meant to have "Note that enabling idempotence requires this config..." before "Allowing retries...". And break the two parts with a paragraph.

I see. Updated. Thanks.

After the latest update:

ijuma · 2022-03-03T06:05:42Z

config/server.properties


 # The interval at which log segments are checked to see if they can be deleted according
 # to the retention policies
-log.retention.check.interval.ms=300000
+


Are these changes accidental?

Oops! Let me revert it!

mimaison · 2022-03-03T13:47:51Z

Back porting to 3.0 and 3.1 since we merged e6db0ca into these branches

Disable idempotence when conflicting config values for acks, retries and max.in.flight.requests.per.connection are set by the user. For the former two configs, we log at info level when we disable idempotence due to conflicting configs. For the latter, we log at warn level since it's due to an implementation detail that is likely to be surprising. This mitigates compatibility impact of enabling idempotence by default. Added unit tests to verify the change in behavior. Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Mickael Maison <mickael.maison@gmail.com>

ijuma · 2022-03-03T13:53:57Z

Merged to trunk, 3.1 and 3.0.

ijuma · 2022-03-03T13:54:17Z

@mimaison I did the backports.

mimaison · 2022-03-03T13:55:15Z

Thanks!

Disable idempotence when conflicting config values for acks, retries and max.in.flight.requests.per.connection are set by the user. For the former two configs, we log at info level when we disable idempotence due to conflicting configs. For the latter, we log at warn level since it's due to an implementation detail that is likely to be surprising. This mitigates compatibility impact of enabling idempotence by default. Added unit tests to verify the change in behavior. Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Mickael Maison <mickael.maison@gmail.com>

…he#11788)" This reverts commit af9e419.

…e#11788)" This reverts commit af1f68a.

…he#11788)" This reverts commit af9e419.

…e#11788)" This reverts commit af1f68a.

Disable idempotence when conflicting config values for acks, retries and max.in.flight.requests.per.connection are set by the user. For the former two configs, we log at info level when we disable idempotence due to conflicting configs. For the latter, we log at warn level since it's due to an implementation detail that is likely to be surprising. This mitigates compatibility impact of enabling idempotence by default. Added unit tests to verify the change in behavior. Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Mickael Maison <mickael.maison@gmail.com>

apache#11788)"" This reverts commit 38bee4a.

KAFKA-13673: disable idempotence when config conflicts

b396cac

showuon commented Feb 17, 2022

View reviewed changes

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java Show resolved Hide resolved

showuon commented Feb 17, 2022

View reviewed changes

vvcephei mentioned this pull request Feb 17, 2022

KAFKA-13598: set log4j appender to default acks #11767

Closed

3 tasks

ijuma mentioned this pull request Feb 17, 2022

KAFKA-13598: enable idempotence producer by default and validate the configs #11691

Merged

3 tasks

ijuma requested a review from hachikuji February 17, 2022 18:56

showuon added 2 commits February 20, 2022 10:32

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-13673

73aa5c2

KAFKA-13673: info log to acks/retries, and add upgrade doc

a90c960

mimaison reviewed Feb 21, 2022

View reviewed changes

KAFKA-13673: rephrase comments and add note in doc

29bb358

showuon commented Feb 22, 2022

View reviewed changes

mimaison approved these changes Feb 22, 2022

View reviewed changes

ijuma reviewed Feb 22, 2022

View reviewed changes

docs/upgrade.html Outdated Show resolved Hide resolved

KAFKA-13673: improve upgrade doc

8a5c21b

mimaison mentioned this pull request Feb 23, 2022

Use default non-zero value for retries in producer #11769

Closed

ijuma reviewed Feb 23, 2022

View reviewed changes

showuon mentioned this pull request Feb 24, 2022

KAFKA-13600: Kafka Streams - Fall back to most caught up client if no caught up clients exist #11760

Merged

3 tasks

KAFKA-13673: refactor to make the description clear

d0e8b07

ijuma reviewed Feb 25, 2022

View reviewed changes

KAFKA-13673: refactor and add note in docs for idempotence

fcb4dfa

showuon commented Feb 26, 2022

View reviewed changes

ijuma reviewed Mar 1, 2022

View reviewed changes

KAFKA-13673: refactor the description

53f0aa4

ijuma approved these changes Mar 2, 2022

View reviewed changes

KAFKA-13673: refactor retries doc

5b7a35c

ijuma reviewed Mar 3, 2022

View reviewed changes

KAFKA-13673: refactor retries doc

187d4fc

ijuma merged commit 7c280c1 into apache:trunk Mar 3, 2022

rajinisivaram mentioned this pull request Mar 3, 2022

CONFLUENT: Cherry pick idempotent-by-default fixes from Apache Kafka confluentinc/kafka#671

Merged

3 tasks

vvcephei mentioned this pull request Mar 8, 2022

In-tree version of ksqlDB server crashes on start confluentinc/ksql#8745

Closed

jeffkbkim added a commit to confluentinc/kafka that referenced this pull request Mar 30, 2022

Revert ":KAFKA-13673: disable idempotence when config conflicts (apac…

4ac640d

…he#11788)" This reverts commit af9e419.

jeffkbkim added a commit to confluentinc/kafka that referenced this pull request Mar 30, 2022

Revert "KAFKA-13673: disable idempotence when config conflicts (apach…

53fae26

…e#11788)" This reverts commit af1f68a.

jeffkbkim added a commit to confluentinc/kafka that referenced this pull request Mar 31, 2022

Revert ":KAFKA-13673: disable idempotence when config conflicts (apac…

63b76b8

…he#11788)" This reverts commit af9e419.

jeffkbkim added a commit to confluentinc/kafka that referenced this pull request Mar 31, 2022

Revert "KAFKA-13673: disable idempotence when config conflicts (apach…

38bee4a

…e#11788)" This reverts commit af1f68a.

dengziming mentioned this pull request Apr 28, 2022

KAFKA-13859: Disable idempotence on SCRAM authentication #12105

Closed

3 tasks

jeffkbkim added a commit to confluentinc/kafka that referenced this pull request Nov 3, 2022

Revert "Revert "KAFKA-13673: disable idempotence when config conflicts (

f24af2d

apache#11788)"" This reverts commit 38bee4a.

jeffkbkim added a commit to confluentinc/kafka that referenced this pull request Nov 7, 2022

Revert "Revert "KAFKA-13673: disable idempotence when config conflicts (

a330e22

apache#11788)"" This reverts commit 38bee4a.

KAFKA-13673: disable idempotence when config conflicts #11788

KAFKA-13673: disable idempotence when config conflicts #11788

Conversation

showuon commented Feb 17, 2022 • edited by ijuma Loading

Committer Checklist (excluded from commit message)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ijuma Feb 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

showuon commented Feb 17, 2022

mimaison left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

showuon Feb 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

showuon commented Feb 22, 2022

mimaison left a comment

Choose a reason for hiding this comment

hachikuji commented Feb 23, 2022

ijuma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

showuon commented Feb 25, 2022

ijuma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalxo commented Feb 28, 2022

showuon commented Mar 1, 2022

mimaison commented Mar 1, 2022

ijuma Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

ijuma commented Mar 1, 2022

ijuma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimaison commented Mar 3, 2022

ijuma commented Mar 3, 2022

ijuma commented Mar 3, 2022

mimaison commented Mar 3, 2022

showuon commented Feb 17, 2022 •

edited by ijuma

Loading

ijuma Feb 18, 2022 •

edited

Loading

showuon Feb 22, 2022 •

edited

Loading

ijuma Mar 1, 2022 •

edited

Loading