KAFKA-3720 : Deprecated BufferExhaustedException and also removed its use and the related sensor metric #1417

MayureshGharat · 2016-05-20T18:49:30Z

BufferExhaustedException is no longerthrown by the new producer. Removed it from the catch clause and deprecated the exception class and removed the corresponding metrics.

ijuma · 2016-05-20T19:17:51Z

Thanks @MayureshGharat. During the the request timeout KIP, was there a discussion about a metric to replace the buffer exhausted one?

ijuma · 2016-05-20T19:18:18Z

cc @junrao

MayureshGharat · 2016-05-20T19:42:23Z

@ijuma I don't think the KIP-19 had any discussion about that. Since now we are failing a request if there is not enough memory by default, its the same behavior as BufferExhaustException and we probably should add another metric for this I think to replace the old one.

MayureshGharat · 2016-05-24T23:34:39Z

@junrao @ijuma any update on this

junrao · 2016-06-02T23:49:29Z

Thanks for the patch. Could you fix the compilation error?

[ant:checkstyle] intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:27:8: Unused import - org.apache.kafka.common.metrics.stats.Rate.
FAILED

ijuma · 2016-06-02T23:56:15Z

@junrao any thoughts on whether we should add a metric to replace the buffer exhausted one?

junrao · 2016-06-03T00:09:31Z

@ijuma : Yes, instead of removing buffer-exhausted-records sensor, we can probably just update it when BufferPool throws a TimeoutException.

MayureshGharat · 2016-06-03T01:03:16Z

@junrao Thanks. I will upload a new PR.

ijuma · 2016-06-16T13:04:37Z

clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java

@@ -472,6 +472,12 @@ private static int parseAcks(String acksString) {
            // handling exceptions and record the errors;
            // for API exceptions return them in the future,
            // for other exceptions throw directly
+        } catch (TimeoutException e) {
+            this.errors.record();
+            this.metrics.sensor("buffer-exhausted-records").record();


Are all cases of this exception due to buffer exhaustion? I thought that it could also happen in other cases? One option would be to keep BufferExhaustedException and have it inherit from TimeoutException. Thoughts?

@ijuma Actually yeah, it can even happen when updating metadata. My bad. This seems like a viable solution or we can throw a BuffereExhaustedException from the Bufferpool.allocate() method instead of TimeoutException and have a proper message of what happened. What do you think?

If I understand you correctly, that's indeed what I had in mind. We throw BufferExhaustedException from BufferPool.allocate. To make it compatible with code that expects a TimeoutException, we make it a subclass of TimeoutException. And then here we catch BufferExhaustedException.

That would mean not deprecating BufferExhaustedException, but updating its documentation to say that it's thrown when a buffer allocation times out.

What do you think @junrao?

Sounds good to me. Will wait for @junrao to comment.

Ping @junrao.

@junrao PING.

kawamuray · 2016-11-18T10:22:09Z

@ijuma @MayureshGharat

Can we proceed this PR?
I'm looking for this PR to complete and make buffer-exhausted-rate metrics meaningful again as it's one of important metrics to me.
Even for other users experience I think this PR should be proceeded ASAP, as as of now users can misunderstand there was no buffer exhaustion by watching this existing, but not working metric always pointing zero.

The idea you two are discussing sounds reasonable to me too, +1.

MayureshGharat · 2016-11-18T17:03:26Z

@junrao : Would you mind taking a look at the comments me and @ijuma discussed? I can re-submit a quick PR for this, if we have a conclusion :)

junrao · 2016-11-18T23:00:40Z

@MayureshGharat : Sorry for the delay. The approach that you and @ijuma described sounds good to me.

…related sensor metric

… thrown on BufferExhaustion

MayureshGharat · 2016-11-21T23:16:49Z

@ijuma I have updated the PR. Would you mind taking another look?

kawamuray · 2016-11-22T04:35:19Z

clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java

+            this.metrics.sensor("buffer-exhausted-records").record();
+            if (this.interceptors != null)
+                this.interceptors.onSendError(record, tp, e);
+            throw e;


I think this is gonna be a slight spec change. Before this, a TimeoutException thrown either by waiting metadata or by waiting buffer allocation were caught by the following clause for ApiException(since TimeoutException extends RetriableException which is an ApiException), so a FutureFailure returned instead of throwing, and the callback triggered too.

As the result of this change, two TimeoutException cases are treated differently:

timeout occurred while waiting metadata update => callback called, FutureFailure returned instead of throw

timeout occurred while waiting buffer allocation => callback not called, throw exception

This sounds confusing and inconsistent. I think we can either take 1. always return FailureFuture for TimeoutException or 2. always throw for timeout exception.
IMO, by design of KafkaProducer(method blocking is part of interface), the latter one makes more sense to me.
WDYT?

This is a good point. The current implementation is a bit misleading with regards to the javadoc which states:

* @throws TimeoutException If the time taken for fetching metadata or allocating memory for the record has surpassed <code>max.block.ms</code>.

We don't actually throw that exception unless you do Future.get. It seems to me that the change in this PR actually fixes the implementation to match the specified contract.

Another option is to change the javadoc to match the implementation (probably less likely to break users) and then we would simply have a check in the ApiException catch block to record the metric for BufferExhaustedException. This seems safer.

Thoughts?

Right, it's indeed an option which is much safer.
Still I think we better throw here by following reasons:

It sounds semantically correct more. Callback and Future should be used for providing result of asynchronous(background) processing. However, these two TimeoutException occurrs while a KafkaProducer is still doing synchronous(foreground) processing and that(kafka producer has to do some foreground processing before it appends record to the accumulator) is the reason why a caller of producer#send is forced to wait until the result turns out, so the caller should receive the result of that call in a synchronous way.

Assuming this is going to be shipped with 0.10.2.0, making a breaking change on behavior isn't preferred but allowed if we leave a correct note on "breaking changes" section.

We can expect this breaking change is relatively less-harm with expect to most users who uses producer in sensitive situation would already using it like below. Users may see some new error logs but it still doesn't break the whole processing(maybe I'm biased, feel free to leave objection if any :D)

try { producer.send(record, (metadata, exception) -> { if (exception != null) { // logging } }); } catch (RuntimeException/* or maybe KafkaException, TimeoutException whatever */ e) { // logging }

kawamuray

Left just one suggestion.

ijuma · 2020-08-02T15:18:53Z

Superseded by #8399.

MayureshGharat force-pushed the kafka-3720 branch from c5b726d to e32ca9a Compare June 10, 2016 01:29

ijuma reviewed Jun 16, 2016
View reviewed changes

MayureshGharat added 3 commits November 21, 2016 15:06

Deprecated BufferExhaustedException and also removed its use and the …

fb66dcf

…related sensor metric

Update the buffer-exhausted-records sensor when a timeoutException is…

ab879b4

… thrown on BufferExhaustion

Made BufferExhaustException a subclass of TimeoutException

d137aff

MayureshGharat force-pushed the kafka-3720 branch from e32ca9a to d137aff Compare November 21, 2016 23:14

kawamuray reviewed Nov 22, 2016

View reviewed changes

kawamuray suggested changes Nov 22, 2016

View reviewed changes

asfgit force-pushed the trunk branch from 1cd01ad to 619fd7a Compare May 3, 2017 00:18

chia7712 mentioned this pull request Apr 1, 2020

KAFKA-3720: Change TimeoutException to BufferExhaustedException when no memory can be allocated for a record within max.block.ms #8399

Merged

3 tasks

ijuma closed this Aug 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-3720 : Deprecated BufferExhaustedException and also removed its use and the related sensor metric #1417

KAFKA-3720 : Deprecated BufferExhaustedException and also removed its use and the related sensor metric #1417

MayureshGharat commented May 20, 2016 •

edited

Loading

ijuma commented May 20, 2016

ijuma commented May 20, 2016

MayureshGharat commented May 20, 2016 •

edited

Loading

MayureshGharat commented May 24, 2016

junrao commented Jun 2, 2016

ijuma commented Jun 2, 2016

junrao commented Jun 3, 2016

MayureshGharat commented Jun 3, 2016

ijuma Jun 16, 2016

MayureshGharat Jun 16, 2016

ijuma Jun 16, 2016

MayureshGharat Jun 16, 2016

ijuma Jul 9, 2016

MayureshGharat Aug 23, 2016

kawamuray commented Nov 18, 2016

MayureshGharat commented Nov 18, 2016 •

edited

Loading

junrao commented Nov 18, 2016

MayureshGharat commented Nov 21, 2016

kawamuray Nov 22, 2016 •

edited

Loading

ijuma Nov 22, 2016

kawamuray Nov 22, 2016 •

edited

Loading

kawamuray left a comment

ijuma commented Aug 2, 2020

KAFKA-3720 : Deprecated BufferExhaustedException and also removed its use and the related sensor metric #1417

KAFKA-3720 : Deprecated BufferExhaustedException and also removed its use and the related sensor metric #1417

Conversation

MayureshGharat commented May 20, 2016 • edited Loading

ijuma commented May 20, 2016

ijuma commented May 20, 2016

MayureshGharat commented May 20, 2016 • edited Loading

MayureshGharat commented May 24, 2016

junrao commented Jun 2, 2016

ijuma commented Jun 2, 2016

junrao commented Jun 3, 2016

MayureshGharat commented Jun 3, 2016

ijuma Jun 16, 2016

Choose a reason for hiding this comment

MayureshGharat Jun 16, 2016

Choose a reason for hiding this comment

ijuma Jun 16, 2016

Choose a reason for hiding this comment

MayureshGharat Jun 16, 2016

Choose a reason for hiding this comment

ijuma Jul 9, 2016

Choose a reason for hiding this comment

MayureshGharat Aug 23, 2016

Choose a reason for hiding this comment

kawamuray commented Nov 18, 2016

MayureshGharat commented Nov 18, 2016 • edited Loading

junrao commented Nov 18, 2016

MayureshGharat commented Nov 21, 2016

kawamuray Nov 22, 2016 • edited Loading

Choose a reason for hiding this comment

ijuma Nov 22, 2016

Choose a reason for hiding this comment

kawamuray Nov 22, 2016 • edited Loading

Choose a reason for hiding this comment

kawamuray left a comment

Choose a reason for hiding this comment

ijuma commented Aug 2, 2020

MayureshGharat commented May 20, 2016 •

edited

Loading

MayureshGharat commented May 20, 2016 •

edited

Loading

MayureshGharat commented Nov 18, 2016 •

edited

Loading

kawamuray Nov 22, 2016 •

edited

Loading

kawamuray Nov 22, 2016 •

edited

Loading