KAFKA-5157: Options for handling corrupt data during deserialization #3423

enothereska · Jun 23, 2017

This is the implementation of KIP-161: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers

More tests coming, but basic structure should be in place.

enothereska · Jun 23, 2017

More tests coming, but basic structure should be in place.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5648/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jun 23, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5648/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5662/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jun 23, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5662/
Test PASSed (JDK 7 and Scala 2.11).

@mjsax @dguy @bbejeck @guozhangwang this is the implementation for KIP-161. Have a look when you can. Thanks.

enothereska · Jun 23, 2017

@mjsax @dguy @bbejeck @guozhangwang this is the implementation for KIP-161. Have a look when you can. Thanks.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5667/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jun 23, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5667/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5653/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jun 23, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5653/
Test PASSed (JDK 8 and Scala 2.12).

mjsax · Jun 24, 2017

mjsax requested changes Jun 24, 2017

View changes

Thanks for the patch. Some minor comments.

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

...java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

...va/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

...t/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

mjsax · Jun 24, 2017

streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java

@@ -205,7 +206,7 @@ public ProcessorTopologyTestDriver(final StreamsConfig config,
            final GlobalStateManagerImpl stateManager = new GlobalStateManagerImpl(globalTopology, globalConsumer, stateDirectory);
            globalStateTask = new GlobalStateUpdateTask(globalTopology,
                                                        new GlobalProcessorContextImpl(config, stateManager, streamsMetrics, cache),
-                                                        stateManager
+                                                        stateManager, new LogAndContinueExceptionHandler()


Guess you missed this one :)

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5705/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jun 26, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5705/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5691/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jun 26, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5691/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5707/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jun 26, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5707/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5693/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jun 26, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5693/
Test PASSed (JDK 8 and Scala 2.12).

@dguy @guozhangwang any further comments? Thanks.

enothereska · Jul 3, 2017

@dguy @guozhangwang any further comments? Thanks.

dguy · Jul 3, 2017

dguy reviewed Jul 3, 2017

View changes

Thanks @enothereska - i've left a few comments

...java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

...ain/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java

...ache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

dguy · Jul 3, 2017

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        // catch and process if we have a deserialization handler
+        try {
+            return deserialize(rawRecord);
+        } catch (Exception e) {


Should we add a metric for records skipped due to deserialization errors?

Yes good idea. I'll piggyback on this KIP.

...va/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

...t/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

...ache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5887/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 4, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5887/
Test PASSed (JDK 7 and Scala 2.11).

Thanks @dguy

enothereska · Jul 4, 2017

Thanks @dguy

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5889/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 4, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5889/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5874/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jul 4, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5874/
Test PASSed (JDK 8 and Scala 2.12).

dguy · Jul 4, 2017

dguy reviewed Jul 4, 2017

View changes

Thanks @enothereska. I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state that @guozhangwang suggested

...java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state.

Thanks @dguy I'm not sure what this refers to. I looked back but I don't see @guozhangwang 's comment. Could you elaborate? Thanks.

enothereska · Jul 4, 2017

I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state.

Thanks @dguy I'm not sure what this refers to. I looked back but I don't see @guozhangwang 's comment. Could you elaborate? Thanks.

@enothereska doh! I've got my PRs crossed! ignore me

dguy · Jul 4, 2017

@enothereska doh! I've got my PRs crossed! ignore me

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5899/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 5, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5899/
Test PASSed (JDK 7 and Scala 2.11).

dguy · Jul 5, 2017

dguy reviewed Jul 5, 2017

View changes

1 more comment otherwise LGTM. @mjsax can you please make another pass over it?

.../main/java/org/apache/kafka/streams/processor/internals/RecordQueue.java

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5884/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jul 5, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5884/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5903/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 5, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5903/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5888/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jul 5, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5888/
Test PASSed (JDK 8 and Scala 2.12).

mjsax · Jul 5, 2017

mjsax reviewed Jul 5, 2017

View changes

Some comments.

...ava/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

...java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

...ain/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java

.../org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java

...ache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

mjsax · Jul 5, 2017

streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java

@@ -205,7 +206,7 @@ public ProcessorTopologyTestDriver(final StreamsConfig config,
            final GlobalStateManagerImpl stateManager = new GlobalStateManagerImpl(globalTopology, globalConsumer, stateDirectory);
            globalStateTask = new GlobalStateUpdateTask(globalTopology,
                                                        new GlobalProcessorContextImpl(config, stateManager, streamsMetrics, cache),
-                                                        stateManager
+                                                        stateManager, new LogAndContinueExceptionHandler()


Guess you missed this one :)

...ain/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5904/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jul 6, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5904/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5919/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 6, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5919/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5943/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · Jul 7, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5943/
Test FAILed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5958/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · Jul 7, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5958/
Test FAILed (JDK 7 and Scala 2.11).

Could not write standard input into: Gradle build daemon

enothereska · Jul 7, 2017

Could not write standard input into: Gradle build daemon

retest this please

enothereska · Jul 7, 2017

retest this please

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5959/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · Jul 7, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5959/
Test FAILed (JDK 7 and Scala 2.11).

org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied failed. There is already a JIRA for it https://issues.apache.org/jira/browse/KAFKA-5566.

enothereska · Jul 7, 2017

org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied failed. There is already a JIRA for it https://issues.apache.org/jira/browse/KAFKA-5566.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5944/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · Jul 7, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5944/
Test FAILed (JDK 8 and Scala 2.12).

@dguy I don't have anything else to add to this. Fixing the QueryableState in separate PR. Thanks. Also I just merged with trunk.

enothereska · Jul 7, 2017

@dguy I don't have anything else to add to this. Fixing the QueryableState in separate PR. Thanks. Also I just merged with trunk.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5962/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 7, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5962/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5947/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jul 7, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5947/
Test PASSed (JDK 8 and Scala 2.12).

@guozhangwang any more comments?

enothereska · Jul 10, 2017

@guozhangwang any more comments?

retest this please

dguy · Jul 10, 2017

retest this please

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6015/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · Jul 10, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6015/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6000/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · Jul 10, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6000/
Test PASSed (JDK 8 and Scala 2.12).

dguy · Jul 10, 2017

dguy approved these changes Jul 10, 2017

View changes

Thanks @enothereska, LGTM

guozhangwang · Jul 10, 2017

guozhangwang reviewed Jul 10, 2017

View changes

Left some comments on this PR. Sorry for the late reply.

guozhangwang · Jul 10, 2017

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+                sourceNode.nodeMetrics.sourceNodeSkippedDueToDeserializationError.record();
+            }
+        }
+        return null;


We should return null in the else branch above, in case we add more options in the response that are beyond Fail or Continue.

guozhangwang · Jul 10, 2017

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java

@@ -186,6 +187,7 @@ public NodeMetrics(StreamsMetrics metrics, String name, String sensorNamePrefix)
            this.nodeCreationSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "create", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.nodeDestructionSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "destroy", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.sourceNodeForwardSensor = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "forward", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+            this.sourceNodeSkippedDueToDeserializationError = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "skippedDueToDeserializationError", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);


As this PR gets in already, we probably need to rebase the metrics fix PR to remove the . + name as well for this new metric as well.

Also I'm wondering what's the right approach of this metric: we already have a thread-level skippedRecordsSensor that covers this as it records total fetched records - total enqueued records. And then we have this source-node level metrics that only tracks the skippedRecordsDuetoDeserError. On the other hand, on the thread-level we do not have any other throughput metrics except the skippedRecordsSensor.

So should we just have a higher-hierarchy sensor at the thread level for number of skipped records rates (for whatever reason), or do we want to have a lower source-node level sensor for skipped records rates of different causes? In either way I think the current hierarchy is a bit messed and should be fixed. WDYT @dguy @enothereska ?

Initial implementation

Loading status checks…

a9892b3

More tests

Loading status checks…

a2210a6

enothereska changed the title from KAFKA-5157: Options for handling corrupt data during deserialization [WiP] to KAFKA-5157: Options for handling corrupt data during deserialization Jun 23, 2017

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…
…zation-exceptions

Loading status checks…

5335587

Matthias' comments

Loading status checks…

580223c

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…
…zation-exceptions

Loading status checks…

ab1a585

Damian's comments

Loading status checks…

72f3202

Damian's comments

Loading status checks…

32c860c

Tweak

Loading status checks…

fb2d82b

Matthias' comments

Loading status checks…

c4d0c67

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…
…zation-exceptions

Loading status checks…

4eff572

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…
…zation-exceptions

Loading status checks…

1da2a1d

asfgit closed this in a1f97c8 Jul 10, 2017

apache/kafka

Join GitHub today

KAFKA-5157: Options for handling corrupt data during deserialization #3423

Conversation

enothereska commented Jun 23, 2017

This comment has been minimized.

enothereska commented Jun 23, 2017

This comment has been minimized.

asfgit commented Jun 23, 2017

This comment has been minimized.

asfgit commented Jun 23, 2017

enothereska changed the title from KAFKA-5157: Options for handling corrupt data during deserialization [WiP] to KAFKA-5157: Options for handling corrupt data during deserialization Jun 23, 2017

This comment has been minimized.

enothereska commented Jun 23, 2017

This comment has been minimized.

asfgit commented Jun 23, 2017

This comment has been minimized.

asfgit commented Jun 23, 2017

mjsax requested changes Jun 24, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

asfgit commented Jun 26, 2017

This comment has been minimized.

asfgit commented Jun 26, 2017

This comment has been minimized.

asfgit commented Jun 26, 2017

This comment has been minimized.

asfgit commented Jun 26, 2017

This comment has been minimized.

enothereska commented Jul 3, 2017

dguy reviewed Jul 3, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

asfgit commented Jul 4, 2017

This comment has been minimized.

enothereska commented Jul 4, 2017

This comment has been minimized.

asfgit commented Jul 4, 2017

This comment has been minimized.

asfgit commented Jul 4, 2017

dguy reviewed Jul 4, 2017 View changes

This comment has been minimized.

enothereska commented Jul 4, 2017 • edited

This comment has been minimized.

dguy commented Jul 4, 2017

This comment has been minimized.

asfgit commented Jul 5, 2017

dguy reviewed Jul 5, 2017 View changes

This comment has been minimized.

asfgit commented Jul 5, 2017

This comment has been minimized.

asfgit commented Jul 5, 2017

This comment has been minimized.

asfgit commented Jul 5, 2017

mjsax reviewed Jul 5, 2017 View changes

This comment has been minimized.

This comment has been minimized.

asfgit commented Jul 6, 2017

This comment has been minimized.

asfgit commented Jul 6, 2017

This comment has been minimized.

asfgit commented Jul 7, 2017

This comment has been minimized.

asfgit commented Jul 7, 2017

This comment has been minimized.

enothereska commented Jul 7, 2017

This comment has been minimized.

enothereska commented Jul 7, 2017

This comment has been minimized.

asfgit commented Jul 7, 2017

This comment has been minimized.

enothereska commented Jul 7, 2017

This comment has been minimized.

asfgit commented Jul 7, 2017

This comment has been minimized.

enothereska commented Jul 7, 2017 • edited

This comment has been minimized.

asfgit commented Jul 7, 2017

mjsax requested changes Jun 24, 2017

View changes

dguy reviewed Jul 3, 2017

View changes

dguy reviewed Jul 4, 2017

View changes

enothereska commented Jul 4, 2017 •

edited

dguy reviewed Jul 5, 2017

View changes

mjsax reviewed Jul 5, 2017

View changes

enothereska commented Jul 7, 2017 •

edited

dguy approved these changes Jul 10, 2017

View changes

asfgit closed this in `a1f97c8` Jul 10, 2017

guozhangwang reviewed Jul 10, 2017

View changes