KAFKA-5157: Options for handling corrupt data during deserialization #3423

enothereska · 2017-06-23T13:35:00Z

This is the implementation of KIP-161: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers


          Initial implementation

enothereska · 2017-06-23T13:35:14Z

More tests coming, but basic structure should be in place.

asfgit · 2017-06-23T14:31:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5648/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-06-23T14:34:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5662/
Test PASSed (JDK 7 and Scala 2.11).


          More tests

enothereska · 2017-06-23T15:17:52Z

@mjsax @dguy @bbejeck @guozhangwang this is the implementation for KIP-161. Have a look when you can. Thanks.

asfgit · 2017-06-23T16:16:04Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5667/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-06-23T16:43:22Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5653/
Test PASSed (JDK 8 and Scala 2.12).

mjsax

Thanks for the patch. Some minor comments.

mjsax · 2017-06-24T23:02:06Z

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

@@ -239,6 +241,13 @@
    private static final String STATE_DIR_DOC = "Directory location for state store.";

    /**
+     * {@code default.deserialization.exception.handler}


Nit: Can you please add this in alphabetic order. Thx.

mjsax · 2017-06-24T23:02:06Z

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

@@ -339,6 +348,11 @@
                    CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
                    Importance.MEDIUM,
                    CommonClientConfigs.SECURITY_PROTOCOL_DOC)
+            .define(DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,


Nit. As above. Alphabetic within the "medium" group.

mjsax · 2017-06-24T23:02:06Z

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

@@ -339,6 +348,11 @@
                    CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
                    Importance.MEDIUM,
                    CommonClientConfigs.SECURITY_PROTOCOL_DOC)
+            .define(DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
+                Type.CLASS,
+                LogAndFailExceptionHandler.class.getName(),


Nit: remove getName()

They all have getName. Should I remove all? Otherwise it will look inconsistent.

Sounds good to me. Not sure why we started with this annoying .getName() in the first place...

mjsax · 2017-06-24T23:02:06Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+    private static final Logger log = LoggerFactory.getLogger(StreamThread.class);
+
+    @Override
+    public DeserializationHandlerResponse handle(ProcessorContext context,


Nit: add final

mjsax · 2017-06-24T23:02:06Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+    }
+
+    @Override
+    public void configure(Map<String, ?> configs) {


mjsax · 2017-06-24T23:02:06Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

@@ -129,6 +135,55 @@ public void shouldProcessRecordsForOtherTopic() throws Exception {
        assertEquals(0, sourceOne.numReceived);
    }

+    @Test(expected = StreamsException.class)


We should not use expected here but try { globalStateTask.update(...); fail(...): } catch(ignore) {} pattern. Also, the ConsumerRecord() should be instantiated outside/before the try-catch blog.

mjsax · 2017-06-24T23:02:06Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

+    }
+
+
+    @Test(expected = StreamsException.class)


mjsax · 2017-06-24T23:02:06Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

    @Test(expected = StreamsException.class)
    public void shouldThrowOnNegativeTimestamp() {
        final List<ConsumerRecord<byte[], byte[]>> records = Collections.singletonList(
                new ConsumerRecord<>("topic", 1, 1, -1L, TimestampType.CREATE_TIME, 0L, 0, 0, recordKey, recordValue));

        final RecordQueue queue = new RecordQueue(new TopicPartition(topics[0], 1),
                                                          new MockSourceNode<>(topics, intDeserializer, intDeserializer),
-                                                          new FailOnInvalidTimestamp());
+                                                          new FailOnInvalidTimestamp(), new LogAndContinueExceptionHandler(), null);


Nit: please one argument per line.

mjsax · 2017-06-24T23:02:06Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

@@ -158,7 +189,7 @@ public void shouldDropOnNegativeTimestamp() {

        final RecordQueue queue = new RecordQueue(new TopicPartition(topics[0], 1),
                                                  new MockSourceNode<>(topics, intDeserializer, intDeserializer),
-                                                  new LogAndSkipOnInvalidTimestamp());
+                                                  new LogAndSkipOnInvalidTimestamp(), new LogAndContinueExceptionHandler(), null);


mjsax · 2017-06-24T23:02:06Z

streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java

@@ -205,7 +206,7 @@ public ProcessorTopologyTestDriver(final StreamsConfig config,
            final GlobalStateManagerImpl stateManager = new GlobalStateManagerImpl(globalTopology, globalConsumer, stateDirectory);
            globalStateTask = new GlobalStateUpdateTask(globalTopology,
                                                        new GlobalProcessorContextImpl(config, stateManager, streamsMetrics, cache),
-                                                        stateManager
+                                                        stateManager, new LogAndContinueExceptionHandler()


Guess you missed this one :)


          Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

…zation-exceptions

asfgit · 2017-06-26T14:19:46Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5705/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-06-26T14:22:24Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5691/
Test PASSed (JDK 8 and Scala 2.12).


          Matthias' comments

asfgit · 2017-06-26T16:01:35Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5707/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-06-26T16:10:15Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5693/
Test PASSed (JDK 8 and Scala 2.12).

enothereska · 2017-07-03T08:06:16Z

@dguy @guozhangwang any further comments? Thanks.

dguy

Thanks @enothereska - i've left a few comments

dguy · 2017-07-03T10:20:29Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+                                                 final Exception exception) {
+
+        log.warn("Deserialization exception {}. Processor context is {} and record is {}",
+            exception.toString(), context, record);


do we want the exception stacktrace?

dguy · 2017-07-03T10:20:29Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java

+                                                 final ConsumerRecord<byte[], byte[]> record,
+                                                 final Exception exception) {
+
+        log.warn("Deserialization exception {}. Processor context is {} and record is {}",


dguy · 2017-07-03T10:20:29Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        } catch (Exception e) {
+            DeserializationExceptionHandler.DeserializationHandlerResponse response =
+                deserializationExceptionHandler.handle(processorContext, rawRecord, e);
+            if (response.id == DeserializationExceptionHandler.DeserializationHandlerResponse.FAIL.id) {


Why not just if(response == DeserializationExceptionHandler.DeserializationHandlerResponse.FAIL) ?

dguy · 2017-07-03T10:20:29Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        try {
+            return deserialize(rawRecord);
+        } catch (Exception e) {
+            DeserializationExceptionHandler.DeserializationHandlerResponse response =


nit: final

dguy · 2017-07-03T10:20:29Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        // catch and process if we have a deserialization handler
+        try {
+            return deserialize(rawRecord);
+        } catch (Exception e) {


Should we add a metric for records skipped due to deserialization errors?

Yes good idea. I'll piggyback on this KIP.

dguy · 2017-07-03T10:20:29Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

+                                  final byte[] key,
+                                  final byte[] recordValue,
+                                  boolean failExpected) {
+        ConsumerRecord record = new ConsumerRecord<>("t2", 1, 1,


nit: final

dguy · 2017-07-03T10:20:29Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

@@ -140,14 +142,45 @@ public void shouldThrowStreamsExceptionWhenValueDeserializationFails() throws Ex
        queue.addRawRecords(records);
    }

+    @Test
+    public void shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler() throws Exception {
+        RecordQueue queue = new RecordQueue(new TopicPartition(topics[0], 1),


maybe extract to a field as it is also used in the test below

dguy · 2017-07-03T10:20:29Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+    public ConsumerRecord<Object, Object> tryDeserialize(final ProcessorContext processorContext,
+                                                         ConsumerRecord<byte[], byte[]> rawRecord) {
+
+        if (deserializationExceptionHandler == null) {


Can we assume and/or ensure that this is non-null? Then we won't need this check here and can just have a single path through this method. I think we default to LogAndFail... if nothing is set - right?


          Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

…zation-exceptions

asfgit · 2017-07-04T09:15:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5887/
Test PASSed (JDK 7 and Scala 2.11).


          Damian's comments

enothereska · 2017-07-04T09:25:40Z

Thanks @dguy

asfgit · 2017-07-04T10:25:07Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5889/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-04T10:52:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5874/
Test PASSed (JDK 8 and Scala 2.12).

dguy

Thanks @enothereska. I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state that @guozhangwang suggested

dguy · 2017-07-04T14:02:23Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+    public DeserializationHandlerResponse handle(final ProcessorContext context,
+                                                 final ConsumerRecord<byte[], byte[]> record,
+                                                 final Exception exception) {
+        StringWriter sWriter = new StringWriter();


You can just pass the exception as the last parameter to i.e, log.warn(" p1: {} p2: {}", p1, p2, exception)

Also, just realised that ProcessorContextImpl doesn't have a toString. Perhaps we just need to log the taskId, topic, partition, and offset? i.e.,
log.warn("Exception caught during Deserialization, taskId: {}, topic:{}, partition:{}, offset:{}", context.taskId(), record.topic(), record.partition(), record.offset(), exception)

Same in the other exception handler

If I pass just the exception, it reverts to exception.toString(). It won't print the stack trace. As a typical NPE example:

with just exception passed: java.lang.NullPointerException

with stack trace:```
java.lang.NullPointerException
at org.rocksdb.RocksDB.get(RocksDB.java:791)
at org.apache.kafka.streams.state.internals.RocksDBStore.getInternal(RocksDBStore.java:235)
at org.apache.kafka.streams.state.internals.RocksDBStore.get(RocksDBStore.java:219)
at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.delete(ChangeLoggingKeyValueBytesStore.java:80)
at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueStore.delete(ChangeLoggingKeyValueStore.java:96)

Shall we add a toString to ProcessorContextImpl? That sounds useful.

@enothereska

If I pass just the exception, it reverts to exception.toString(). It won't print the stack trace.

My guess is that you are adding a {} for the exception, in which case it probably will just print the string. You don't need to add it. For example, from another one we have:
log.warn("{} Failed offset commits {}: ", logPrefix, consumedOffsetsAndMetadata, e)

Having toString on ProcessorContextImpl might be useful, but it also might be too much information for this particular log message

enothereska · 2017-07-04T14:51:31Z

I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state.

Thanks @dguy I'm not sure what this refers to. I looked back but I don't see @guozhangwang 's comment. Could you elaborate? Thanks.

dguy · 2017-07-04T15:00:12Z

@enothereska doh! I've got my PRs crossed! ignore me


          Damian's comments

asfgit · 2017-07-05T10:00:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5899/
Test PASSed (JDK 7 and Scala 2.11).

dguy

1 more comment otherwise LGTM. @mjsax can you please make another pass over it?

dguy · 2017-07-05T10:20:39Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordQueue.java

@@ -80,8 +88,14 @@ public TopicPartition partition() {
     * @return the size of this queue
     */
    public int addRawRecords(Iterable<ConsumerRecord<byte[], byte[]>> rawRecords) {
+        ConsumerRecord<Object, Object> record = null;


I don't see any reason for this variable to not be declared inside the loop as it was previously

asfgit · 2017-07-05T10:24:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5884/
Test PASSed (JDK 8 and Scala 2.12).


          Tweak

asfgit · 2017-07-05T14:02:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5903/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-05T14:25:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5888/
Test PASSed (JDK 8 and Scala 2.12).

mjsax

Some comments.

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+import org.apache.kafka.common.Configurable;
+import org.apache.kafka.streams.processor.ProcessorContext;
+
+public interface DeserializationExceptionHandler extends Configurable {


We should have proper class JavaDocs for all public interfaces.

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+
+public interface DeserializationExceptionHandler extends Configurable {
+    /**
+     * Inspect a record and the exception received


Nit: . missing at the end

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+        public final short id;
+
+        DeserializationHandlerResponse(int id, String name) {
+            this.id = (short) id;


iIf we go with short, the parameter should be short, too?

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+        /** the permanent and immutable id of an API--this can't change ever */
+        public final short id;
+
+        DeserializationHandlerResponse(int id, String name) {


nit: add final

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+                                                 final Exception exception) {
+
+        log.warn("Exception caught during Deserialization, " +
+                "taskId: {}, topic:{}, partition:{}, offset:{}",


Nit: some whitespaces are missing

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java

+                                                 final Exception exception) {
+
+        log.warn("Exception caught during Deserialization, " +
+                        "taskId: {}, topic:{}, partition:{}, offset:{}",


Nit: fix indention and some whitespaces are missing

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java

@@ -63,7 +56,7 @@ public GlobalStateUpdateTask(final ProcessorTopology topology,
        for (final String storeName : storeNames) {
            final String sourceTopic = storeNameToTopic.get(storeName);
            final SourceNode source = topology.source(sourceTopic);
-            deserializers.put(sourceTopic, new SourceNodeAndDeserializer(source, new SourceNodeRecordDeserializer(source)));
+            deserializers.put(sourceTopic, new SourceNodeRecordDeserializer(source, this.deserializationExceptionHandler));


Nit: remove this

mjsax · 2017-07-05T20:24:27Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+            final DeserializationExceptionHandler.DeserializationHandlerResponse response =
+                deserializationExceptionHandler.handle(processorContext, rawRecord, e);
+            if (response == DeserializationExceptionHandler.DeserializationHandlerResponse.FAIL) {
+                throw e;


Should we wrap this with a StreamsException? This way, we could add a detailed error message and explain in the msg text that uses can set a different exception handler etc (similar to timestamp extractor?) -- so users don't need to ask at the mailing list :)

mjsax · 2017-07-05T20:24:27Z

streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java

@@ -205,7 +206,7 @@ public ProcessorTopologyTestDriver(final StreamsConfig config,
            final GlobalStateManagerImpl stateManager = new GlobalStateManagerImpl(globalTopology, globalConsumer, stateDirectory);
            globalStateTask = new GlobalStateUpdateTask(globalTopology,
                                                        new GlobalProcessorContextImpl(config, stateManager, streamsMetrics, cache),
-                                                        stateManager
+                                                        stateManager, new LogAndContinueExceptionHandler()


Guess you missed this one :)

mjsax · 2017-07-05T20:24:27Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java

@@ -186,6 +187,8 @@ public NodeMetrics(StreamsMetrics metrics, String name, String sensorNamePrefix)
            this.nodeCreationSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "create", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.nodeDestructionSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "destroy", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.sourceNodeForwardSensor = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "forward", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+            this.sourceNodeSkippedDueToDeserializationError = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "skippedDueToDeserializationError", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+


Nit: remove empty line


          Matthias' comments

asfgit · 2017-07-06T09:37:11Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5904/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-07-06T09:42:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5919/
Test PASSed (JDK 7 and Scala 2.11).


          Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

…zation-exceptions

asfgit · 2017-07-07T09:42:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5943/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-07-07T09:43:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5958/
Test FAILed (JDK 7 and Scala 2.11).

enothereska · 2017-07-07T09:44:37Z

Could not write standard input into: Gradle build daemon

enothereska · 2017-07-07T09:44:43Z

retest this please

asfgit · 2017-07-07T10:44:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5959/
Test FAILed (JDK 7 and Scala 2.11).

enothereska · 2017-07-07T10:46:04Z

org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied failed. There is already a JIRA for it https://issues.apache.org/jira/browse/KAFKA-5566.

asfgit · 2017-07-07T10:48:20Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5944/
Test FAILed (JDK 8 and Scala 2.12).

enothereska · 2017-07-07T11:01:24Z

@dguy I don't have anything else to add to this. Fixing the QueryableState in separate PR. Thanks. Also I just merged with trunk.


          Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

…zation-exceptions

asfgit · 2017-07-07T13:30:28Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5962/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-07T13:35:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5947/
Test PASSed (JDK 8 and Scala 2.12).

enothereska · 2017-07-10T08:43:55Z

@guozhangwang any more comments?

dguy · 2017-07-10T17:13:38Z

retest this please

asfgit · 2017-07-10T18:18:50Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6015/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-10T18:42:30Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6000/
Test PASSed (JDK 8 and Scala 2.12).

dguy

Thanks @enothereska, LGTM

guozhangwang

Left some comments on this PR. Sorry for the late reply.

guozhangwang · 2017-07-10T20:06:44Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+                sourceNode.nodeMetrics.sourceNodeSkippedDueToDeserializationError.record();
+            }
+        }
+        return null;


We should return null in the else branch above, in case we add more options in the response that are beyond Fail or Continue.

guozhangwang · 2017-07-10T20:06:44Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java

@@ -186,6 +187,7 @@ public NodeMetrics(StreamsMetrics metrics, String name, String sensorNamePrefix)
            this.nodeCreationSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "create", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.nodeDestructionSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "destroy", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.sourceNodeForwardSensor = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "forward", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+            this.sourceNodeSkippedDueToDeserializationError = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "skippedDueToDeserializationError", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);


As this PR gets in already, we probably need to rebase the metrics fix PR to remove the . + name as well for this new metric as well.

Also I'm wondering what's the right approach of this metric: we already have a thread-level skippedRecordsSensor that covers this as it records total fetched records - total enqueued records. And then we have this source-node level metrics that only tracks the skippedRecordsDuetoDeserError. On the other hand, on the thread-level we do not have any other throughput metrics except the skippedRecordsSensor.

So should we just have a higher-hierarchy sensor at the thread level for number of skipped records rates (for whatever reason), or do we want to have a lower source-node level sensor for skipped records rates of different causes? In either way I think the current hierarchy is a bit messed and should be fixed. WDYT @dguy @enothereska ?

Initial implementation

Loading status checks…

a9892b3

More tests

Loading status checks…

a2210a6

enothereska changed the title ~~KAFKA-5157: Options for handling corrupt data during deserialization [WiP]~~ KAFKA-5157: Options for handling corrupt data during deserialization Jun 23, 2017

mjsax requested changes Jun 24, 2017

View changes

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

Loading status checks…

5335587

…zation-exceptions

Matthias' comments

Loading status checks…

580223c

dguy reviewed Jul 3, 2017

View changes

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

Loading status checks…

ab1a585

…zation-exceptions

Damian's comments

Loading status checks…

72f3202

dguy reviewed Jul 4, 2017

View changes

Damian's comments

Loading status checks…

32c860c

dguy reviewed Jul 5, 2017

View changes

Tweak

Loading status checks…

fb2d82b

mjsax reviewed Jul 5, 2017

View changes

Matthias' comments

Loading status checks…

c4d0c67

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

Loading status checks…

4eff572

…zation-exceptions

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

Loading status checks…

1da2a1d

…zation-exceptions

dguy approved these changes Jul 10, 2017

View changes

asfgit closed this in a1f97c8 Jul 10, 2017

guozhangwang reviewed Jul 10, 2017

View changes

apache / kafka

Join GitHub today

KAFKA-5157: Options for handling corrupt data during deserialization #3423

KAFKA-5157: Options for handling corrupt data during deserialization #3423

Conversation

enothereska commented Jun 23, 2017

enothereska commented Jun 23, 2017

asfgit commented Jun 23, 2017

asfgit commented Jun 23, 2017

enothereska commented Jun 23, 2017

asfgit commented Jun 23, 2017

asfgit commented Jun 23, 2017

mjsax left a comment

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

enothereska Jun 26, 2017 Author Contributor

This comment has been minimized.

mjsax Jun 26, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jun 24, 2017 Member

This comment has been minimized.

mjsax Jul 5, 2017 Member

asfgit commented Jun 26, 2017

asfgit commented Jun 26, 2017

asfgit commented Jun 26, 2017

asfgit commented Jun 26, 2017

enothereska commented Jul 3, 2017

dguy left a comment

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

enothereska Jul 4, 2017 Author Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

This comment has been minimized.

dguy Jul 3, 2017 Contributor

asfgit commented Jul 4, 2017

enothereska commented Jul 4, 2017

asfgit commented Jul 4, 2017

asfgit commented Jul 4, 2017

dguy left a comment

This comment has been minimized.

dguy Jul 4, 2017 Contributor

This comment has been minimized.

enothereska Jul 4, 2017 Author Contributor

This comment has been minimized.

enothereska Jul 4, 2017 Author Contributor

This comment has been minimized.

dguy Jul 4, 2017 Contributor

This comment has been minimized.

dguy Jul 4, 2017 Contributor

enothereska commented Jul 4, 2017 • edited

dguy commented Jul 4, 2017

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

enothereska Jun 26, 2017
Author Contributor

mjsax Jun 26, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jun 24, 2017
Member

mjsax Jul 5, 2017
Member

dguy Jul 3, 2017
Contributor

dguy Jul 3, 2017
Contributor

dguy Jul 3, 2017
Contributor

dguy Jul 3, 2017
Contributor

dguy Jul 3, 2017
Contributor

enothereska Jul 4, 2017
Author Contributor

dguy Jul 3, 2017
Contributor

dguy Jul 3, 2017
Contributor

dguy Jul 3, 2017
Contributor

dguy Jul 4, 2017
Contributor

enothereska Jul 4, 2017
Author Contributor

enothereska Jul 4, 2017
Author Contributor

dguy Jul 4, 2017
Contributor

dguy Jul 4, 2017
Contributor

enothereska commented Jul 4, 2017 •

edited

dguy Jul 5, 2017
Contributor

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

mjsax Jul 5, 2017
Member

enothereska commented Jul 7, 2017 •

edited

guozhangwang Jul 10, 2017
Contributor

guozhangwang Jul 10, 2017
Contributor

guozhangwang Jul 10, 2017
Contributor