KAFKA-5157: Options for handling corrupt data during deserialization #3423

enothereska · 2017-06-23T13:35:00Z

This is the implementation of KIP-161: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers

enothereska · 2017-06-23T13:35:14Z

More tests coming, but basic structure should be in place.

asfgit · 2017-06-23T14:31:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5648/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-06-23T14:34:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5662/
Test PASSed (JDK 7 and Scala 2.11).

enothereska · 2017-06-23T15:17:52Z

@mjsax @dguy @bbejeck @guozhangwang this is the implementation for KIP-161. Have a look when you can. Thanks.

asfgit · 2017-06-23T16:16:04Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5667/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-06-23T16:43:22Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5653/
Test PASSed (JDK 8 and Scala 2.12).

mjsax

Thanks for the patch. Some minor comments.

mjsax · 2017-06-24T20:43:52Z

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

@@ -239,6 +241,13 @@
    private static final String STATE_DIR_DOC = "Directory location for state store.";

    /**
+     * {@code default.deserialization.exception.handler}


Nit: Can you please add this in alphabetic order. Thx.

mjsax · 2017-06-24T20:44:25Z

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

@@ -339,6 +348,11 @@
                    CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
                    Importance.MEDIUM,
                    CommonClientConfigs.SECURITY_PROTOCOL_DOC)
+            .define(DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,


Nit. As above. Alphabetic within the "medium" group.

mjsax · 2017-06-24T20:44:46Z

streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java

@@ -339,6 +348,11 @@
                    CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
                    Importance.MEDIUM,
                    CommonClientConfigs.SECURITY_PROTOCOL_DOC)
+            .define(DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
+                Type.CLASS,
+                LogAndFailExceptionHandler.class.getName(),


Nit: remove getName()

They all have getName. Should I remove all? Otherwise it will look inconsistent.

Sounds good to me. Not sure why we started with this annoying .getName() in the first place...

mjsax · 2017-06-24T20:50:06Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+    private static final Logger log = LoggerFactory.getLogger(StreamThread.class);
+
+    @Override
+    public DeserializationHandlerResponse handle(ProcessorContext context,


Nit: add final

mjsax · 2017-06-24T20:50:17Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+    }
+
+    @Override
+    public void configure(Map<String, ?> configs) {


mjsax · 2017-06-24T22:58:35Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

@@ -129,6 +135,55 @@ public void shouldProcessRecordsForOtherTopic() throws Exception {
        assertEquals(0, sourceOne.numReceived);
    }

+    @Test(expected = StreamsException.class)


We should not use expected here but try { globalStateTask.update(...); fail(...): } catch(ignore) {} pattern. Also, the ConsumerRecord() should be instantiated outside/before the try-catch blog.

mjsax · 2017-06-24T22:59:35Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

+    }
+
+
+    @Test(expected = StreamsException.class)


mjsax · 2017-06-24T23:01:24Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

    @Test(expected = StreamsException.class)
    public void shouldThrowOnNegativeTimestamp() {
        final List<ConsumerRecord<byte[], byte[]>> records = Collections.singletonList(
                new ConsumerRecord<>("topic", 1, 1, -1L, TimestampType.CREATE_TIME, 0L, 0, 0, recordKey, recordValue));

        final RecordQueue queue = new RecordQueue(new TopicPartition(topics[0], 1),
                                                          new MockSourceNode<>(topics, intDeserializer, intDeserializer),
-                                                          new FailOnInvalidTimestamp());
+                                                          new FailOnInvalidTimestamp(), new LogAndContinueExceptionHandler(), null);


Nit: please one argument per line.

mjsax · 2017-06-24T23:01:36Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

@@ -158,7 +189,7 @@ public void shouldDropOnNegativeTimestamp() {

        final RecordQueue queue = new RecordQueue(new TopicPartition(topics[0], 1),
                                                  new MockSourceNode<>(topics, intDeserializer, intDeserializer),
-                                                  new LogAndSkipOnInvalidTimestamp());
+                                                  new LogAndSkipOnInvalidTimestamp(), new LogAndContinueExceptionHandler(), null);


mjsax · 2017-06-24T23:01:49Z

streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java

@@ -205,7 +206,7 @@ public ProcessorTopologyTestDriver(final StreamsConfig config,
            final GlobalStateManagerImpl stateManager = new GlobalStateManagerImpl(globalTopology, globalConsumer, stateDirectory);
            globalStateTask = new GlobalStateUpdateTask(globalTopology,
                                                        new GlobalProcessorContextImpl(config, stateManager, streamsMetrics, cache),
-                                                        stateManager
+                                                        stateManager, new LogAndContinueExceptionHandler()


Guess you missed this one :)

…zation-exceptions

asfgit · 2017-06-26T14:19:46Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5705/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-06-26T14:22:24Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5691/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-06-26T16:01:35Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5707/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-06-26T16:10:15Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5693/
Test PASSed (JDK 8 and Scala 2.12).

enothereska · 2017-07-03T08:06:16Z

@dguy @guozhangwang any further comments? Thanks.

dguy

Thanks @enothereska - i've left a few comments

dguy · 2017-07-03T10:06:57Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+                                                 final Exception exception) {
+
+        log.warn("Deserialization exception {}. Processor context is {} and record is {}",
+            exception.toString(), context, record);


do we want the exception stacktrace?

dguy · 2017-07-03T10:07:22Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java

+                                                 final ConsumerRecord<byte[], byte[]> record,
+                                                 final Exception exception) {
+
+        log.warn("Deserialization exception {}. Processor context is {} and record is {}",


dguy · 2017-07-03T10:11:08Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        } catch (Exception e) {
+            DeserializationExceptionHandler.DeserializationHandlerResponse response =
+                deserializationExceptionHandler.handle(processorContext, rawRecord, e);
+            if (response.id == DeserializationExceptionHandler.DeserializationHandlerResponse.FAIL.id) {


Why not just if(response == DeserializationExceptionHandler.DeserializationHandlerResponse.FAIL) ?

dguy · 2017-07-03T10:11:16Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        try {
+            return deserialize(rawRecord);
+        } catch (Exception e) {
+            DeserializationExceptionHandler.DeserializationHandlerResponse response =


nit: final

dguy · 2017-07-03T10:11:49Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+        // catch and process if we have a deserialization handler
+        try {
+            return deserialize(rawRecord);
+        } catch (Exception e) {


Should we add a metric for records skipped due to deserialization errors?

Yes good idea. I'll piggyback on this KIP.

dguy · 2017-07-03T10:14:14Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateTaskTest.java

+                                  final byte[] key,
+                                  final byte[] recordValue,
+                                  boolean failExpected) {
+        ConsumerRecord record = new ConsumerRecord<>("t2", 1, 1,


nit: final

dguy · 2017-07-03T10:16:33Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java

@@ -140,14 +142,45 @@ public void shouldThrowStreamsExceptionWhenValueDeserializationFails() throws Ex
        queue.addRawRecords(records);
    }

+    @Test
+    public void shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler() throws Exception {
+        RecordQueue queue = new RecordQueue(new TopicPartition(topics[0], 1),


maybe extract to a field as it is also used in the test below

dguy · 2017-07-03T10:20:05Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+    public ConsumerRecord<Object, Object> tryDeserialize(final ProcessorContext processorContext,
+                                                         ConsumerRecord<byte[], byte[]> rawRecord) {
+
+        if (deserializationExceptionHandler == null) {


Can we assume and/or ensure that this is non-null? Then we won't need this check here and can just have a single path through this method. I think we default to LogAndFail... if nothing is set - right?

…zation-exceptions

asfgit · 2017-07-04T09:15:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5887/
Test PASSed (JDK 7 and Scala 2.11).

enothereska · 2017-07-04T09:25:40Z

Thanks @dguy

asfgit · 2017-07-04T10:25:07Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5889/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-04T10:52:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5874/
Test PASSed (JDK 8 and Scala 2.12).

dguy

Thanks @enothereska. I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state that @guozhangwang suggested

dguy · 2017-07-04T13:59:43Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+    public DeserializationHandlerResponse handle(final ProcessorContext context,
+                                                 final ConsumerRecord<byte[], byte[]> record,
+                                                 final Exception exception) {
+        StringWriter sWriter = new StringWriter();


You can just pass the exception as the last parameter to i.e, log.warn(" p1: {} p2: {}", p1, p2, exception)

Also, just realised that ProcessorContextImpl doesn't have a toString. Perhaps we just need to log the taskId, topic, partition, and offset? i.e.,
log.warn("Exception caught during Deserialization, taskId: {}, topic:{}, partition:{}, offset:{}", context.taskId(), record.topic(), record.partition(), record.offset(), exception)

Same in the other exception handler

If I pass just the exception, it reverts to exception.toString(). It won't print the stack trace. As a typical NPE example:

with just exception passed: java.lang.NullPointerException

with stack trace:```
java.lang.NullPointerException
at org.rocksdb.RocksDB.get(RocksDB.java:791)
at org.apache.kafka.streams.state.internals.RocksDBStore.getInternal(RocksDBStore.java:235)
at org.apache.kafka.streams.state.internals.RocksDBStore.get(RocksDBStore.java:219)
at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.delete(ChangeLoggingKeyValueBytesStore.java:80)
at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueStore.delete(ChangeLoggingKeyValueStore.java:96)

Shall we add a toString to ProcessorContextImpl? That sounds useful.

@enothereska

If I pass just the exception, it reverts to exception.toString(). It won't print the stack trace.

My guess is that you are adding a {} for the exception, in which case it probably will just print the string. You don't need to add it. For example, from another one we have:
log.warn("{} Failed offset commits {}: ", logPrefix, consumedOffsetsAndMetadata, e)

Having toString on ProcessorContextImpl might be useful, but it also might be too much information for this particular log message

enothereska · 2017-07-04T14:51:31Z

I've left one comment. I think the only other thing remaining is whether or not we should implement the pattern for setting state.

Thanks @dguy I'm not sure what this refers to. I looked back but I don't see @guozhangwang 's comment. Could you elaborate? Thanks.

dguy · 2017-07-04T15:00:12Z

@enothereska doh! I've got my PRs crossed! ignore me

asfgit · 2017-07-05T10:00:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5899/
Test PASSed (JDK 7 and Scala 2.11).

dguy

1 more comment otherwise LGTM. @mjsax can you please make another pass over it?

dguy · 2017-07-05T10:05:02Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordQueue.java

@@ -80,8 +88,14 @@ public TopicPartition partition() {
     * @return the size of this queue
     */
    public int addRawRecords(Iterable<ConsumerRecord<byte[], byte[]>> rawRecords) {
+        ConsumerRecord<Object, Object> record = null;


I don't see any reason for this variable to not be declared inside the loop as it was previously

asfgit · 2017-07-05T10:24:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5884/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-07-05T14:02:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5903/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-05T14:25:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5888/
Test PASSed (JDK 8 and Scala 2.12).

mjsax

Some comments.

mjsax · 2017-07-05T18:44:26Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+import org.apache.kafka.common.Configurable;
+import org.apache.kafka.streams.processor.ProcessorContext;
+
+public interface DeserializationExceptionHandler extends Configurable {


We should have proper class JavaDocs for all public interfaces.

mjsax · 2017-07-05T18:44:55Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+
+public interface DeserializationExceptionHandler extends Configurable {
+    /**
+     * Inspect a record and the exception received


Nit: . missing at the end

mjsax · 2017-07-05T18:46:47Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+        public final short id;
+
+        DeserializationHandlerResponse(int id, String name) {
+            this.id = (short) id;


iIf we go with short, the parameter should be short, too?

mjsax · 2017-07-05T18:48:11Z

streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java

+        /** the permanent and immutable id of an API--this can't change ever */
+        public final short id;
+
+        DeserializationHandlerResponse(int id, String name) {


nit: add final

mjsax · 2017-07-05T18:48:53Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java

+                                                 final Exception exception) {
+
+        log.warn("Exception caught during Deserialization, " +
+                "taskId: {}, topic:{}, partition:{}, offset:{}",


Nit: some whitespaces are missing

mjsax · 2017-07-05T18:49:35Z

streams/src/main/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java

+                                                 final Exception exception) {
+
+        log.warn("Exception caught during Deserialization, " +
+                        "taskId: {}, topic:{}, partition:{}, offset:{}",


Nit: fix indention and some whitespaces are missing

mjsax · 2017-07-05T20:14:02Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java

@@ -63,7 +56,7 @@ public GlobalStateUpdateTask(final ProcessorTopology topology,
        for (final String storeName : storeNames) {
            final String sourceTopic = storeNameToTopic.get(storeName);
            final SourceNode source = topology.source(sourceTopic);
-            deserializers.put(sourceTopic, new SourceNodeAndDeserializer(source, new SourceNodeRecordDeserializer(source)));
+            deserializers.put(sourceTopic, new SourceNodeRecordDeserializer(source, this.deserializationExceptionHandler));


Nit: remove this

mjsax · 2017-07-05T20:19:41Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+            final DeserializationExceptionHandler.DeserializationHandlerResponse response =
+                deserializationExceptionHandler.handle(processorContext, rawRecord, e);
+            if (response == DeserializationExceptionHandler.DeserializationHandlerResponse.FAIL) {
+                throw e;


Should we wrap this with a StreamsException? This way, we could add a detailed error message and explain in the msg text that uses can set a different exception handler etc (similar to timestamp extractor?) -- so users don't need to ask at the mailing list :)

mjsax · 2017-07-05T20:23:05Z

streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java

@@ -205,7 +206,7 @@ public ProcessorTopologyTestDriver(final StreamsConfig config,
            final GlobalStateManagerImpl stateManager = new GlobalStateManagerImpl(globalTopology, globalConsumer, stateDirectory);
            globalStateTask = new GlobalStateUpdateTask(globalTopology,
                                                        new GlobalProcessorContextImpl(config, stateManager, streamsMetrics, cache),
-                                                        stateManager
+                                                        stateManager, new LogAndContinueExceptionHandler()


Guess you missed this one :)

mjsax · 2017-07-05T20:24:14Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java

@@ -186,6 +187,8 @@ public NodeMetrics(StreamsMetrics metrics, String name, String sensorNamePrefix)
            this.nodeCreationSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "create", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.nodeDestructionSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "destroy", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.sourceNodeForwardSensor = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "forward", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+            this.sourceNodeSkippedDueToDeserializationError = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "skippedDueToDeserializationError", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+


Nit: remove empty line

asfgit · 2017-07-06T09:37:11Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5904/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-07-06T09:42:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5919/
Test PASSed (JDK 7 and Scala 2.11).

…zation-exceptions

asfgit · 2017-07-07T09:42:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5943/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-07-07T09:43:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5958/
Test FAILed (JDK 7 and Scala 2.11).

enothereska · 2017-07-07T09:44:37Z

Could not write standard input into: Gradle build daemon

enothereska · 2017-07-07T09:44:43Z

retest this please

asfgit · 2017-07-07T10:44:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5959/
Test FAILed (JDK 7 and Scala 2.11).

enothereska · 2017-07-07T10:46:04Z

org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied failed. There is already a JIRA for it https://issues.apache.org/jira/browse/KAFKA-5566.

asfgit · 2017-07-07T10:48:20Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5944/
Test FAILed (JDK 8 and Scala 2.12).

enothereska · 2017-07-07T11:01:24Z

@dguy I don't have anything else to add to this. Fixing the QueryableState in separate PR. Thanks. Also I just merged with trunk.

…zation-exceptions

asfgit · 2017-07-07T13:30:28Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5962/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-07T13:35:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5947/
Test PASSed (JDK 8 and Scala 2.12).

enothereska · 2017-07-10T08:43:55Z

@guozhangwang any more comments?

dguy · 2017-07-10T17:13:38Z

retest this please

asfgit · 2017-07-10T18:18:50Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6015/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-10T18:42:30Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6000/
Test PASSed (JDK 8 and Scala 2.12).

dguy

Thanks @enothereska, LGTM

guozhangwang

Left some comments on this PR. Sorry for the late reply.

guozhangwang · 2017-07-10T19:20:35Z

...src/main/java/org/apache/kafka/streams/processor/internals/SourceNodeRecordDeserializer.java

+                sourceNode.nodeMetrics.sourceNodeSkippedDueToDeserializationError.record();
+            }
+        }
+        return null;


We should return null in the else branch above, in case we add more options in the response that are beyond Fail or Continue.

guozhangwang · 2017-07-10T19:57:22Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java

@@ -186,6 +187,7 @@ public NodeMetrics(StreamsMetrics metrics, String name, String sensorNamePrefix)
            this.nodeCreationSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "create", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.nodeDestructionSensor = metrics.addLatencyAndThroughputSensor(scope, sensorNamePrefix + "." + name, "destroy", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
            this.sourceNodeForwardSensor = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "forward", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);
+            this.sourceNodeSkippedDueToDeserializationError = metrics.addThroughputSensor(scope, sensorNamePrefix + "." + name, "skippedDueToDeserializationError", Sensor.RecordingLevel.DEBUG, tagKey, tagValue);


As this PR gets in already, we probably need to rebase the metrics fix PR to remove the . + name as well for this new metric as well.

Also I'm wondering what's the right approach of this metric: we already have a thread-level skippedRecordsSensor that covers this as it records total fetched records - total enqueued records. And then we have this source-node level metrics that only tracks the skippedRecordsDuetoDeserError. On the other hand, on the thread-level we do not have any other throughput metrics except the skippedRecordsSensor.

So should we just have a higher-hierarchy sensor at the thread level for number of skipped records rates (for whatever reason), or do we want to have a lower source-node level sensor for skipped records rates of different causes? In either way I think the current hierarchy is a bit messed and should be fixed. WDYT @dguy @enothereska ?

Initial implementation

a9892b3

More tests

a2210a6

enothereska changed the title ~~KAFKA-5157: Options for handling corrupt data during deserialization [WiP]~~ KAFKA-5157: Options for handling corrupt data during deserialization Jun 23, 2017

mjsax requested changes Jun 24, 2017

View reviewed changes

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

5335587

…zation-exceptions

Matthias' comments

580223c

dguy reviewed Jul 3, 2017

View reviewed changes

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

ab1a585

…zation-exceptions

Damian's comments

72f3202

dguy reviewed Jul 4, 2017

View reviewed changes

Damian's comments

32c860c

dguy reviewed Jul 5, 2017

View reviewed changes

Tweak

fb2d82b

mjsax reviewed Jul 5, 2017

View reviewed changes

Matthias' comments

c4d0c67

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

4eff572

…zation-exceptions

Merge remote-tracking branch 'origin/trunk' into KAFKA-5157-deseriali…

1da2a1d

…zation-exceptions

dguy approved these changes Jul 10, 2017

View reviewed changes

asfgit closed this in a1f97c8 Jul 10, 2017

guozhangwang reviewed Jul 10, 2017

View reviewed changes

KAFKA-5157: Options for handling corrupt data during deserialization #3423

KAFKA-5157: Options for handling corrupt data during deserialization #3423

Uh oh!

Conversation

enothereska commented Jun 23, 2017

Uh oh!

enothereska commented Jun 23, 2017

Uh oh!

asfgit commented Jun 23, 2017

Uh oh!

asfgit commented Jun 23, 2017

Uh oh!

enothereska commented Jun 23, 2017

Uh oh!

asfgit commented Jun 23, 2017

Uh oh!

asfgit commented Jun 23, 2017

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asfgit commented Jun 26, 2017

Uh oh!

asfgit commented Jun 26, 2017

Uh oh!

asfgit commented Jun 26, 2017

Uh oh!

asfgit commented Jun 26, 2017

Uh oh!

enothereska commented Jul 3, 2017

Uh oh!

dguy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asfgit commented Jul 4, 2017

Uh oh!

enothereska commented Jul 4, 2017 •

edited

Loading