Kafka engine Polled batch of 0 messages

[simPod]

I have created a new Kafka engine, however when queried it doesn't fetch any messages

CREATE TABLE test_consumer (
    forDateTime UInt32,
    valueIn     String,
    valueOut    String,
    id2  Nullable(String)
)
    ENGINE = Kafka SETTINGS
        kafka_broker_list = 'kafka-1.domain.eu:9092,kafka-2.domain.eu:9092,kafka-3.domain.eu:9092',
        kafka_topic_list = 'my_topic',
        kafka_group_name = 'clickhouse.test.group1',
        kafka_format = 'JSONEachRow',
        kafka_row_delimiter = '\n',
        kafka_num_consumers = 3;

2019.02.28 00:59:18.739977 [ 304 ] {54a59bea-ed0a-4525-9820-c66be024fcc3} <Trace> StorageKafka (test_consumer): Setting brokers: kafka-1.domain.eu:9092,kafka-2.domain.eu:9092,kafka-3.domain.eu:9092
2019.02.28 00:59:18.740025 [ 304 ] {54a59bea-ed0a-4525-9820-c66be024fcc3} <Trace> StorageKafka (test_consumer): Setting Group ID: clickhouse.test.group1 Client ID: clickhouse
2019.02.28 00:59:18.740657 [ 304 ] {54a59bea-ed0a-4525-9820-c66be024fcc3} <Trace> StorageKafka (test_consumer): Setting brokers: kafka-1.domain.eu:9092,kafka-2.domain.eu:9092,kafka-3.domain.eu:9092
2019.02.28 00:59:18.740673 [ 304 ] {54a59bea-ed0a-4525-9820-c66be024fcc3} <Trace> StorageKafka (test_consumer): Setting Group ID: clickhouse.test.group1 Client ID: clickhouse
2019.02.28 00:59:18.741090 [ 304 ] {54a59bea-ed0a-4525-9820-c66be024fcc3} <Trace> StorageKafka (test_consumer): Setting brokers: kafka-1.domain.eu:9092,kafka-2.domain.eu:9092,kafka-3.domain.eu:9092
2019.02.28 00:59:18.741104 [ 304 ] {54a59bea-ed0a-4525-9820-c66be024fcc3} <Trace> StorageKafka (test_consumer): Setting Group ID: clickhouse.test.group1 Client ID: clickhouse
2019.02.28 00:59:28.071398 [ 111 ] {e1c5f859-c001-4ccd-b092-7ea43c43877e} <Debug> StorageKafka (test_consumer): Starting reading 1 streams
   Kafka
2019.02.28 00:59:28.571783 [ 111 ] {e1c5f859-c001-4ccd-b092-7ea43c43877e} <Trace> StorageKafka (test_consumer): Polled batch of 0 messages
2019.02.28 00:59:29.072022 [ 111 ] {e1c5f859-c001-4ccd-b092-7ea43c43877e} <Trace> StorageKafka (test_consumer): Polled batch of 0 messages
2019.02.28 00:59:29.572192 [ 111 ] {e1c5f859-c001-4ccd-b092-7ea43c43877e} <Trace> StorageKafka (test_consumer): Polled batch of 0 messages
2019.02.28 00:59:30.072391 [ 111 ] {e1c5f859-c001-4ccd-b092-7ea43c43877e} <Trace> StorageKafka (test_consumer): Polled batch of 0 messages
2019.02.28 00:59:30.572589 [ 111 ] {e1c5f859-c001-4ccd-b092-7ea43c43877e} <Trace> StorageKafka (test_consumer): Polled batch of 0 messages

I don't see anything else helpful in log.

I can see new messages flowing to the topic but clickhouse doesn't consume anything 🤔 What might be the reason?

[josepowera]

Check if it is related to #4442

[simPod]

I have not encountered such error yet. Only that "Polled batch of 0 messages"

[simPod]

And yes, I have found in error log, that I have incorrect column types. Defined. Then I saw

2019.02.28 11:17:20.473187 [ 777 ] {578bc0ef-be9f-4cf8-90e2-0fb1e4b95081} HTTPHandler: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected { before: \0: (at row 2)

But adding kafka_row_delimiter = '\n', helped. Thanks