[FLINK-5368] Log msg if kafka topic doesn't have any partitions #3036
Conversation
|
Looks like the buildjob ran out of memory on 13075.1: All others went fine. |
| @@ -208,13 +208,12 @@ public FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T> d | |||
| if (partitionsForTopic != null) { | |||
| partitions.addAll(convertToFlinkKafkaTopicPartition(partitionsForTopic)); | |||
| } | |||
| else{ | |||
| throw new RuntimeException("Unable to retrieve any partitions for the requested topics " + topic); | |||
There was a problem hiding this comment.
I think throwing an exception here is not a good idea.
If you have 10 topics, and only one of them doesn't have any partitions, the consumer won't start anymore.
Also the exception message is not accurate. It says Flink can not retrieve ANY partitions for the requested topicS. However, its just the partitions of one topic.
I suggest to only log a INFO-level message in that case.
There was a problem hiding this comment.
Agree. I will change the exception message to INFO log message.
HungUnicorn
changed the title
| } | ||
| } | ||
|
|
||
| if (partitions.isEmpty()) { | ||
| throw new RuntimeException("Unable to retrieve any partitions for the requested topics " + topics); | ||
| } |
There was a problem hiding this comment.
I think we still want to fail if there are no partitions at all for ALL topics, correct?
This change here will remove that behaviour.
There was a problem hiding this comment.
To throw exception fits our case better than to log INFO message, but @rmetzger suggested that some cases might be fine and to log INFO is more general if I understood correctly.
There was a problem hiding this comment.
I think Robert was suggesting to log messages only for cases where some topics don't have partitions, so that the consumer doesn't just fail if for example only 1 out of 10 topics don't have partitions.
If ALL topics failed to return partitions in the end, we probably should still fail the consumer, like before.
|
is the issue still open |
|
+1 to merge. Thank you for the reminder @Rohithyeravothula. |
|
Merging this ... |
As a developer when reading data from many topics, I want Kafka consumer to show something if any topic is not available.
The motivation is we read many topics as list at one time, and sometimes we fail to recognize that one or two topics' names have been changed or deprecated, and Flink Kafka connector didn't show the error.