SPARK-2803: add Kafka stream feature in according to specified starting offset position to fetch messages #1602
Conversation
…sition to fetch messages
…sition to fetch messages
|
Can one of the admins verify this patch? |
|
@pengyanhong Can you please follow the usual process of creating a JIRA corresponding to this patch, and add the JIRA number to the title of PR? (See other PRs for reference) Also, particular to this patch, can you give some background behind this patch - what is the problem it is try to solve, what is the design of the solution? |
…ied starting offset position
pengyanhong
changed the title
…sition to fetch messages
…sition to fetch messages
…ied starting offset position
Conflicts: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaSimpleConsumer.scala external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaSimpleInputDStream.scala external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
|
Can one of the admins verify this patch? |
|
@pengyanhong This is a very neat feature to add in Kafka utils. However, right now we are focused on improving fault-tolerance and semantic guarantees of the existing stream. This is not in our roadmap as of now. However, we have now added a new way for contributors to showcase their contributions to the community and maintain it themselves - spark-packages.org . Please consider adding your feature here. Others have done it too, e.g. http://spark-packages.org/package/5 In the meantime, mind closing this PR? |
create Kafka input DStream to fetch batch messages from specified starting position