Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

Investigate consumer.id for better Spark Streaming + Kafka failure recovery #93

Closed
srowen opened this issue Nov 26, 2014 · 0 comments
Closed

Investigate consumer.id for better Spark Streaming + Kafka failure recovery #93

srowen opened this issue Nov 26, 2014 · 0 comments
Assignees
@srowen
Labels
BatchLayer enhancement LambdaTier SpeedLayer
Milestone
2.0.0-alpha-1

Comments

@srowen
Copy link
Member

srowen commented Nov 26, 2014

Spark 1.2.0 should no longer force consumers to always start from the beginning of a topic after recovering from failure. It may / should be possible to just use a consistent consumer.id so that the job picks up reading where it left off. This would probably be better semantically for the Batch and Speed Layer.
@srowen srowen self-assigned this Nov 26, 2014
@srowen srowen added this to the 2.0.0 milestone Nov 26, 2014
srowen added a commit to srowen/oryx that referenced this issue Jan 25, 2015
@srowen
Closes issue OryxProject#93 : Add ID, and use it to determine consume…
f3d6156 
…r.id, and use it when reading the input queue to read from where reads left off
@srowen srowen modified the milestones: 2.0.0-alpha-1, 2.0.0 Jan 25, 2015
@srowen srowen closed this as completed Jan 25, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@srowen srowen

Labels
BatchLayer enhancement LambdaTier SpeedLayer
Projects
None yet
Milestone
2.0.0-alpha-1
Development

No branches or pull requests
1 participant
@srowen