STORM-563. Kafka Spout doesn't pick up from the beginning of the queue unless forceFromStart specified.

[harshach]

No description provided.

[harshach]

@ptgoetz @nathanmarz @revans2 Please take a look at the patch. I renamed forceFromStart to ignoreZkOffsets and users can configure where they want to start based on startOffsetTime .

[Parth-Brahmbhatt]

💯

[ptgoetz]

I would also document the actual values of EarliestTime() (-2) and LatestTime() (-1), and that it can also be set to a point in time (a la System.currentTimeMillis()).

My reasoning behind documenting the values (as opposed to the kafka API constants) is that the start offset time is likely to be specified via configuration (i.e. outside java code). Either that, or add spout constants that would get evaluated to EarliestTime()/LatestTime() if for some reason those values were ever changed in the Kafka API -- that seems like a less "leaky" solution.

[harshach]

@ptgoetz updated the doc as per your suggestion. Please take a look.

[harshach]

@ptgoetz addressed your comments. Can you please take a look at this.

[ptgoetz]

It wouldn't hurt to expand on what System.currentTimeMillis() means in that context (i.e. if you have a specific time stored in epoch format, you can start from there).

[ptgoetz]

That being said, I'm +1.

[Renkai]

If I set ignoreZkOffsets to true,will a spout recover from failure read from zk offsets or use startOffsetTime?
How do spout detect it is first started or recover from failure?

[harshach]

@Renkai ignoreZkOffsets is rename of forcefromStart. So if you set ignoreZkOffsets it wil ignore already set offsets in zookeeper and start from the startOffsetTime.
"How do spout detect it is first started or recover from failure?"
can you explain bit more on that.
Here ignoreZkOffsets works when you deployed a topology and later killed it but want to start from startOffset instead of offset stored in zookeeper.

[Renkai]

when ignoreZkOffsets set true and startOffsetTime = kafka.api.OffsetRequest.EarliestTime().
workers running -> topology shutdown by user and restart -> workers will read from earliest time again
workers running -> one of workers shutdown by accident and supervisor restart the worker -> what offset will the restarted worker read from?
@harshach

[harshach]

@Renkai In this case yes it won't read from zk offsets. Which is incorrect behavior. Can you file a jira on this.

[Renkai]

@harshach Issue created at https://issues.apache.org/jira/browse/STORM-1017

[harshach]

Thanks @Renkai