Not dumping all events

[bamos]

I'm not sure if this is a quirk of Kinesis or something I did wrong, but
sometimes this only prints the first 8 events rather than all 10.

[alexanderdean]

Interesting... Wonder if it could be something to do with record ordering somehow...

[bamos]

Hi @alexanderdean, using the Kinesis API directly in the commit above didn't fix the issue. I'm still seeing records disappear. What should I try next? Do you want me to write up a bug report for you to send to Amazon?

[bamos]

By the way, the horizon is part of the shard iterator and determines if the iterator should start at the beginning and pass through all shards, or at the end and only iterate through new shards.

Scalazon and this example uses TRIM_HORIZON, which starts the iterator at the beginning of the stream, but using LATEST also seems helpful, especially since we're interested in real-time processing.
The documentation reads like a real-time consumer application should continuously poll for new data, but I haven't tried anything myself.

Open a new issue if you want me to add real-time processing as an option to our consumer.

[alexanderdean]

Thanks @bamos - new ticket opened.

The fact that we are starting explicitly from the beginning of the stream and still missing records makes it sound like an Amazon issue to me... If you could write up the bug report in a public gist or similar, I will file it with AWS Support today.

Thanks @bamos!

[bamos]

Hi @alexanderdean, can you send https://gist.github.com/bamos/8221224?

[alexanderdean]

Thanks @bamos! Opened Support case: 146910911. You are on CC so should see updates as they come through.

[bamos]

Awesome, will be interested to see what they say!

[alexanderdean]

We shall see! Normally the bigger bug, the faster their response :-)

[bamos]

Hi @alexanderdean, I've merged everything into the feature/bamos-updates branch and everything's working smoothly for these issues. Please open (or reopen) if you want me to change anything on the Kinesis consumer.