unhandled kinesis error event

[angleman]

Upgrading to kinesis@0.2.2 allows it to run on node@0.11.9; however, after a few minutes of pushing records to kinesis (3 to 30/second @ ~120bytes per push), I'm getting the following:

events.js:79
      throw er; // Unhandled 'error' event
            ^
Error
    at IncomingMessage.<anonymous> (/home/angleman/test/node_modules/kinesis/index.js:229:15)
    at IncomingMessage.EventEmitter.emit (events.js:120:20)
    at _stream_readable.js:896:16
    at process._tickCallback (node.js:598:11)

Issue occurs under node@0.11.9 and node@0.10.25

Thoughts?

[mhart]

I'm sorry, I'm not able to reproduce this.

Can you please post some code that reproduces this?

[angleman]

I built a test case, see put.js in issue #5. It reproduced the error. Then I changed out the base64 encode library as I was getting occasional garbage data out of get.js, also in #5, and since then I have not seen this particular problem show up again.

Other than needing to kinesisStream.write(base64_encode(stringData)) instead of kinesisStream.write(stringData), I suspect this issue was caused by the faulty base64 library. Using base64 from github pkrumins/node-base64, installed by npm install base64 seems to work 😸

The only remaining thought is to consider detecting the type of data being passed to kinesisStream.write() and base64 encoding it if needed.

Thanks again, @mhart, for looking into this 😄

[mhart]

base64 encoding/decoding is built into node.js :-) No need for third party libraries!

new Buffer('hello', 'utf8').toString('base64')
'aGVsbG8='

[mhart]

Also, streams use Buffers by default - you could consider using the objectMode option if you want objects to stream out.

[angleman]

Re: base64 built in, sweet! 👍

Re: objectMode. In get.js in #5, I use objectMode and it seems the data.data attribute was still a buffer. I discovered this when after kinesisStream.on('data', function(data) { console.log(data) } showed data.data to be a buffer when I was building the test.

[angleman]

What do you think about auto base64 encoding data passed into kinesisStream.write(data)?

[mhart]

If you're using objectMode for the read stream, then data.data will be a Buffer, yes (because Kinesis uses binary data). But that doesn't mean you need to convert to base64 if you don't want to. You can do data.data.toString('utf8') or whatever you'd like with the Buffer.

The write stream takes Buffers - but again, you can construct these however you like (eg, new Buffer('hello', 'utf8') - you shouldn't need base64 anywhere.

[mhart]

(perhaps you're confusing Buffers with base64? Buffers don't have to use base64 - they're just a storage for arbitrary binary data)

[angleman]

Makes sense. n00b to streams. I'm guessing it was a side effect of a lower layer that allowed base64 to work. new Buffer('hello', 'utf8') and data.data.toString('utf8') works great!