Intermittent BUFFER_SHORTAGE errors

[SEAPUNK]

I have no idea what triggers this error, but every so often, when decoding (apparently large) msgpack data, the BUFFER_SHORTAGE error gets thrown.

Why does this happen? Is this a bug?

[kawanet]

BUFFER_SHORTAGE is fired when the given byte stream is not enough to represent a single complete msgpack object.

For example, a larger msgpack object may be divided into multiple TCP packets of 64KB each when transferred through the Internet. In that case, you have to concatenate the series of packets before decoding, or use the stream decoder.

[SEAPUNK]

That's strange, because the way I was decoding the data was by downloading all of it into memory (until the HTTP request ends), and then decoding it. Can incomplete binaries (e.g. some data trimmed a little bit off the end) cause something like this? Judging from your first sentence, it sounds like it.

[kawanet]

You mean the byte stream has no lack. Who is the encoder? msgpack-lite itself?

[SEAPUNK]

msgpack-lite encoded the message, yeah.

I'm not sure what you mean by "lack", but this is the scenario:

// server.js
const bigMsgpackBuffer = msgpack.encode(arrayWith100KElements)

httpServer.get('/thing', (req, res) => {
  res.end(bigMsgpackBuffer)
})

// client.js
request('http://server/thing', (response) => {
  console.log(response instanceof Buffer) // true
  const data = msgpack.decode(response.body) // intermittent Error('BUFFER_SHORTAGE')
})

Sometimes the decode at the end of the request works fine, sometimes the decode at the end of the request throws a BUFFER_SHORTAGE error.

[alax]

For anyone looking at this thread now, msgpack-lite seems to throw BUFFER_SHORTAGE when trying decode zero-length buffers or typed arrays. I don't know if that's the same issue, but simply checking for zero-length before decoding got rid of this issue for me.

[LoganDark]

msgpack-lite throws BUFFER_SHORTAGE when the buffer is too short. For example, when you have a string that's 8 characters long and the buffer is only 8 bytes long (one too short, the first is the string length), it will throw BUFFER_SHORTAGE.

[SEAPUNK]

I'm closing this issue, since it's very possible that the HTTP client "finished" a request before the transfer actually completed.

[LoganDark]

Yeah, this isn't an issue with msgpack-lite. msgpack-lite is doing what it's supposed to - failing when you give it bad data instead of giving you bad data back

[jvsteiner]

I ended up here trying to debug what ended up being an encoding problem. My one suggestion is that "BUFFER_SHORTAGE" is perhaps not a very good error message. I would recommend something like "BUFFER_TOO_SHORT" instead, since "BUFFER_SHORTAGE" make it sound like something is having trouble allocating a buffer, due to memory constraints, or something. You don't have a shortage of buffers in this case - you have a buffer that is too short.