Stream 'readable' event oddness

[vsemozhetbyt]

• Version: 6.2.2
• Platform: Windows 7 x64

The 'readable' event is emitted when there is data available to be read from the stream... The 'readable' event will also be emitted once the end of the stream data has been reached but before the 'end' event is emitted.

Why in this script (saved in UTF-8 with BOM) the handler is called no more and no less than twice?

/******************************************************************************/
'use strict';
/******************************************************************************/
const fs = require('fs');

const readStream = fs.createReadStream(__filename, { encoding: 'utf8' });

readStream.on('readable', () => {
  console.log(readStream.read(3));
});

The output:

 /*
***

[vsemozhetbyt]

Call stacks for both event firing. Is the second emitting taken into account by the documentation?

[addaleax]

I think the end of stream data part of The 'readable' event will also be emitted once the end of the stream data has been reached but before the 'end' event is emitted. line refers to the onEofChunk, so yeah, I think the docs are correct.

I don’t know the specific reasons for this, though.

[vsemozhetbyt]

@addaleax Does onEofChunk means end of Chunk or EOF Chunk? If the first, is end of chunk (in the function) == end of the stream data (in the doc)? It seems not obvious. If the second, why does it fire if I does not read all the data, only some bytes? Is data read to the end anyway on the exit?

[addaleax]

onEofChunk means EOF, i.e. after all data from the source has been read into the Readable.

I’m not entirely sure I understand the question, but maybe the source of confusion is the fact that Readable streams actually buffer data to a certain degree, to have more data available for reading when the stream becomes readable.

The size of that buffer is configurable through the highWaterMark option, and if you set that low enough, the stream will not read then entire file:

/******************************************************************************/
'use strict';
/******************************************************************************/
const fs = require('fs');

const readStream = fs.createReadStream(__filename, {
  encoding: 'utf8',
  highWaterMark: 5
});

readStream.on('readable', () => {
  console.log(readStream.read(3));
});

Here the 'readable' listener only fires once, because only 5 bytes are read from the source into the stream’s internal buffer, of which only 3 bytes are used. After the .read() call, there will be 2 more bytes in the streams internal buffer; the stream will remain paused until these have been extracted using .read(), too (e.g. try setting the 3 to 5 and see what happens).

[vsemozhetbyt]

O, I think I see now. So once the end of the stream data has been reached also means including being exhausted by internal buffer?

[addaleax]

It is rather fired “as soon as it is clear that no more data will be added to the internal buffer, because the data source (i.e. the file itself) contains no more content”. After the last readable event (the onEofChunk one), the user can call .read() a few times to get the rest of the buffered data.

The event that is fired when all data has been read and the internal buffer has fully been cleared is the end event.

[vsemozhetbyt]

Phew) Got it. Thank you for all your time and explanations!

[addaleax]

Glad if it helped, and of course feel free to ask more questions here!

[saharshjain]

@addaleax I have a csv file and it contains 6K records and I am using streams to read and csv-parse package only.
I could see the readable event getting fired after few chunks of rows. Shall I understand that as soon as it starts reading the chunk and no more could be buffered it emits 'readable' event and when it start reading next chunk and end of chunk is reached it is fired again and so on.

So practically

no. of readable events will be equal to = start event + no. of chunks read at a time(in buffer) + end of file