Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extra empty line outputted when CRLF straddles internal buffer #29

Open
adhooaa opened this issue May 17, 2019 · 1 comment
Open

Extra empty line outputted when CRLF straddles internal buffer #29

adhooaa opened this issue May 17, 2019 · 1 comment

Comments

@adhooaa
Copy link

adhooaa commented May 17, 2019

What I see:
When streaming a large file with CRLF (\r\n) line endings, occasionally an extra "empty" line is emitted by linereader.on('line', ...).

i.e. When "printing" the file/stream "line by line", an empty line appears in the output when there are no empty lines in the original file.

What I expect:
No extra blank lines are outputted.

Reproduction:
See attached test case - had to rename index.js and packagejson file to txt for github upload
the test will:

  • generates a file just bigger than 65536; which I suspect is the internal buffer size
    • also attached a sample data file, but it is easy to see with the test that there is nothing special about the data except for where the last \r\n lands.
  • the test then goes through the file line-by-lines checking for an erroneous empty line

Other Notes:

  • Could be the cause of this: Skips a line and makes a line at the end?? #26 (comment)
  • I didn't see an api to set/read the internal buffer size which would have helped writing a test and debugging Set internal buffer size? #4
  • I didn't see an api to return to me the end of lines Option to include line endings #11 - which would help me inspect the original file. (I would have just streamed the file back out verbatim and then diff their output and compare their hash). Especially when I don't control the original input file and could even have mixed-line-endings (so I can't presume to just add '\n' to the end of each emitted 'line')

index.js.txt

package.json.txt

data.txt

@Sanj718
Copy link

Sanj718 commented Jul 18, 2022

This is due to buffer size, i guess fs.createReadStream just cuts once buffer size limit reached. I've increased using highWaterMark: 256 * 1024 , and it worked. but its not a good solution, have you found it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants