A generator based line reader. This node package will return the lines of a file as a generator when given file descriptor and the size of the file.
This is a fork of the original project with additional features and fixes. Just one remained for a short time, after the mainainer kindly merged the changes to the upstream, and then this fork can be archived:
- Allow passing the three parameters
bufferSize
,position
,maxLineLength
toreadlines
separately for a compatibility, or in an object like in the upstream project.
DEPRECATION: The following readlines
prototype will be deprecated and removed in the next major version:
readlines(fd, filesize, bufferSize?, position?, maxLineLength?)
Migrate to the following prototype, please, which aggregates the trailing arguments to an object:
readlines(fd, filesize, { bufferSize?, position?, maxLineLength? }?)
I created this project primarily for better flow control of reading lines in a file. Instead of using callbacks for reading lines within a file, this will use a generator which has some unique benefits.
- Because it is a generator, the lines of the file are not stored in a large array
- Code is synchronous, so you don't have to worry about the asynchronous effects of waiting for a callback
- Performant, our micro-benchmark demonstrates it can compete with callback-based line readers and in fact appears to be faster.
Also, there are no external depencies and the library was built using TypeScript.
npm install @prantlf/gen-readlines
import { fromFile } from 'gen-readlines';
async function readFile() {
for (let line of fromFile('./file.txt')) {
console.log(line.toString());
}
}
If you already have the file open and you know the filesize you can use gen-readlines
and it will
create a generator which will iterate through all the lines in that file.
import fs from 'fs';
import util from 'util';
import readlines from '@prantlf/gen-readlines';
const open = util.promisify(fs.open);
const fstat = util.promisify(fs.fstat);
async function readFile() {
const fd = await open('./file.txt');
const stat = await fstat(fd);
const fileSize = stat.size;
for (let line of readlines(fd, fileSize)) {
console.log(line.toString());
}
console.log('continue code execution, no callbacks!');
fs.closeSync(fd);
}
readlines
returns a generator object and calling next
will get the next
line as a buffer object:
var file = readlines(fd, stats.size);
var line = file.next();
console.log(line);
// { value: <Buffer 42 65 73 70 ... >, done: false }
Convert the buffer to a string:
line.toString();
// This is the first line of the file
You can limit the maximum line length. When the specified length is reached while reading a line, the buffer will be returned as a new line just like when a line break was encountered:
// If original lines are longer than 255 characters, an artificial line break
// will be enforced after each 255 characters reached on a single line. More
// then original lines will be returned by the generator then.
var file = readlines(fd, stats.size, { maxLineLength: 255 });
for (let line of readlines(fd, fileSize)) {
console.log(line.toString());
}
You can change the maximum line length for each generated line. If you do not
specify the maximum length, when you read the next line, the original maximum
line length passed to readlines
will be used:
// Lines will not be longer than 255 characters by default.
var file = readlines(fd, stats.size, { maxLineLength: 255 });
var line = file.next(); // 255 characters maximum
line = file.next(127); // 127 characters maximum
line = file.next(); // 255 characters maximum again
Note: The very first generation (call to the next
method) cannot accept an
alternative maximum line length. It will always use the default value passed to
readlines
. First the following calls to next
allow to specify alternative
values. This is caused by the nature of JavaScript generators, which obtain the
value from yield
first when when resuming the generation.
Also you can use the simplified version of readlines
:
import readlines from '@prantlf/gen-readlines';
for (let line of readlines.fromFile('./file.txt')) {
console.log(line.toString());
}
- node 6 or newer, to get access to generators and new methods of the
Buffer
object.
- fd {Number} The file descriptor
- filesize {Number} The size of the file in bytes
- bufferSize {Number} The size of the buffer in bytes, default: 64*1024
- position {Number} The position where to start reading the file in bytes, default: 0
- maxLineLength {Number} The length to stop reading at if no line break has been reached, default: Infinity
- filename {string} Name of input file
- bufferSize {Number} The size of the buffer in bytes, default: 64*1024
- maxLineLength {Number} The length to stop reading at if no line break has been reached, default: Infinity
We are using mocha
for unit testing
npm test
./perf
contains benchmarks of several libraries, including this one:
Reading lines using...
byline x 159 ops/sec ±1.71% (76 runs sampled)
line-reader x 158 ops/sec ±1.35% (75 runs sampled)
linebyline x 111 ops/sec ±1.14% (85 runs sampled)
readline x 244 ops/sec ±1.13% (81 runs sampled)
gen-readlines returning each line x 96.68 ops/sec ±1.08% (76 runs sampled)
gen-readlines skipping all lines x 143 ops/sec ±1.55% (83 runs sampled)