Skip to content
Tiny and fast async generator to read streams line by line
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmark
test
.gitignore
.travis.yml
LICENSE
README.md
index.js
package.json

README.md

Tiny and fast async generator to read streams line by line

Build Status

Node 10+ is required. This package provides very simple but feature-reach functionality to consume readable streams or files line by line. It's also quite fast (up to 30% faster than readline) and has no dependencies.

Features:

  • reading of files (by name)
  • reading of readable streams
  • generator mode via readlines
  • readable stream mode via LineStream
  • customizable separator (default is \n)
  • raw bytes mode if encoding option is null
  • configurable lookahead buffer size

Documentation

Have a look at the index.js, it's really tiny. Or check the usage example below.

Usage examples:

const { readlines, LineStream } = require('readlines-ng');

// As a generator
for await (const line of readlines(filename, { encoding: 'utf8' })) {
  console.log(line);
}

// As a readable stream
const reader = new LineStream(filename, {
  chunkSize: 256*1024,
  encoding: null,
});
for await (const line of reader) {
  console.log(line);
}

// Custom separator
const { PassThrough } = require('stream');
const datasource = new PassThrough();
datasource.write('qux42');
datasource.write('foobar42');
datasource.end();

const reader = new LineStream(filename, { sep: '42' });
for await (const line of reader) {
  console.log(line);
}

Tests

npm run test

Benchmark

Trivial wc -l impl using this package exposes the performance better than the standard readline module.

npm run bench </path/to/huge/file>  # Node 11+ is required

# File size ~2.8 GB
> Benchmark: lines counter (readline module)
> 5974719 lines found
> Done in 5.757290349 sec

> Benchmark: lines counter (async generator)
> 5974719 lines found
> Done in 4.707053160 sec

> Benchmark: lines counter (async generator UTF-8)
> 5974719 lines found
> Done in 3.937784544 sec

> Benchmark: lines counter (readable stream)
> 5974719 lines found
> Done in 4.984460949 sec

> Benchmark: lines counter (readable stream UTF-8)
> 5974719 lines found
> Done in 4.354594750 sec
You can’t perform that action at this time.