Skip to content

astalwick/streamfuz

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

StreamFuz - Simple Node.js Stream Combiner/Merger

StreamFuz is a stream utility that will take a list of smaller streams and merge them into a single, large stream. It fully supports all Node.js Streams2 functionality, including piping and back-pressure.

Installation

npm install streamfuz

Use

StreamFuz is Readable stream (actually, technically, it's a PassThrough right now, but nevermind). That means that you can pipe the output of StreamFuz to any other Writable stream, or you can just read() from it directly.

StreamFuz takes all of the typical Readable stream options on creation, in addition to the following:

  • parallel: Number of parallel part streams to be reading from simultaneously. If you're using StreamFuz to combine resources that are not local, it is useful (and faster) to parallelize the work. Defaults to 1 (no parallelization).
  • bufferSize: Amount of buffering per part. If you're reading a few streams in parallel, you'll need to buffer at least a small amount (otherwise there's no point in parllelizing, right). Defaults to 16384.
  • streams: An ordered array of streams to fuz. (optional)

Once you've created your StreamFuz stream, you need to enqueue up the parts that are to be combined.

  • enqueue(stream): Adds a single stream to the queue of parts that are waiting to be merged in.
  • enqueue([array of streams]): Appends an ordered array of streams to the end of the queue of streams.

So, briefly, usage looks something like this:

var streams = [];
for(var i = 0; i < 10; i++)
  streams.push(fs.createReadStream('./read_stream_' + i + '.txt'));

var streamFuz = new StreamFuz({ parallel: 3, bufferSize: 50000 });
streamFuz.enqueue(streams);

var writeStream = fs.createWriteStream('./output_stream.txt');
streamFuz.pipe(writeStream);

Licence

MIT

About

Simple stream handler that accepts multiple small streams and merges them into one larger stream.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published