Binary chunking that can be reassembled out-of-order.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dist
rollup
src
tests
.gitignore
.travis.yml
CHANGELOG.md
LICENSE-APACHE
LICENSE-MIT
README.md
RELEASING.md
chunked-dc.d.ts
karma.conf.js
package-lock.json
package.json
tests.html
tsconfig.json
tslint.json

README.md

Binary Chunking for WebRTC DataChannels

Travis branch Supported ES Standard npm Version npm Downloads License

This library allows you to split up large binary messages into multiple chunks of a certain size.

When converting data to chunks, a 9 byte header is prepended to each chunk. This allows you to send the chunks to the receiver in any order.

While the library was originally written for use with WebRTC DataChannels, it can also be used outside of that scope.

The full specification for the chunking format can be found here.

Installing

If you're writing a browser application, simply use the normal or minified ES5 distribution from the dist directory.

<script src="chunked-dc.es5.min.polyfill.js"></script>

If you have a build pipeline yourself, you may also want to use the ES2015 version instead. The ES5 polyfill version is considerably larger because it also contains ES5 polyfills.

Alternatively, simply install the library via npm:

npm install --save @saltyrtc/chunked-dc

All classes in the ES5 version are namespaced under chunkedDc:

  • chunkedDc.Chunker
  • chunkedDc.Unchunker

To build the distributions yourself, simply run npm install && npm run dist in the main directory.

Usage

Chunking

For each message that you want to split into chunks, pass it to a Chunker.

let messageId = 1337;
let message = Uint8Array.of(1, 2, 3, 4, 5, 6, 7, 8);
let chunkSize = 12; // Chunk size *including* 9 byte header
let chunker = new Chunker(messageId, message, chunkSize);

You can then process all chunks using the iterator/iterable protocol:

for (let chunk of chunker) {
    // Send chunk to peer
}

Alternatively you can also use the next() method directly:

while (chunker.hasNext) {
    let chunk = chunker.next().value;
    // Send chunk to peer
}

The example above will return 3 chunks (header prefix not shown): [1, 2, 3], [4, 5, 6], [7, 8].

Unchunking

This library works both if chunks are sent in ordered or unordered manner. Because ordering is not guaranteed, the Unchunker instance accepts chunks and stores them in an internal data structure. As soon as all chunks of a message have arrived, a listener will be notified. Repeated chunks with the same serial will be ignored.

Create the Unchunker instance:

let unchunker = new Unchunker();

Register a message listener:

unchunker.onMessage = (message: Uint8Array, context: any[]) => {
    // Do something with the received message
};

Finally, when new chunks arrive, simply add them to the Unchunker instance:

let chunk = ...; // ArrayBuffer
unchunker.add(chunk);

You may also pass some context object to the unchunker which will be stored together with the chunk. When the onMessage handler is notified, these context objects will be passed in as a list ordered by chunk serial.

Cleanup

Because the Unchunker instance needs to keep track of arrived chunks, it's possible that incomplete messages add up and use a lot of memory without ever being freed.

To avoid this, simply call the Unchunker.gc(maxAge: number) method regularly. It will remove all incomplete messages that haven't been updated for more than maxAge milliseconds.

Constants

This library exposes the following constants:

  • HEADER_LENGTH: The number of bytes in the chunk header

Format

The chunking format is described in the specification.

Type Declarations

If you use TypeScript, simply reference the chunked-dc.d.ts type declaration file to get type checking and autocompletion.

Testing

This library has an extensive test suite. To run it:

npm install
npm run rollup_tests

Then open tests.html in your browser to run the test suite.

Linting

To run linting checks:

npm run lint

You can also install a pre-push hook to do the linting:

echo -e '#!/bin/sh\nnpm run lint' > .git/hooks/pre-push
chmod +x .git/hooks/pre-push

License

Copyright (c) 2016-2018 Threema GmbH

Licensed under the Apache License, Version 2.0, <see LICENSE-APACHE file>
or the MIT license <see LICENSE-MIT file>, at your option. This file may not be
copied, modified, or distributed except according to those terms.