Wasm Content Defined Chunking Evaluation

Wasm library for content-based slicing. Convenience wrapper on existing rolling hash implementations in rust, such as provided by fastcdc, asuran chunker, etc.

Build requirements

• Rust toolchain
• Wasm-pack

Test

wasm-pack test --headless --firefox

Wasm build

wasm-pack build --out-dir pkg/webpack --out-name chunking --target bundler --scope dstanesc
wasm-pack build --out-dir pkg/node --out-name chunking --target nodejs --scope dstanesc

Packing for local usage

npm pack pkg/node/
npm pack pkg/webpack/

Install for NodeJS

npm install @dstanesc/wasm-chunking-node-eval

Install for Webpack bundling

npm install @dstanesc/wasm-chunking-webpack-eval

Usage

// import {compute_chunks_buzhash, compute_chunks_fastcdc} from "@dstanesc/wasm-chunking-webpack-eval";
import {compute_chunks_buzhash, compute_chunks_fastcdc} from "@dstanesc/wasm-chunking-node-eval";

const buf = ...
// mask 0b11111111111111
const offsets_buz = compute_chunks_buzhash(buf, 15).values(); 
// chunk sizes: min 16 KiB, avg 32 KiB, max 64 KiB
const offsets_fast = compute_chunks_fastcdc(buf, 16384, 32768, 65536).values();   

Licenses

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Fastcdc is distributed under the terms of the MIT license

Asuran Chunker is distributed under the terms of the BSD 2 Clause + Patent License.