Skip to content

Commit

Permalink
Initial import; this is just a range coder w/ some tests.
Browse files Browse the repository at this point in the history
  • Loading branch information
cscott committed Mar 8, 2013
0 parents commit c55af7b
Show file tree
Hide file tree
Showing 9 changed files with 659 additions and 0 deletions.
2 changes: 2 additions & 0 deletions .gitignore
@@ -0,0 +1,2 @@
node_modules
*~
114 changes: 114 additions & 0 deletions README.md
@@ -0,0 +1,114 @@
# dmcjs

`dmcjs` is a fast pure JavaScript implementation of Dynamic Markov Chain
compression/decompression. It is written by C. Scott Ananian.
The Range Coder used is a port of Michael Schindler's C range coder,
found at http://www.compressconsult.com/rangecoder.

## How to install

```
npm install dmcjs
```
or
```
volo add cscott/dmcjs
```

This package uses
[Typed Arrays](https://developer.mozilla.org/en-US/docs/JavaScript/Typed_arrays)
and so requires node.js >= 0.5.5. Full browser compatibility table
is available at [caniuse.com](http://caniuse.com/typedarrays); briefly:
IE 10, Firefox 4, Chrome 7, or Safari 5.1.

## Testing

```
npm install
npm test
```

## Usage

There is a binary available in bin:
```
$ bin/dmcjs --help
$ echo "Test me" | bin/dmcjs -z > test.dmc
$ bin/dmcjs -d test.dmc
Test me
```

From JavaScript:
```
var dmcjs = require('dmcjs');
var data = new Buffer('Example data', 'utf8');
var compressed = dmcjs.compressFile(data);
var uncompressed = dmcjs.uncompressFile(compressed);
// convert from array back to string
var data2 = new Buffer(uncompressed).toString('utf8');
console.log(data2);
```
There is a streaming interface as well.

See the tests in the `tests/` directory for further usage examples.

## Documentation

`require('dmcjs')` returns a `dmcjs` object. It contains two main
methods. The first is a function accepting one, two or three parameters:

`dmcjs.compressFile = function(input, [output], [Number compressionLevel])`

The `input` argument can be a "stream" object (which must implement the
`readByte` method), or a `Uint8Array`, `Buffer`, or array.

If you omit the second argument, `compressFile` will return a JavaScript
array containing the byte values of the compressed data. If you pass
a second argument, it must be a "stream" object (which must implement the
`writeByte` method).

The third argument may be omitted, or a number between 1 and 9 indicating
a compression level (1 being largest/fastest compression and 9 being
smallest/slowest compression). The default is `1`. `6` is about twice
as slow but creates 10% smaller files.

The second exported method is a function accepting one or two parameters:

`dmcjs.decompressFile = function(input, [output])`

The `input` parameter is as above.

If you omit the second argument, `decompressFile` will return a
`Uint8Array`, `Buffer` or JavaScript array with the decompressed
data, depending on what your platform supports. For most modern
platforms (modern browsers, recent node.js releases) the returned
value will be a `Uint8Array`.

If you provide the second argument, it must be a "stream", implementing
the `writeByte` method.

## Related projects

* wikipedia article XXX
* range coder source XXX

## Other JavaScript compressors

* https://github.com/cscott/lzjb LZJB
* https://github.com/cscott/lzma-purejs LZMA
* https://github.com/cscott/seek-bzip Bzip2 (random-access decompression)

## License (GPLv2)

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.
127 changes: 127 additions & 0 deletions bin/dmcjs
@@ -0,0 +1,127 @@
#!/usr/bin/env node

var program = require('commander');
var dmcjs = require('../');
var fs = require('fs');

program
.version(dmcjs.version)
.usage('-d|-z [infile] [outfile]')
.option('-d, --decompress',
'Decompress stdin to stdout')
.option('-z, --compress',
'Compress stdin to stdout')
.option('-1', 'Fastest/largest compression')
.option('-2')
.option('-3')
.option('-4')
.option('-5')
.option('-6')
.option('-7')
.option('-8')
.option('-9', 'Slowest/smallest compression');
program.on('--help', function() {
console.log(' If <infile> is omitted, reads from stdin.');
console.log(' If <outfile> is omitted, writes to stdout.');
});
program.parse(process.argv);

if (!program.decompress) { program.compress = true; }

if (program.decompress && program.compress) {
console.error('Must specify either -d or -z.');
return;
}

var level = undefined;
for (var l=1; l<=9; l++) {
if (program[''+l]) {
if (level) {
console.error("Can't specify both -"+level+" and -"+l);
return;
}
level = l;
}
}
if (level && program.decompress) {
console.error('Compression level has no effect when decompressing.');
return;
}

var makeInStream = function(in_fd) {
var stat = fs.fstatSync(in_fd);
var stream = {
buffer: new Buffer(4096),
pos: 0,
end: 0,
_fillBuffer: function() {
this.end = fs.readSync(in_fd, this.buffer, 0, this.buffer.length);
this.pos = 0;
},
readByte: function() {
if (this.pos >= this.end) { this._fillBuffer(); }
if (this.pos < this.end) {
return this.buffer[this.pos++];
}
return -1;
},
read: function(buffer, bufOffset, length) {
if (this.pos >= this.end) { this._fillBuffer(); }
var bytesRead = 0;
while (bytesRead < length && this.pos < this.end) {
buffer[bufOffset++] = this.buffer[this.pos++];
bytesRead++;
}
return bytesRead;
}
};
if (stat.size) {
stream.size = stat.size;
}
return stream;
};

var makeOutStream = function(out_fd) {
return {
buffer: new Buffer(4096),
pos: 0,
flush: function() {
fs.writeSync(out_fd, this.buffer, 0, this.pos);
this.pos = 0;
},
writeByte: function(byte) {
if (this.pos >= this.buffer.length) { this.flush(); }
this.buffer[this.pos++] = byte;
}
};
};

var in_fd = 0, close_in = function(){};
var out_fd = 1, close_out = function(){};
if (program.args.length > 0) {
in_fd = fs.openSync(program.args.shift(), 'r');
close_in = function() { fs.closeSync(in_fd); };
}
if (program.args.length > 0) {
out_fd = fs.openSync(program.args.shift(), 'w');
close_out = function() { fs.closeSync(out_fd); };
}

var inStream = makeInStream(in_fd);
var outStream= makeOutStream(out_fd);

if (program.decompress) {
dmcjs.decompressFile(inStream, outStream);
outStream.flush();
close_in();
close_out();
return 0;
}
if (program.compress) {
dmcjs.compressFile(inStream, outStream, level);
outStream.flush();
close_in();
close_out();
return 0;
}
return 1;

0 comments on commit c55af7b

Please sign in to comment.