Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

LZMA Encoding is not Binary #1

Open
yuguang opened this Issue · 10 comments

5 participants

@yuguang

LZMA-JS compress returns an int array, which isn't as space saving as the real compression algorithm. I suggest sending the binary data in an image, turn that into an int array, then decode. Otherwise, we are deceiving ourselves. There is a javascript library for reading binary data http://blog.vjeux.com/2011/javascript/jdataview-read-binary-file.html

@nmrugg nmrugg was assigned
@nmrugg
Owner

Yeah, right now, it is up to the user to convert the data into something useful. Personally, the binary data wasn't useful to me when I started this project, but since it appears that others are interested in LZMA compression, I've been thinking about ways of handling the data to be more useful to others. I was planning on using typed arrays. I'll take a look at jDataView.

@yuguang
@yuguang
@strk

@nmrugg .decompress doesn't expect an array, right? what' format should compressed data be sent ?

@strk

found: back to an array of integers...

@nmrugg
Owner

Currently, it handles the data just using normal JavaScript variables. So the "bytes" are really just integers, which is sub-optimal, but handling bytes and binary data in JavaScript isn't straightforward.

I planning to make it use something like Uint8Array or Node.js buffers.

Any suggestions or pull requests are greatly appreciated.

@heniotierra

Well, after much effort trying to use jDataview in my project, I gave it up, and wrote the code below to convert the (compressed data) string to an array of bytes. "_" is underscore, but that should work with any loop. Pretty obvious, but I thought someone else might find it useful. BTW, thank you so much, @nmrugg .

function stringOfByteArrayToArrayOfBytes(str){
var byteArray = str.split(",");
_.each(byteArray, function(byte, key){
byteArray[key] = parseInt(byte);
});
return byteArray;
}

@nmrugg
Owner

Just in case anyone is interested, I created a fundraiser on BountySource to help me have more time to work on this issue (among others).

@jvilk

While I realize this project hasn't been touched in a year, I have a solution to your binary problems. My project, BrowserFS, emulates Node's Buffer API completely faithfully in the browser, including all string interpretation functionality (e.g. binary to UTF-8, UCS, UTF-16, etc) (it passes Node's unit tests):

https://github.com/jvilk/BrowserFS/blob/master/src/core/buffer.ts

If you are in a browser with TypedArray support, you can wrap an ArrayBuffer by passing it to the Buffer constructor.

It's written in TypeScript, with compiles directly to JavaScript. And it's MIT licensed, so feel free to rip it out of my project and incorporate it into yours. Just credit me if you decide to do this. :smile:

One note: You cannot use BrowserFS's Buffer implementation with array notation (e.g. buff[0] will not work). Instead, explicitly use the readUint8() function.

@nmrugg
Owner

Looks like a good project. Thanks for the idea. I'm looking in to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.