Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't compress small buffers #19

Open
cb1kenobi opened this issue Jul 14, 2017 · 3 comments
Open

Doesn't compress small buffers #19

cb1kenobi opened this issue Jul 14, 2017 · 3 comments

Comments

@cb1kenobi
Copy link

@cb1kenobi cb1kenobi commented Jul 14, 2017

$ node
> var brotli = require('brotli');
undefined
> brotli.compress(Buffer.from([255, 255, 255]))
null
> brotli.compress(Buffer.from([255, 255, 255, 255, 255, 255, 255, 255, 255]))
null
> brotli.compress(Buffer.from([255, 255, 255, 255, 255, 255, 255, 255, 255, 255]))
Uint8Array [ 27, 9, 0, 0, 38, 254, 227, 145, 64, 0 ]

I need to have 10 bytes or more in order to compress the buffer. I was unable to find any documentation regarding this as a limitation of the Brotli algorithm, so I assume it's a bug.

As a workaround, I have to pad my data just to get it to compress.

@mynameiswhm
Copy link

@mynameiswhm mynameiswhm commented Aug 24, 2017

I have similar issue, but in my case minimum buffer size is 91:

$ node
> var brotli = require('brotli');
undefined
> // 90-bytes buffer:
> brotli.compress(Buffer.from('webpackJsonp([0],{"2twT":function(o,n){console.log("456                     ")}},["2twT"])'))
null
> // 91 bytes buffer - add another space:
> brotli.compress(Buffer.from('webpackJsonp([0],{"2twT":function(o,n){console.log("456                      ")}},["2twT"])'))
Uint8Array [ 27, 90, 0, 0, 60, 20, 111, 76, 146... ]

(But if you pad string from the right, you'll need 6 extra spaces, for example)

Btw, iltorb handles this cases just fine:

$ node
> var iltorb = require('iltorb')
undefined
> iltorb.compress(Buffer.from([255, 255, 255]), console.log)
undefined
> null <Buffer 0b 01 80 ff ff ff 03>
> iltorb.compress(Buffer.from([255, 255, 255, 255, 255, 255, 255, 255, 255, 255]), console.log)
undefined
> null <Buffer 1b 09 00 00 26 fe e3 91 40 00>
> iltorb.compress(Buffer.from('webpackJsonp([0],{"2twT":function(o,n){console.log("456                     ")}},["2twT"])'), console.log)
undefined
> null <Buffer 1b 59 00 00 3c 14 6f 4c 92 dc 0d 71 8d a9 b1 08 54 5f 3d 16 35 82 60 7d f8 e5 dd 5b 4a 8a c0 cc 95 0c 4d d7 e8 74 d1 69 91 ad b7 3d c9 92 d2 97 6e 67 ... >

@maxfie1d
Copy link

@maxfie1d maxfie1d commented Nov 11, 2019

Any update?

@tobiasmuehl
Copy link

@tobiasmuehl tobiasmuehl commented Mar 2, 2021

I also ran into this issue. These days the zlib module which is part of NodeJS has a brotli implementation that does not suffer from this issue.

Sample

import * as zlib from 'zlib'; // ships with node, no need to npm install

async function brotliCompressText(data: string) {
	const buffer = Buffer.from(data, 'utf8') as ArrayBuffer

	return await promisify(zlib.brotliCompress)(buffer, {
		params: {
			[zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
		}
	})
}

function brotliDecompress(buffer: Buffer) {
	return (await promisify(zlib.brotliDecompress)(buffer, {
		params: {
			[zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
		}
	})).toString()
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants