New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Idea: Brotli support in core #18964

Open
MylesBorins opened this Issue Feb 23, 2018 · 24 comments

Comments

Projects
None yet
@MylesBorins
Member

MylesBorins commented Feb 23, 2018

Taking a quick peek at "can i use" shows that brotli is now supported on Edge, Firefox, Chrome, Safari, and iOS Safari.

The iltorb package has 912k downloads a month.
The brotli npm package has 250k download a month

If people are open to this I would be willing to collaborate on a patch to add this functionality.

@addaleax

This comment has been minimized.

Member

addaleax commented Feb 23, 2018

Sounds like a good idea to me. I’m somewhat familiar with iltorbs code base, but the brotli API has also gotten to a point where it’s similar enough to zlib that we probably wouldn’t even have to write much code to integrate it into Node.

/cc @MayhemYDG

@devsnek

This comment has been minimized.

Member

devsnek commented Feb 23, 2018

+1

p.s. if we're adding more compression stuff in core perhaps we should put it under one builtin module name require('compression').zlib/brotli/whateverelsecomesnext

@MylesBorins

This comment has been minimized.

Member

MylesBorins commented Feb 23, 2018

to be honest this may be the point where we decide if we want to use a nodejs namespace... as there is already a compression lib with 9M downloads a month and as mentioned above the brotli lib has 250k downloads a month. That being said I think it is a separate conversation that can be worked through independently

@MayhemYDG

This comment has been minimized.

Contributor

MayhemYDG commented Feb 23, 2018

I think it makes sense. IMO Node, as a web server, should strive to support low-level modern web features, much like http2.

@silverwind

This comment has been minimized.

Contributor

silverwind commented Feb 26, 2018

Yes, please. Brotli usage in Node.js has been held back by the barrier of native modules, this would certainly allow for broader adoption.

@MylesBorins

This comment has been minimized.

Member

MylesBorins commented Mar 13, 2018

Pinging this to see if anyone is interested in taking this on as an action item

@vkurchatkin

This comment has been minimized.

Member

vkurchatkin commented Mar 13, 2018

It seems weird that "can I use" and download numbers are enough for adding brotli, but were never enough for adding websockets

@MylesBorins

This comment has been minimized.

Member

MylesBorins commented Mar 13, 2018

@vkurchatkin tbh I think that we should revisit adding websockets.

If we look at the original thread #1010 some of the people who originally were -1 changed their opinions. In fact we ship a partial implementation of ws already, and have an open EPS exploring adding it nodejs/node-eps#6

The original decision by the TC at the time was to rather implement lower level buffer method, but that was abandoned.

#1202

I thinks I'll open another issue to discuss

edit:

issue for reference #19308

@MatonAnthony

This comment has been minimized.

Contributor

MatonAnthony commented Mar 13, 2018

Hello,

Which kind of skillset would it require? Would someone be willing to mentor me if I want to try it?

@jasnell

This comment has been minimized.

Member

jasnell commented Mar 13, 2018

Quick question: why would this need to be added to core vs user land? Is there a notable perf improvement or impl improvement to be gained relative to the userland module?

@MylesBorins

This comment has been minimized.

Member

MylesBorins commented Mar 13, 2018

This originally came up in a conversation with @kristoferbaxter where they mentioned having issues with userland implementations, and a lack of support in core being a reason for companies / enterprises to not adopt the format despite performance improvements compared to zlib.

I'm paraphrasing and will let them chime in

@kristoferbaxter

This comment has been minimized.

kristoferbaxter commented Mar 13, 2018

Hi there.

There are a few userland implementations with differing drawbacks, but the most prominent are iltorb and wasm-brotli afaik.

iltorbhttps://github.com/MayhemYDG/iltorb
Uses prebuild to download a pre-compiled binary for your platform, if it exists. Otherwise, it will use node-gyp to build the module.

Pros: Works well for those who can leverage the pre-compiled binaries. Supports compression options.
Cons: Projects including it encounter issues when a pre-compiled binary is not available.

wasm-brotlihttps://github.com/dfrankland/wasm-brotli
WebAssembly compiled Brotli library.

Pros: No need for a pre-built binary.
Cons: Performance. Lacks compression options.


TLDR;
Since Brotli is quite well supported, can support runtime compression (using quality levels 4-6), and buildtime compression (using quality level 11), it is a prime candidate for use targeting modern browsers or native applications. Its inclusion in core would reduce friction to leverage it and hopefully make a meaningful difference in application performance for many usecases.

@jasnell

This comment has been minimized.

Member

jasnell commented Mar 13, 2018

Aside from the convenience factor around native modules, are there specific performance or other ease of implementation considerations?

@kristoferbaxter

This comment has been minimized.

kristoferbaxter commented Mar 13, 2018

Aside from the convenience factor around native modules

Convenience might be understating this. Every installation of iltorb is slower than other JS only modules, and has been problematic to install on some environments in my past experience. It's the main point for using something like wasm-brotli according to it's repo ("The awesome thing about wasm-brotli is that it does not need to compile or download any prebuilt binaries!").

Lowering the cost to try out brotli compression might allow a greater number of transmissions to be compressed using it versus gzip. For end users this can be a meaningful difference.

are there specific performance considerations?

The specific performance concerns mostly relate to usage of wasm-brotli versus iltorb. These performance concerns may not be significant enough to warrant the change, but metrics like this leave me concerned.

## payload size: 1024
iltorb (native compress) x 244 ops/sec ±0.94% (78 runs sampled)
wasm-brotli (rust wasm compress) x 5.21 ops/sec ±1.64% (29 runs sampled)
Fastest is iltorb (native compress)

are there specific implementation considerations?

The API of iltorb seems straight forward in my opinion (supports async, sync, streams and compression options) but there are likely others who can weigh in better.


Let's flip the question around a bit too, if common compression schemes like brotli should not be in core... should gzip implementations? Should http2? The larger question of what should be in core is hard to answer. I'm certainly not qualified to answer it, but am happy to share my experiences.

Edit: Added link for performance data. Sorry for updating without saying what was updated!

@jasnell

This comment has been minimized.

Member

jasnell commented Mar 13, 2018

Cool thank you for the details. Please don't take my questions as push back, just need to build the case for the addition

@kristoferbaxter

This comment has been minimized.

kristoferbaxter commented Mar 13, 2018

No worries, never came across that way to me!

Thanks for taking a look and hopefully this helps to explain a bit more why this might be a good idea.

@MatonAnthony

This comment has been minimized.

Contributor

MatonAnthony commented Mar 14, 2018

If I understand this properly, we could bind to the V8implementation of Brotli?

(I'm sorry if my question is stupid, all of this is quite new to me)

@hashseed

This comment has been minimized.

Member

hashseed commented Mar 14, 2018

V8 doesn't include Brotli. Chromium does.

But yeah would be interesting to see how wasm performs.

Also, hi @MayhemYDG!

@MatonAnthony

This comment has been minimized.

Contributor

MatonAnthony commented Mar 14, 2018

Oh ok, so it's writing our own binding to Node. Is there a standard way or an example I could read about this?

@hashseed

This comment has been minimized.

Member

hashseed commented Mar 14, 2018

Maybe take a look at how zlib binding is done?

@hashseed

This comment has been minimized.

Member

hashseed commented Mar 14, 2018

Also, there is already one. https://github.com/MayhemYDG/iltorb

@bnoordhuis

This comment has been minimized.

Member

bnoordhuis commented May 2, 2018

There's an open pull request now, #20458. I'm curious if anyone has usage numbers of brotli on the internet.

I've been tracking my own internet usage but it's almost exclusively *.google.com that sends brotli back. Guess that makes sense with it being a Google thing.

@edmorley

This comment has been minimized.

Contributor

edmorley commented May 2, 2018

I'm curious if anyone has usage numbers of brotli on the internet.

Using the BigQuery query mentioned here (see the BigQuery guide here), there were 33K domains found using Brotli. For comparison that linked discussion post found only 357 domains when run in November 2016.

I've exported the top 500 domains by frequency (it refused to export the full list) to:
https://docs.google.com/spreadsheets/d/1bGPXJNUsaeJNYYgqUZQjW_rOzveHkzNbGyVBNoLm4E0/edit?usp=sharing

(For anyone wanting to repeat the above, the query only uses 5.8GB of BigQuery analysis allowance, and the first 1TB per month is free.)

@bnoordhuis

This comment has been minimized.

Member

bnoordhuis commented May 3, 2018

For anyone else wondering, that means just shy of 7% of the top 500k websites serve up brotli. More than I'd expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment