Pre-compress Files With Brotli And Fallback To Gzip

[philsward]

Description of the need

Formal request to add Brotli compression and make gzip the fallback compression

Background

Long story short, Brotli is a newer compression algorithm designed by Google around 2013 to replace gzip which was created in 1992. Brotli produces larger compression ratios over gzip, allowing for even smaller file sizes.

I won't go into detail about what Brotli is, how to get it going or why Backdrop should support it because this article does a fantastic job explaining it: https://csswizardry.com/2020/04/real-world-effectiveness-of-brotli/

Proposed solution

It looks like some work will need to be done to add Brotli as a compression within Backdrop itself, much like gzip. This is mostly over my head, but looks like the same code that runs gzip can be used to do the same with Brotli

Additional information

Should this be a contrib for now until it gains more traction? Maybe it will gain more traction if it is actually part of core.

[philsward]

Boo... I apparently never got it properly working... Looking into this on a recent server migration, I'm seeing it's accepted, just not served.

Ran across this stackexchange that might need to be implemented to make this work on Backdrop. Doing some testing to narrow it down.

https://drupal.stackexchange.com/questions/215639/how-to-modify-htaccess-to-serve-brotli-compressed-static-assets-over-gziped-one

[stpaultim]

@philsward - Not sure what problems you are having or how that might effect this issue. BUT, this looks like a pretty easy Pull Request to create. Given the ease of the issue (if that remains true), I recommend creating the Pull Request despite the lack of feedback, because once the issue has a Pull Request it's more likely to get attention and/or feedback. I would't spend too much time on a PR without some feedback, unless (like in this case) it's an easy PR to create.

Also, if it's worth it to you, please drop by one of our Thursday dev meetings to talk about the issue and get feedback (unless you get the feedback you need here in the issue queue).

[philsward]

Thanks @stpaultim

Yeah, doing some more digging, it look like I never had Brotli setup at all. Some of this is way over my head but digging in a bit it looks like Drupal7/Backdrop generate gz files through code and they will have to do the same with Brotli files as well. Heh... all this time I thought I was smart ha!

Yeah, this request is going to have to go a lot deeper than just adding a few chars to the .htaccess file.

[stpaultim]

Well then it might be best to hold off on a Pull Request until we know more and have some idea if there is interest in this issue. To be honest, this one is out of my area of expertise. I have no opinion at this time.

[earlyburg]

Virtual machines can be allocated with plenty of RAM and a powerful CPU to run as fast as possible, but I have measured network throughput in a certain cloud envrionment on a supposedly gigabyte network where data transfer was actually about 400mbps between virtual machine instances. This compression technology is probably related to the cloud paradigm. I'm mindful also that Alphabet is currently embroiled in antitrust litigation, which could affect our project in a negative way (speculating here about end of life for google tech).
https://www.wsj.com/articles/justice-department-to-file-long-awaited-antitrust-suit-against-google-11603195203

[klonos]

Unless I'm terribly wrong about this, support for various compression methods is provided by the operating system on the server + php extensions (unless natively supported by php core). There is nothing to be done on our end until there's better support for this (or any other for that matter) type of compression.

[philsward]

@klonos yeah, at first I thought this was purely an "on-the-fly" request which would simply be the addition of some lines in the .htaccess to make it happen.

What I have now learned is that there are two methods of implementation:

Dynamic (on-the-fly)

Static

In order to pull off dynamic, the .htaccess needs to be rewritten such that if Brotli is available, serve it first and fall back to gzip

I played around with this last night and couldn't get Brotli to work unless it was below the gzip lines. (Otherwise it always wanted to serve gzip). The problem with dynamic is that it is slower to compress and serve on the fly depending on compression level than it is to use pre-compressed gzip. Static is the best route.

In order to serve static, the same approach needs to be taken where a .br file is pre-compressed just like the .gz file, then served if available.

I'll agree that the hosting platform needs to support it, but it literally is as easy as enabling mod_brotli for Apache. There are some similar instructions for Nginx. VPS platforms this is easy. Shared hosting is another story and the reason for fallback.

As opposed to waiting for more people to enable it, then add it, my proposal is to add it and encourage people to use it.

I believe the hardest part of this discussion is how to pre-compress the files. I ran across a PHP library here on git for it, but I'm not sure if that would work.

Based on what you've said, it sounds like a contrib might be the better route until the idea gains more traction but I'm still on the mindset that "if it's available in core, more folks will start using it". Something like 96% of browsers support it, so now it's a matter of the applications using it.

[earlyburg]

Brotli is supported in the following web servers:

Apache HTTPD – through mod_brotli (for release after 2.4.26)
Nginx – ngx_brotli (provided by Google)
Node.js (through shrink-ray module
LightSpeed (since version 5.2)
Microsoft IIS (through IIS-brotli extension, for IIS 7.5 and above)

That being said, it does not come installed out of the box. Ubuntu has a package for aptitude, CentOS requires you to compile the extension, not sure about the rest.

In addition there may be some security concerns when used over HTTPS
https://security.stackexchange.com/questions/150351/brotli-compression-for-https

[philsward]

That being said, it does not come enabled out of the box.

Right. Until the applications start using it to require it enabled ootb, it will be a sysadmin choice to enable it.

Using Backdrop as an example, the support for it isn't built-in, therefore the practicality of using it is non-existent. "If backdrop, WordPress, Joomla, Drupal etc doesn't support it, why enable it?"

Instead, if these platforms supported it, Brotli would just work the moment it was turned on, pushing more people to ask for it on their hosting companies, further pushing for it to be enabled ootb.

[findlabnet]

further pushing for it to be enabled ootb.

Even mod_rewrite not come enabled out of the box, although most web applications prefer to use it. So?

[philsward]

Even mod_rewrite not come enabled out of the box, although most web applications prefer to use it. So?

I don't feel like that's a fair statement. How many shared hosting platforms don't enable mod_rewrite for their users?

I believe cPanel enables it by default as does plesk. If not, it is easy to enable on self-hosted platforms.

The enabled by default refers to what hosting companies turn on as a default hosting environment, not what Apache or Nginx enable by default.

[earlyburg]

further pushing for it to be enabled ootb.

Even mod_rewrite not come enabled out of the box, although most web applications prefer to use it. So?

For instance, with apache2 on Ubuntu, in /etc/apache2/mods_available you can find mod_rewrite module already installed and ready to be enabled, but if a sysadmin wanted to use brotli they would have to either compile it from source or use the package manager. I should have written "installed out of the box". I have fixed it above. Sorry for the confusion.

[philsward]

@earlyburg ah

Yeah, that does make it less appealing.

I guess at this point, the discussion is started.

Considering it has been around 7-8 years now, I think it's worth adopting for the long run.

The major benefit is reduced file size. In some circumstances, faster speeds due to the size reduction, though this isn't as apparent due to how TCP works from my understanding.

It does appear there is a bit of work to get this full scope accomplished, but for a quick pull request, I do suggest at least adding the E=no-brotli:1 for css and js double gzip. This workaround does need to be there to keep sites from breaking "if" Brotli is enabled on the server.

I see now I need to add that as a separate issue. This one can remain for future full Brotli support.

[klonos]

I understand the benefit of compression and all, but I'm unclear re what we are talking about here. What would core allow to be compressed? ...or is this about data transfer compression? If so, then this happens on the webserver layer. No?

Anyway, I would need to also know what scale of changes are required in core before I decide whether this is a good idea for core or better left for contrib. Perhaps if there was a working contrib module available, it would be able to provide some idea of the extend of changes required 🤷

[philsward]

@klonos

Found this quote which might help the confusion about Apache:

there's a difference between brotli in Apache and Brotli in PHP. One is to compress using brotli for the transport, and Brotli in PHP is to be able to handle brotli compression within a given application.

I believe I am mistaken with Apaches role in all of this. Sorry I'm not a superwiz at this stuff 😟

It looks like this moves to being an issue of PHP handling Brotli which there is an extension for it, just not as widely accepted.

It might be worth looking at the advagg module on Drupal. I believe they got Brotli working over there without anything special. Maybe they've bundled a binary that can compress on the fly where the server or PHP aren't actually necessary. I haven't had a chance to look into it myself.

[stpaultim]

I understand the benefit of compression and all, but I'm unclear re what we are talking about here. What would core allow to be compressed? ...or is this about data transfer compression? If so, then this happens on the webserver layer. No?

@klonos - At the time this issue was opened, I think the idea was that a minor change in the .htaccess file might make it easier for users to configure their server for this feature. I'm a little lost, but the issue is getting more complicated and it may very well turn out that this is not a core issue. But, I don't think it hurts to discuss it here until they figure that out.

It does seem like this MIGHT be better addressed in contrib. But, until there is a specific proposal it's hard to tell (I think folks are researching that now). Personally, I'm going to just wait and see where this discussion goes.

[klonos]

...until there is a specific proposal it's hard to tell (I think folks are researching that now). Personally, I'm going to just wait and see where this discussion goes.

On the same boat here 👍

[philsward]

I'll be honest, I'm more confused now than when I started.

What prompted this is the server where I do testing, didn't have redirect to https. When I switched over to https, none of the css would load. This led me remember that Brotli is enabled on the server and without E=no-brotli:1 for the double gzip .htaccess rules, the css / JS won't properly decode. Brotli by design, only works over https. Chrome for example, completely ignores it unless the request is https.

I thought Brotli was doing it's thing, but I have since realized it isn't doing what I expect and has led me down this rabbit hole.

I'll see if I can loosely figure out how advagg is pulling it off and point us in that direction.

In the meantime, if Brotli is enabled on a server, without the proper no-brotli in the rule it breaks css and JS which is a completely separate issue in and of itself.

This has turned out to be a lot more complicated than I originally thought and I know there are far more pressing issues to take care of right now for Backdrop. Maybe when the core folks have some free time, dig into Brotli a bit and we'll go from there?

@klonos you asked how core would handle this: simply pre-compress css and JS files for now, just like gzip. At least that's what I envision. If a static html cache (like boost) becomes a thing later, add it there as well. Basically replicate system.tar.inc for Brotli which may not be possible without preinstalled binaries or PHP extensions. This thought takes me back to advagg and figuring out how they handle it.

[philsward]

Added new issue to address the problem with Brotli compressing pre-compressed css and js files:

#4748

[philsward]

Alright, so Advagg is using the third party php extension:

(This is a grep looking for brotli within advagg files)

./advagg.admin.inc:480:    $form['global_container']['global']['obscure']['advagg_brotli']['#description'] .= ' ' . t('The <a href="@url">PHP brotli extension</a> is needed in order to enable this feature.', array('@url' => 'https://github.com/kjdev/php-ext-brotli'));
./advagg.admin.inc:484:        '@iis' => 'https://www.iis.net/downloads/community/2016/03/iis-brotli',

It looks like in order to properly support brotli compression, the PHP extension will need to be installed or figure out some sort of alternative that is easier to use globally.

[philsward]

Looks like cPanel installs the binary:

https://forums.cpanel.net/threads/brotli-extension-for-php.645765/

[lourdas]

Hey all,

What's the status of this issue? I understand that in order to have my front page served as compressed (gzip) in my recently upgraded from Drupal 7 site to Backdrop, I have to enable the page compression checkbox from the Performance administration page. However, I have enabled brotli compression for all the sites in my vps and I don't understand why this does not apply to Backdrop, unless this checkbox is active. If I enable it, the html content from the site is served as gzip compressed, not as brotli compressed.

I have another Drupal 7 site in the same vps. The page compression checkbox is not checked there and the html content of this site is served as brotli compressed. This is a bit puzzling.

I will examine the site's Apache vhost configuration just in case I missed something.

[stpaultim]

@lourdas

This issue seems to be stalled. Did you make any progress solving this for your site?