Support the BPG image format

[oparoz]

From @oparoz on June 22, 2015 18:22

Introduction

BPG (Better Portable Graphics) is a new image format. Its purpose is to replace the JPEG image format when quality or file size is an issue. Its main advantages are:

• High compression ratio. Files are much smaller than JPEG for similar quality.
• Supported by most Web browsers with a small Javascript decoder (gzipped size: 56 KB).
• Based on a subset of the HEVC open video compression standard.
• Supports the same chroma formats as JPEG (grayscale, YCbCr 4:2:0, 4:2:2, 4:4:4) to reduce the losses during the conversion. An alpha channel is supported. The RGB, YCgCo and CMYK color spaces are also supported.
• Native support of 8 to 14 bits per channel for a higher dynamic range.
• Lossless compression is supported.
• Various metadata (such as EXIF, ICC profile, XMP) can be included.
• Animation support.

Source: http://bellard.org/bpg/

Converters

• Online converter: http://webencoder.libbpg.org/
• Windows batch converter http://www.romeolight.com/products/bpgconv/

Plan

After a preliminary analysis, I concluded that it would not be too difficult to add support to ownCloud via the Gallery app, but the use cases are quite limited at the moment, since mobile browsers would not be able to show the images.

• Add image/bpg to the list of core media types Adding BPG media type owncloud/core#17206
• Add the media type to the list of supported media types in the Gallery app
• Send the raw file, like SVG
• Send to decoding queue which converts to PNG via ImageMagick so that we can serve those on mobile
• Add the decoder to both the Gallery and Files app
• Decode images received via EventSource
• Resize canvas to the maximum size allowed via the oC config
• Load canvases in the slideshow
• Make support optional since it uses patented compression algorithms and loads an large library which needs to be compiled in real-time. This requires settings to be stored in the database if we want to avoid loading the scripts themselves

Avenues to explore

• Convert to PNGs and cache on server for instances which don't have access to ImageMagick
• How to catch decoding errors

Advantages

• Saves an enormous amount of space if you're using ownCloud as a sharing tool, not as a tool to store the originals
• Once all thumbnails are loaded, they could be used as-is for the slideshow, assuming you've got enough memory to keep all the large canvases in memory

Caveats

• No caching, means re-calculating everything at every session
• No thumbnails in the Files app until it supports the format, which probably won't happen any time soon since it doesn't support SVG today
• No support in mobile apps, but everybody can start lobbying for it to happen. Support the BPG image format owncloud/android#1026
• Cannot be used on mobile due to the lack of asm.js and limited amount of RAM available to the browser
• Can't be used as a replacement for PNG for previews since it would introduce a dependency with a 3rd party library

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Copied from original issue: owncloud/gallery#185

[oparoz]

From @jospoortvliet on June 23, 2015 11:50

aw this is really cool ;-)

[oparoz]

Except that almost nobody has collections in that format :D .
It also remains to be seen how much time the processing takes when you have photowalls with 50 BPG pictures.

I think this would be most useful as a format for thumbnails, but it won't happen before ownCloud uses PHP7 as the minimum version of PHP...

[oparoz]

Let's see what the GD team has to say about it :)
libgd/libgd#168

[oparoz]

From @vi on June 23, 2015 14:40

Can it use the JavaScript version as a fallback when libbpg is not available?

[oparoz]

I suppose you're talking about using it for thumbnails in core, but unfortunately, it wouldn't work.
The javascript converts BPG files to an HTML5 canvas object. What we need is something which converts PSD, RAW, JPG, etc. to BPG and only ImageMagick can do that today.

I could create an app which would only work with that format, but it would only work with Gallery and would mean having 2 copies of each thumbnail, one in PNG and one in BPG.

So currently, the only thing we can do is show BPG images and let users convert their collections using their favourite tools.

[oparoz]

From @vi on June 23, 2015 15:30

I don't want ownCloud to convert pictures, I just want to save storage and traffic by using BPGs which I provide.
Typically middle/low-quality photo in BPG already have size of a thumbnail (50-100 KB), so can be used both as a thumnail and a a full-sized photo (thumbnails being just rescaled on client instead of on server).
bpgdec.js should provide BPG decoding features to clients that does not have native BPG decoding.

bpgdec.js can turn <img>s with bpg to <canvas> with decoded data on the fly. Can't ownCloud just connect bpgdec.js and use plain <img> for bpgdec.js to decode them?

[oparoz]

Since there is not a single browser supporting BGP, the app would need to always decode files.
As mentioned in the OP, it would work like for SVGs. We would be sending the raw BPG file and let the browser do the resize, but it would only work in the Gallery app and its slideshow and the resized images would look different since no cropping would take place (unless someone writes a cropping method in JS).

[oparoz]

Also, since there is no Javascript source code for the decoder, it might not be possible to integrate it. It's an all-or-nothing approach and we would probably need to manually call the decoder.
I think I've found the source code.

[oparoz]

From @vi on June 23, 2015 16:28

Why no JavaScript source code for the decoder? It's probably this: https://github.com/mirrorer/libbpg/blob/master/post.js . It probably requires some Emscripten and friends to complile...

Here is example of using BPGDecoder from other JavaScript...

[oparoz]

Yes, that's the one. There is no need to compile it as it provides a load(url) method, so the approach is the same as the one used in the SO question. Build the canvas and paste the result on it.

[oparoz]

Preliminary analysis

I did some testing and it's a not easy to work with. We would need our own decoders since the methods available are too rigid and since we would need to add our own.
Also, it's clear that this can't be used on mobile, so shared albums would be empty... Not a great experience.

PNGs - 16 pictures - 36.1MB - GD resizing

116 requests - 3.9MB - 5.5s

BPG - 18 pictures - 1.5MB - Browser resizing

116 requests - 4.5MB - 8.5s

BPG - 18 pictures - 1.5MB - Hermite algo resizing

116 requests - 4.5MB - 10s

I did not enable asset pipe-lining in the tests, so you could probably shave off 4s to the loading time, which means that you get from 1.5s with PNGs to 6s with BPG with a resizing algorithm and that's with only 18 pictures...
BUT
All the large pictures for the slideshow are already loaded, so we would probably regain the wasted time, but not everybody is using the slideshow and people will probably have a mix of pictures.
so, what you save in disk space 36.1MB vs 1.5MB, you pay in waiting time when navigating through albums.

[oparoz]

Things to test:

• Overhead when album contains just one BPG picture. Library would need to be loaded (first time we encounter a BPG picture) and the picture processed
• Overhead when 2nd album containing a BPG picture is loaded. That would be the processing time for a large image

[oparoz]

Now that BPG has evolved into a standard we should reconsider this.
Struktur has made a library we can use:
https://github.com/strukturag/libheif

[pereorga]

BPG is different than HEIF, and it is more complex. There is also AVIF, which is a simpler (subset?) version of HEIF, probably less problematic with patents.

BPG, HEIF, and AVIF have little or zero support in browsers. IMHO, broad support is more likely to come to AVIF in the mid-term, but that is not the current situation. I don't think BPG has future (e.g.: probably less than FLIF).

[vi]

BPG is different than HEIF

Different container, same codec

it is more complex

As far as I understand, bpg container is simpler than HEIF container (which is basically mp4). Features seem to match.

There is also AVIF, which is a simpler (subset?) version of HEIF

AVIF is based on AV1 video codec. I don't think it is subset of HEIF (based on HEVC codec). The main common part is usage of MP4-like container.

probably less problematic with patents

Probably.

I don't think BPG has future

Maybe it is correct, given "future" and "large vendor support" is very correlated. I'm thinking about a tool that losslessly converts bpg files to heic files...

[pereorga]

Different container, same codec

Not really. HEIF can contain images and image sequences coded in different formats.

As far as I understand, bpg container is simpler than HEIF container (which is basically mp4). Features seem to match.

Sorry, I wrote the opposite of what I wanted to say, HEIF can be more complex to implement.

AVIF is based on AV1 video codec

I think AVIF files are valid HEIF files that use the AV1 codec. Therefore, the format is based on both HEIF and AV1.

[vi]

I think AVIF files are valid HEIF files that use the AV1 codec. Therefore, the format is based on both HEIF and AV1.

Now I understand what is difference between HEIC and HEIF:

The two main filename extensions are .heif (for any codec) or .heic (for HEVC codec)
(enwiki)