Comparison with mozjpeg?

[magicgoose]

README says:

Guetzli-generated images are typically 20-30% smaller than images of equivalent quality generated by libjpeg

But what about mozjpeg? It existed long before this project, by the way — 1.0 release was in Mar 5, 2014, and it does similar job.

[robryk]

We will soon publish a more detailed comparison that takes mozjpeg and libjpeg into account.

Note that comparing two JPEG encoders is more complicated than it might at first appear, because one needs to compare quality of the compressed image. Obviously, we actually care about quality as perceived by humans, but human evaluation takes time, is costly, and is hard to do correctly, so we often make do with proxies. Traditional metrics (SSIM, PSNR-HVS) are commonly used as proxies; Butteraugli is designed to be such a proxy. These proxies can and do disagree with each other.

[magicgoose]

We will soon publish a more detailed comparison

How soon is soon?

[magicgoose]

By the way, I am almost sure that it if so happens that mozjpeg is more efficient, the comparison won't be published because mozjpeg is Mozilla's project and this is Google™ and Google™ doesn't like to admit failures after scratching their https://en.wikipedia.org/wiki/Not_invented_here syndrome.
And this may very well be the reason why there are still no comparisons with mozjpeg in README.

btw, mozjpeg is also ABI-compatible with libjpeg-turbo, this is a valuable thing, and if I understand it right, guetzli isn't
that means, it's a lot easier to integrate mozjpeg into existing software

[jyrkialakuijala]

We are still preparing the publication. Other people have run some tests in the mean time. Guetzli is both better and worse than Mozjpeg. https://twitter.com/fg118942/status/820984186974584832 shows an interesting reversal of codecs. Guetzli is the worst on SSIM and PSNR, but best on butteraugli scoring. This is really not a miracle nor a proof of its superiority since guetzli is just a complex optimizer for butteraugli rating.

Ordering of codecs with ssim:

1. BPG
2. JPEG XR/JPEG 2000
3. Mozjpeg
4. JPEG
5. Guetzli

Ordering of codecs with butteraugli:

1. Guetzli
2. BPG
3. Mozjpeg/JPEG
4. JPEG 2000
5. JPEG XR

Note, that butteraugli only looks at the worst area of the image, where as ssim and psnr aggregate errors from everywhere in the image.

I have read that human raters like JPEGs more than similarly sized JPEG 2000 and JPEG XR, but SSIM and PSNR consistently rank them the opposite as humans tend to do.

[acdha]

@magicgoose I'd like to see a rigorous comparison too, but … do you really think the tone of that message is helpful? I don't know what the mood is like at Google but I'm sure it's not helped by unnecessarily politicizing this issue.

[magicgoose]

A little bit of trolling is always good, that's sort of my motto. I'm pleased to see that this was unnecessary and these conspiracy theories are not true. 😄

[MaikWagner]

With some simple convert actions on the console comparing mozjpeg, libjpeg and guetzli with your bees.png sample image, I can't perceive any significant differences in matter of filesize.

convert -quality 85 bees.png bees-libjpeg-85.jpg (using libjpeg)
cjpeg -quality 85 bees.png > bees-mozjpeg-85.png.jpg (mozjpeg)
guetzli --quality 85 bees.png bees-guetzli-85.png.jpg

-rw-rw-r-- 1 maik maik  20316  6. Mär 13:07 bees-guetzli-85.png.jpg
-rw-rw-r-- 1 maik maik  20128  6. Mär 13:09 bees-libjpeg-85.png.jpg
-rw-rw-r-- 1 maik maik  20359  6. Mär 13:06 bees-mozjpeg-85.png.jpg
-rw-rw-r-- 1 maik maik 177424 24. Okt 17:11 bees.png

Wich comparison method do I have to use to get the 20-30% "typically 20-30% smaller" images with guetzli. Do I have to use it with butteraugli anyway?

[jyrkialakuijala]

Thanks for running these tests on guetzli. This is very helpful.

We have done two different comparisons. One where we keep the filesize the same between guetzli and libjpeg and get humans to look at the pictures, and another where we keep the butteraugli-quality the same and we look at filesize. The first shows a 75-80 % preference to guetzli generated images over the libjpeg images. The latter shows 35 % filesize reduction from libjpeg, and possibly around 30 % from mozjpeg. We didn't do a full human rater study between guetzli and mozjpeg, but a few samples indicated that mozjpeg is closer to libjpeg than guetzli in human viewing.

Our comparisons are done at around qualities 90 to 95. We used a calibrated high quality monitor intended for photo editing work.

Differences at these qualities are subtle.

I think the bees image is one that doesn't suffer a lot from YUV420, and guetzli's aversion to YUV420 might explain why that image works well with convert and mozjpeg. If you try YUV420 with an image where there are red things next to green things, you can observe what is happening in more detail.

[magicgoose]

Not sure why you are pointing at YUV420, mozjpeg can encode full resolution chroma as well.

[robryk]

@magicgoose Note that mozjpeg encodes with YUV420 by default. When you ask mozjpeg to encode with YUV444, the produced image is ~45% larger:

$ ~/mozjpeg/cjpeg -quality 85 bees.png > bees-mj.jpg
$ djpeg -v < bees-mj.jpg 2>&1 >/dev/null | grep -A3 'Start Of Frame'
Start Of Frame 0xc2: width=444, height=258, components=3
    Component 1: 2hx2v q=0
    Component 2: 1hx1v q=1
    Component 3: 1hx1v q=1
$ ~/mozjpeg/cjpeg -sample 1x1 -quality 85 bees.png > bees-mj-444.jpg
$ ls -l bees-mj*
-rw-r----- 1 robryk eng 29705 Mar  6 19:27 bees-mj-444.jpg
-rw-r----- 1 robryk eng 20039 Mar  6 19:21 bees-mj.jpg

[magicgoose]

@robryk I think it's YUV420 by default when quality is less than some number…

also, from my testing, -dct float option in mozjpeg improves efficiency a little bit.
-dc-scan-opt 2 may reduce file size a little bit without affecting quality.

[xlz]

http://screenshotcomparison.com/comparison/203655

Some blocky artifacts are more obvious. Just FYI.

[magicgoose]

@xlz can you also upload the source somewhere? I think it'd be interesting to compare it with mozjpeg too (not sure if GIMP uses mozjpeg already)

[jyrkialakuijala]

@xlz Try running guetzli with a higher quality. This shouldn't happen if you use quality 94 or higher.

Guetzli works in a way where image degradation starts to happen everywhere at roughly the same quality settings, it doesn't try to 'save' the easy areas from degradation.

[vinhlh]

I did some tests on my own product images, it seems mozjpeg is better in my case.

Average:

guetzli: reduce 19%.
mozjpeg: reduce 41%.

After reading guetzli papers, it seems my own product images aren't good candidates for guetzli, don't have a lot of pixels can be down-sampled.
https://github.com/vinhlh/guetzli-product-image-tests
https://vinhlh.github.io/guetzli-tests/public/stats/compare_guetzli.html
https://vinhlh.github.io/guetzli-tests/public/stats/compare_mozjpeg.html

[robryk]

@vinhlh Size comparisons make sense between images equivalent in quality. What method did you use to ensure that quality was the same on both sides?

I've taken a look at your repository. It seems that you've taken a set of images and did two things with them:
a) compressed them with Guetzli, at default quality of 95,
b) losslessly reencoded them with Mozjpeg's jpegtran as progressive JPEGs.

Note that (b) doesn't introduce any loss at all. You can do (b) to Guetzlified images too. We've decided not to do anything like (b) in Guetzli, because progressive JPEGs are up to 2x slower to decode.

I've tried re-encoding Guetzlified images in your corpus. The results, compared to yours only-Guetzli and only-reencoding, look as follows (percentages differ slightly from yours, because I've compared the total size change as opposed to averaging the percentages across images):

	size	size reduction
original	16492 kB
Guetzli only	11697 kB	30%
re-encoding only	10030 kB	39%
Guetzli & re-encoding	7490 kB	54%

Thanks for assembling this image corpus and pointing me at it; on most corpora I've seen re-encoding as progressive had smaller gains.

[vinhlh]

@robryk sorry, I didn't know mozjpeg use progressive by default.

And I have a question, what you guys meant by

Guetzli-generated images are typically 20-30% smaller than images of equivalent quality generated by libjpeg

is Guetzli with quality = 100? or with Butteraugli score < 0.97. Could I understand it as if Butteraugli score < 0.97, the compressed and the original image have similar perceived visual quality; and it's very hard for human eyes able to realize the differences.
(Same Butteraugli distance)

There's also a Google guy, recommends an optimal Butteraugli score = 1.1.
https://medium.com/@duhroach/reducing-jpg-file-size-e5b27df3257c#.a6cx2d5ea

I will do more tests with baseline images and a larger set.

[niutech]

Could you please add JPEGmini to the comparison?

One commenter Ashley Wilson claims his "16MP, 1.71MB file became 1.32MB in JPEGMini, whereas Guetzli only managed to get it to 1.51MB".

[magicgoose]

@niutech who cares? it's a proprietary software, so it's irrelevant.

[acdha]

@magicgoose If JPEGMini gets significantly better results I'm sure a lot of people would like to see an OSS solution become more competitive

[thehelp1]

You all are doing amazing work. Depending on circleci.com you may request certain demands. Within reason!!!

…

On Mar 27, 2017 2:07 PM, "magicgoose" ***@***.***> wrote: @niutech <https://github.com/niutech> who cares? it's a proprietary software, so it's irrelevant. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZNZWIf2EPSDjJhlnT9kq3GNANoVfQGUks5rqBb_gaJpZM4LdX6Q> .

[magicgoose]

@acdha yeah that's true, being more competitive is always good. Regardless of whether there exists some non-free software which is claimed to be more powerful.
What bothers me is the claim that some non-free software is better, without any evidence. It looks like a regular marketing job, and IMO it has no place in an issue tracker of a free software project.

[kornelski]

I'd like to remind everyone that any benchmark that only looks at file sizes is essentially invalid. Any such benchmark can simply be won by the worst codec that lowers image quality more than all other tools.

You can say one codec is better than another if it wins on both file size and quality.

But also be careful with "close enough" quality. In JPEG file size grows exponentially with increase in quality, so even differences in quality that are so small that you'd say "looks the same to me", can significantly vary file size, and not because the codec is technically better, but because the settings that were used or re-encoding process allowed it to lose more humanly-imperceptible information.

[kornelski]

In another issue @pflacson posted a link to a direct comparison: https://css-ig.net/articles/guetzli

Note about the comparison: it uses DSSIM metric, but Guetzli deliberately optimizes for its own Butteraugli metric. It definitely succeeds optimizing for its own metric, so in the end it result depends on what measurement you believe is more accurate (and neither of them is perfect).

[jmespadero]

Thanks for bringing here that comparative, Kornel.
I have been testing for a week with a large corpus of images, and my conclusions are similar. In my opinion, the better butteraugli distance achieved with guetzli does not pay the incredible amount of resources it needs to encode a file.
It is true that with guetzli you can easily beat the filesize of jpeg files created with imagemagick, but if you have in mind to reduce the size of a big set of personal images, you should better read about the parameters of mozjpeg and achieve the same (visual) result in x200 speed.

Also, I must say that this week guetzli has received a set of enhancements that have notably reduced the resources (memory to encode an image went from 300 to 125 kBytes bytes per pixel), so it is good to keep an eye on it.

[robryk]

@jmespadero Small correction: it's 125 bytes per pixel, not KB per pixel.

[jmespadero]

True. Receive one more time my congratulations for the set of optimizations and enhancements you have done recently.

[vinhlh]

I've tested with a larger set: 500 product images, this is the result:

❯❯❯ ./generate_size_values.sh                                                                                                                                                             
original 79925 kB
guetzli_images 55584 kB, 30% smaller (with guetzli)
jpegtran_images 60094 kB, 24% smaller (with jpegtran -optimize)
moz_images 55325 kB, 30% smaller (with mozjpeg -optimize)
guetzli_prog_images 40178 kB, 49% smaller (with guetzli then mozjpeg -optimize)

[magicgoose]

@vinhlh what about quality metrics?

[vinhlh]

@magicgoose

• guetzli with default quality (95).
• jpegtran/ mozjpeg is lossless, only use -optimize parameter.

Note: tested on a c4.2xlarge instance, it takes 66 minutes for guetzli compression.

[magicgoose]

@vinhlh and what's the point of this comparison?

[swrobel]

@vinhlh I'm a bit late to this discussion but I looked at your image comparisons and the mozjpeg colors seem to be significantly different to me. I do not see any difference with the guetzli ones.

[kornelski]

If color is significantly off in the entire image it's almost always not due to compression, but merely due to embedded color profile being interpreted differently.

Try converting images to sRGB or double-check that the same profile is embedded in the same way by all tools you compare.

[Slind14]

I ran another test with +2k user images from a forum:

original                        326598 kB
jpegtran_images                 306822 kB, 6% smaller
moz_images                      292494 kB, 10% smaller
guetzli_prog_jpegtran_images    270136 kB, 17% smaller
guetzli_images                  250229 kB, 23% smaller
guetzli_prog_moz_images         239275 kB, 26% smaller

[kornelski]

@Slind14 #10 (comment)

[Slind14]

Yes, I read that. I do not see a quality loss between guetzli vs guetzli + moz. The only diff I see is between original and guetzli at sharp edges.

[kornelski]

Again, please read the linked article, and see the example which shows 40% difference in compression efficiency and no obvious quality loss — by doing nothing.

[Slind14]

So what does it matter? Images are looked at by humans, if there is no visible difference for the human eye..

[Slind14]

Btw. in that example the quality difference is visible at the edges, the autor of the post messed up the file description. The one on the right has sharper edges.

[kornelski]

"no visible difference" is a fine goal, but your benchmark measures a different thing than what you think.

"smaller with no visible difference" does not actually decide which codec is better, because it does require a Pareto improvement. Even summing up file sizes is subject to Simpson's paradox.

Basically, it's easy to get statistics wrong. Codec comparison is a tricky 2D problem, and it's tempting to reduce it to a 1D problem, but that reduction is harder than it seems. I can make any codec win this way.

I've made a presentation about this that describes the fallacies in depth:
https://ldnwebperf.org/sessions/snake-oil-in-image-compression-how-to-ensure-youre-getting-best-quality-and-file-size/

[Slind14]

I never made those claims, its also not my comparison, see previous comments. I just ran it with my inventory because the prev. test was run on product images with solid color backgrounds, all encoded the same way... so there was no mix of different encoded images, qualities, sizes, content...

The goal for me is to cut file size/storage requirements without sacrificing (human visible) quality. This needs to work on thousands of different images, so it can't just work on a specific kind.
I don't see why running a few different setups on a big inventory and doing human eye comparison would be wrong in this case.

[kornelski]

The method you've used would have shown similar results if you've used the same codec, any JPEG codec, twice.

It shows that re-encoding and small variations in quality make a big difference. It says nothing about relative performance of codecs. Getting smaller files by re-encoding them at quality you're OK with is absolutely fine, but posting that in a thread about "comparison with mozjpeg" makes misleading claims.

[Slind14]

#10 (comment)
#10 (comment)

So you mean, if I run guetzli again (guetzli + guetzli), I end up with a comparable result as to guetzli + moz progressive?

[kornelski]

I mean that a simple method of "re-encoded a bunch of images, got smaller files, and quality is almost the same" is so inaccurate that it will show you a "improvement" even if you compare the same codec with itself. Since a codec obviously can't be better than itself, it shows that the method is misleading.

It's a perfectly valid thing to do if you want to have smaller files. It's just not a codec benchmark.

Take a bunch of random files, re-encode with mozjpeg at q 90.
Re-encode the images with mozjpeg at q 85. You will get result that can be described as "the only difference is at sharp edges".
Re-encode the images with mozjpeg at q 83. You will get even smaller files! And compared to the previous batch there will be no visible difference to human eyes.

Because lossy codecs can produce basically any file size they want, looking at just at the file size is not enough.

Because in JPEG relationship between quality and file size is close to exponential, differences in quality smaller than human can see create file size differences big enough to seem important. It doesn't mean technical improvement in the codec, it means quality was set higher than necessary.

[Slind14]

Lets just say, I re-encode all images with q90, since I do not have the original raw file, the result will be worse than doing the same with the original file. It could even be that the file size increases because it was encoded with a smaller quality before.

[tomByrer]

One can reduce filesizes for JPEG that is not lossy using MozJPEG. This is exactly what ImageOptim does (unless you turn on lossy, which is not default).

@Slind14 usually filesizes go down in recompess-lossy. I have not seen it go up.

[Slind14]

Doesn't guetzli claim to be lossless too?

…

On 30 Jul 2017 04:25, "Tom Byrer" ***@***.***> wrote: One can reduce filesizes for JPEG that is not lossy using MozJPEG. This is exactly what ImageOptim <https://imageoptim.com/mac> does (unless you turn on lossy, which is not default). @Slind14 <https://github.com/slind14> usually filesizes go down in recompess-lossy. I have not seen it go up. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADu7tdLjvQ3bm8PUxaSZkMj9mm5zIBRWks5sS-mfgaJpZM4LdX6Q> .

[kornelski]

Guetzli is lossy. It just claims to be smart about how much lossy it is.

[jyrkialakuijala]

@Slind14 Yes, lossy. Guetzli tries to be "psychovisually lossless" at default settings. However, the psychovisual model is not perfect and some observable loss may occur. Generally the worst artefacts in a Guetzli image should be less visible than with other JPEG codecs at similar byte budgets.

[Slind14]

So, the best way would be to re-encode images with a "medium-high" quality level (e.g. 80) then pick these if they are +10% smaller than the original, if not, pick the original?
Do you recommend moz for this? What would you use for png's?

[romainmenke]

@Slind14

If you have critical visual quality requirements you should start from a lossless source image. If you don't have that, there is no point in using Guetzli IMHO.

If you do have a lossless source image Guetzli will be a good fit to compress the image while minimising visual quality loss.

If however you have bandwidth restrictions and don't care about quality there are compressors out there that are better suited than Guetzli.

In no case does it ever make sense to compress an image multiple times. Doing so will have an effect, but never one that can't be achieved by choosing the right compressor in the first place and using the correct setting.

There will always be other ways to get smaller images or higher quality images. Guetzli only attempts to find a good compromise without requiring you to choose the best setting for each image.

[Slind14]

I do not have the source image every time. Images are from users, I'm looking for a way to improve mobile loading (save bandwidth) and storage requirements.

[romainmenke]

In that case I would not look into running Guetzli multiple times but analyse if the benefit of Guetzli (better visual quality) is worth it. If your application/users don't care about a few artefacts it would make sense to choose a compressor which is capable of much smaller file sizes.