New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comparison with mozjpeg? #10

Open
magicgoose opened this Issue Jan 7, 2017 · 53 comments

Comments

Projects
None yet
@magicgoose

magicgoose commented Jan 7, 2017

README says:

Guetzli-generated images are typically 20-30% smaller than images of equivalent quality generated by libjpeg

But what about mozjpeg? It existed long before this project, by the way — 1.0 release was in Mar 5, 2014, and it does similar job.

@robryk

This comment has been minimized.

Show comment
Hide comment
@robryk

robryk Jan 12, 2017

Collaborator

We will soon publish a more detailed comparison that takes mozjpeg and libjpeg into account.

Note that comparing two JPEG encoders is more complicated than it might at first appear, because one needs to compare quality of the compressed image. Obviously, we actually care about quality as perceived by humans, but human evaluation takes time, is costly, and is hard to do correctly, so we often make do with proxies. Traditional metrics (SSIM, PSNR-HVS) are commonly used as proxies; Butteraugli is designed to be such a proxy. These proxies can and do disagree with each other.

Collaborator

robryk commented Jan 12, 2017

We will soon publish a more detailed comparison that takes mozjpeg and libjpeg into account.

Note that comparing two JPEG encoders is more complicated than it might at first appear, because one needs to compare quality of the compressed image. Obviously, we actually care about quality as perceived by humans, but human evaluation takes time, is costly, and is hard to do correctly, so we often make do with proxies. Traditional metrics (SSIM, PSNR-HVS) are commonly used as proxies; Butteraugli is designed to be such a proxy. These proxies can and do disagree with each other.

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Jan 26, 2017

We will soon publish a more detailed comparison

How soon is soon?

magicgoose commented Jan 26, 2017

We will soon publish a more detailed comparison

How soon is soon?

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Jan 31, 2017

By the way, I am almost sure that it if so happens that mozjpeg is more efficient, the comparison won't be published because mozjpeg is Mozilla's project and this is Google™ and Google™ doesn't like to admit failures after scratching their https://en.wikipedia.org/wiki/Not_invented_here syndrome.
And this may very well be the reason why there are still no comparisons with mozjpeg in README.

btw, mozjpeg is also ABI-compatible with libjpeg-turbo, this is a valuable thing, and if I understand it right, guetzli isn't
that means, it's a lot easier to integrate mozjpeg into existing software

magicgoose commented Jan 31, 2017

By the way, I am almost sure that it if so happens that mozjpeg is more efficient, the comparison won't be published because mozjpeg is Mozilla's project and this is Google™ and Google™ doesn't like to admit failures after scratching their https://en.wikipedia.org/wiki/Not_invented_here syndrome.
And this may very well be the reason why there are still no comparisons with mozjpeg in README.

btw, mozjpeg is also ABI-compatible with libjpeg-turbo, this is a valuable thing, and if I understand it right, guetzli isn't
that means, it's a lot easier to integrate mozjpeg into existing software

@jyrkialakuijala

This comment has been minimized.

Show comment
Hide comment
@jyrkialakuijala

jyrkialakuijala Jan 31, 2017

Collaborator

We are still preparing the publication. Other people have run some tests in the mean time. Guetzli is both better and worse than Mozjpeg. https://twitter.com/fg118942/status/820984186974584832 shows an interesting reversal of codecs. Guetzli is the worst on SSIM and PSNR, but best on butteraugli scoring. This is really not a miracle nor a proof of its superiority since guetzli is just a complex optimizer for butteraugli rating.

Ordering of codecs with ssim:

  1. BPG
  2. JPEG XR/JPEG 2000
  3. Mozjpeg
  4. JPEG
  5. Guetzli

Ordering of codecs with butteraugli:

  1. Guetzli
  2. BPG
  3. Mozjpeg/JPEG
  4. JPEG 2000
  5. JPEG XR

Note, that butteraugli only looks at the worst area of the image, where as ssim and psnr aggregate errors from everywhere in the image.

I have read that human raters like JPEGs more than similarly sized JPEG 2000 and JPEG XR, but SSIM and PSNR consistently rank them the opposite as humans tend to do.

Collaborator

jyrkialakuijala commented Jan 31, 2017

We are still preparing the publication. Other people have run some tests in the mean time. Guetzli is both better and worse than Mozjpeg. https://twitter.com/fg118942/status/820984186974584832 shows an interesting reversal of codecs. Guetzli is the worst on SSIM and PSNR, but best on butteraugli scoring. This is really not a miracle nor a proof of its superiority since guetzli is just a complex optimizer for butteraugli rating.

Ordering of codecs with ssim:

  1. BPG
  2. JPEG XR/JPEG 2000
  3. Mozjpeg
  4. JPEG
  5. Guetzli

Ordering of codecs with butteraugli:

  1. Guetzli
  2. BPG
  3. Mozjpeg/JPEG
  4. JPEG 2000
  5. JPEG XR

Note, that butteraugli only looks at the worst area of the image, where as ssim and psnr aggregate errors from everywhere in the image.

I have read that human raters like JPEGs more than similarly sized JPEG 2000 and JPEG XR, but SSIM and PSNR consistently rank them the opposite as humans tend to do.

@acdha

This comment has been minimized.

Show comment
Hide comment
@acdha

acdha Jan 31, 2017

@magicgoose I'd like to see a rigorous comparison too, but … do you really think the tone of that message is helpful? I don't know what the mood is like at Google but I'm sure it's not helped by unnecessarily politicizing this issue.

acdha commented Jan 31, 2017

@magicgoose I'd like to see a rigorous comparison too, but … do you really think the tone of that message is helpful? I don't know what the mood is like at Google but I'm sure it's not helped by unnecessarily politicizing this issue.

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Jan 31, 2017

A little bit of trolling is always good, that's sort of my motto. I'm pleased to see that this was unnecessary and these conspiracy theories are not true. 😄

magicgoose commented Jan 31, 2017

A little bit of trolling is always good, that's sort of my motto. I'm pleased to see that this was unnecessary and these conspiracy theories are not true. 😄

@MaikWagner

This comment has been minimized.

Show comment
Hide comment
@MaikWagner

MaikWagner Mar 6, 2017

With some simple convert actions on the console comparing mozjpeg, libjpeg and guetzli with your bees.png sample image, I can't perceive any significant differences in matter of filesize.

convert -quality 85 bees.png bees-libjpeg-85.jpg (using libjpeg)
cjpeg -quality 85 bees.png > bees-mozjpeg-85.png.jpg (mozjpeg)
guetzli --quality 85 bees.png bees-guetzli-85.png.jpg

-rw-rw-r-- 1 maik maik  20316  6. Mär 13:07 bees-guetzli-85.png.jpg
-rw-rw-r-- 1 maik maik  20128  6. Mär 13:09 bees-libjpeg-85.png.jpg
-rw-rw-r-- 1 maik maik  20359  6. Mär 13:06 bees-mozjpeg-85.png.jpg
-rw-rw-r-- 1 maik maik 177424 24. Okt 17:11 bees.png

Wich comparison method do I have to use to get the 20-30% "typically 20-30% smaller" images with guetzli. Do I have to use it with butteraugli anyway?

MaikWagner commented Mar 6, 2017

With some simple convert actions on the console comparing mozjpeg, libjpeg and guetzli with your bees.png sample image, I can't perceive any significant differences in matter of filesize.

convert -quality 85 bees.png bees-libjpeg-85.jpg (using libjpeg)
cjpeg -quality 85 bees.png > bees-mozjpeg-85.png.jpg (mozjpeg)
guetzli --quality 85 bees.png bees-guetzli-85.png.jpg

-rw-rw-r-- 1 maik maik  20316  6. Mär 13:07 bees-guetzli-85.png.jpg
-rw-rw-r-- 1 maik maik  20128  6. Mär 13:09 bees-libjpeg-85.png.jpg
-rw-rw-r-- 1 maik maik  20359  6. Mär 13:06 bees-mozjpeg-85.png.jpg
-rw-rw-r-- 1 maik maik 177424 24. Okt 17:11 bees.png

Wich comparison method do I have to use to get the 20-30% "typically 20-30% smaller" images with guetzli. Do I have to use it with butteraugli anyway?

@jyrkialakuijala

This comment has been minimized.

Show comment
Hide comment
@jyrkialakuijala

jyrkialakuijala Mar 6, 2017

Collaborator

Thanks for running these tests on guetzli. This is very helpful.

We have done two different comparisons. One where we keep the filesize the same between guetzli and libjpeg and get humans to look at the pictures, and another where we keep the butteraugli-quality the same and we look at filesize. The first shows a 75-80 % preference to guetzli generated images over the libjpeg images. The latter shows 35 % filesize reduction from libjpeg, and possibly around 30 % from mozjpeg. We didn't do a full human rater study between guetzli and mozjpeg, but a few samples indicated that mozjpeg is closer to libjpeg than guetzli in human viewing.

Our comparisons are done at around qualities 90 to 95. We used a calibrated high quality monitor intended for photo editing work.

Differences at these qualities are subtle.

I think the bees image is one that doesn't suffer a lot from YUV420, and guetzli's aversion to YUV420 might explain why that image works well with convert and mozjpeg. If you try YUV420 with an image where there are red things next to green things, you can observe what is happening in more detail.

Collaborator

jyrkialakuijala commented Mar 6, 2017

Thanks for running these tests on guetzli. This is very helpful.

We have done two different comparisons. One where we keep the filesize the same between guetzli and libjpeg and get humans to look at the pictures, and another where we keep the butteraugli-quality the same and we look at filesize. The first shows a 75-80 % preference to guetzli generated images over the libjpeg images. The latter shows 35 % filesize reduction from libjpeg, and possibly around 30 % from mozjpeg. We didn't do a full human rater study between guetzli and mozjpeg, but a few samples indicated that mozjpeg is closer to libjpeg than guetzli in human viewing.

Our comparisons are done at around qualities 90 to 95. We used a calibrated high quality monitor intended for photo editing work.

Differences at these qualities are subtle.

I think the bees image is one that doesn't suffer a lot from YUV420, and guetzli's aversion to YUV420 might explain why that image works well with convert and mozjpeg. If you try YUV420 with an image where there are red things next to green things, you can observe what is happening in more detail.

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Mar 6, 2017

Not sure why you are pointing at YUV420, mozjpeg can encode full resolution chroma as well.

magicgoose commented Mar 6, 2017

Not sure why you are pointing at YUV420, mozjpeg can encode full resolution chroma as well.

@robryk

This comment has been minimized.

Show comment
Hide comment
@robryk

robryk Mar 6, 2017

Collaborator

@magicgoose Note that mozjpeg encodes with YUV420 by default. When you ask mozjpeg to encode with YUV444, the produced image is ~45% larger:

$ ~/mozjpeg/cjpeg -quality 85 bees.png > bees-mj.jpg
$ djpeg -v < bees-mj.jpg 2>&1 >/dev/null | grep -A3 'Start Of Frame'
Start Of Frame 0xc2: width=444, height=258, components=3
    Component 1: 2hx2v q=0
    Component 2: 1hx1v q=1
    Component 3: 1hx1v q=1
$ ~/mozjpeg/cjpeg -sample 1x1 -quality 85 bees.png > bees-mj-444.jpg
$ ls -l bees-mj*
-rw-r----- 1 robryk eng 29705 Mar  6 19:27 bees-mj-444.jpg
-rw-r----- 1 robryk eng 20039 Mar  6 19:21 bees-mj.jpg
Collaborator

robryk commented Mar 6, 2017

@magicgoose Note that mozjpeg encodes with YUV420 by default. When you ask mozjpeg to encode with YUV444, the produced image is ~45% larger:

$ ~/mozjpeg/cjpeg -quality 85 bees.png > bees-mj.jpg
$ djpeg -v < bees-mj.jpg 2>&1 >/dev/null | grep -A3 'Start Of Frame'
Start Of Frame 0xc2: width=444, height=258, components=3
    Component 1: 2hx2v q=0
    Component 2: 1hx1v q=1
    Component 3: 1hx1v q=1
$ ~/mozjpeg/cjpeg -sample 1x1 -quality 85 bees.png > bees-mj-444.jpg
$ ls -l bees-mj*
-rw-r----- 1 robryk eng 29705 Mar  6 19:27 bees-mj-444.jpg
-rw-r----- 1 robryk eng 20039 Mar  6 19:21 bees-mj.jpg
@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Mar 6, 2017

@robryk I think it's YUV420 by default when quality is less than some number…

also, from my testing, -dct float option in mozjpeg improves efficiency a little bit.
-dc-scan-opt 2 may reduce file size a little bit without affecting quality.

magicgoose commented Mar 6, 2017

@robryk I think it's YUV420 by default when quality is less than some number…

also, from my testing, -dct float option in mozjpeg improves efficiency a little bit.
-dc-scan-opt 2 may reduce file size a little bit without affecting quality.

@xlz

This comment has been minimized.

Show comment
Hide comment
@xlz

xlz Mar 18, 2017

http://screenshotcomparison.com/comparison/203655

Some blocky artifacts are more obvious. Just FYI.

xlz commented Mar 18, 2017

http://screenshotcomparison.com/comparison/203655

Some blocky artifacts are more obvious. Just FYI.

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Mar 18, 2017

@xlz can you also upload the source somewhere? I think it'd be interesting to compare it with mozjpeg too (not sure if GIMP uses mozjpeg already)

magicgoose commented Mar 18, 2017

@xlz can you also upload the source somewhere? I think it'd be interesting to compare it with mozjpeg too (not sure if GIMP uses mozjpeg already)

@jyrkialakuijala

This comment has been minimized.

Show comment
Hide comment
@jyrkialakuijala

jyrkialakuijala Mar 21, 2017

Collaborator

@xlz Try running guetzli with a higher quality. This shouldn't happen if you use quality 94 or higher.

Guetzli works in a way where image degradation starts to happen everywhere at roughly the same quality settings, it doesn't try to 'save' the easy areas from degradation.

Collaborator

jyrkialakuijala commented Mar 21, 2017

@xlz Try running guetzli with a higher quality. This shouldn't happen if you use quality 94 or higher.

Guetzli works in a way where image degradation starts to happen everywhere at roughly the same quality settings, it doesn't try to 'save' the easy areas from degradation.

@vinhlh

This comment has been minimized.

Show comment
Hide comment
@vinhlh

vinhlh Mar 22, 2017

I did some tests on my own product images, it seems mozjpeg is better in my case.

Average:

guetzli: reduce 19%.
mozjpeg: reduce 41%.

After reading guetzli papers, it seems my own product images aren't good candidates for guetzli, don't have a lot of pixels can be down-sampled.
https://github.com/vinhlh/guetzli-product-image-tests
https://vinhlh.github.io/guetzli-tests/public/stats/compare_guetzli.html
https://vinhlh.github.io/guetzli-tests/public/stats/compare_mozjpeg.html

vinhlh commented Mar 22, 2017

I did some tests on my own product images, it seems mozjpeg is better in my case.

Average:

guetzli: reduce 19%.
mozjpeg: reduce 41%.

After reading guetzli papers, it seems my own product images aren't good candidates for guetzli, don't have a lot of pixels can be down-sampled.
https://github.com/vinhlh/guetzli-product-image-tests
https://vinhlh.github.io/guetzli-tests/public/stats/compare_guetzli.html
https://vinhlh.github.io/guetzli-tests/public/stats/compare_mozjpeg.html

@robryk

This comment has been minimized.

Show comment
Hide comment
@robryk

robryk Mar 22, 2017

Collaborator

@vinhlh Size comparisons make sense between images equivalent in quality. What method did you use to ensure that quality was the same on both sides?

I've taken a look at your repository. It seems that you've taken a set of images and did two things with them:
a) compressed them with Guetzli, at default quality of 95,
b) losslessly reencoded them with Mozjpeg's jpegtran as progressive JPEGs.

Note that (b) doesn't introduce any loss at all. You can do (b) to Guetzlified images too. We've decided not to do anything like (b) in Guetzli, because progressive JPEGs are up to 2x slower to decode.

I've tried re-encoding Guetzlified images in your corpus. The results, compared to yours only-Guetzli and only-reencoding, look as follows (percentages differ slightly from yours, because I've compared the total size change as opposed to averaging the percentages across images):

size size reduction
original 16492 kB
Guetzli only 11697 kB 30%
re-encoding only 10030 kB 39%
Guetzli & re-encoding 7490 kB 54%

Thanks for assembling this image corpus and pointing me at it; on most corpora I've seen re-encoding as progressive had smaller gains.

Collaborator

robryk commented Mar 22, 2017

@vinhlh Size comparisons make sense between images equivalent in quality. What method did you use to ensure that quality was the same on both sides?

I've taken a look at your repository. It seems that you've taken a set of images and did two things with them:
a) compressed them with Guetzli, at default quality of 95,
b) losslessly reencoded them with Mozjpeg's jpegtran as progressive JPEGs.

Note that (b) doesn't introduce any loss at all. You can do (b) to Guetzlified images too. We've decided not to do anything like (b) in Guetzli, because progressive JPEGs are up to 2x slower to decode.

I've tried re-encoding Guetzlified images in your corpus. The results, compared to yours only-Guetzli and only-reencoding, look as follows (percentages differ slightly from yours, because I've compared the total size change as opposed to averaging the percentages across images):

size size reduction
original 16492 kB
Guetzli only 11697 kB 30%
re-encoding only 10030 kB 39%
Guetzli & re-encoding 7490 kB 54%

Thanks for assembling this image corpus and pointing me at it; on most corpora I've seen re-encoding as progressive had smaller gains.

@vinhlh

This comment has been minimized.

Show comment
Hide comment
@vinhlh

vinhlh Mar 23, 2017

@robryk sorry, I didn't know mozjpeg use progressive by default.

And I have a question, what you guys meant by

Guetzli-generated images are typically 20-30% smaller than images of equivalent quality generated by libjpeg

is Guetzli with quality = 100? or with Butteraugli score < 0.97. Could I understand it as if Butteraugli score < 0.97, the compressed and the original image have similar perceived visual quality; and it's very hard for human eyes able to realize the differences.
(Same Butteraugli distance)

There's also a Google guy, recommends an optimal Butteraugli score = 1.1.
https://medium.com/@duhroach/reducing-jpg-file-size-e5b27df3257c#.a6cx2d5ea

I will do more tests with baseline images and a larger set.

vinhlh commented Mar 23, 2017

@robryk sorry, I didn't know mozjpeg use progressive by default.

And I have a question, what you guys meant by

Guetzli-generated images are typically 20-30% smaller than images of equivalent quality generated by libjpeg

is Guetzli with quality = 100? or with Butteraugli score < 0.97. Could I understand it as if Butteraugli score < 0.97, the compressed and the original image have similar perceived visual quality; and it's very hard for human eyes able to realize the differences.
(Same Butteraugli distance)

There's also a Google guy, recommends an optimal Butteraugli score = 1.1.
https://medium.com/@duhroach/reducing-jpg-file-size-e5b27df3257c#.a6cx2d5ea

I will do more tests with baseline images and a larger set.

@niutech

This comment has been minimized.

Show comment
Hide comment
@niutech

niutech Mar 27, 2017

Could you please add JPEGmini to the comparison?

One commenter Ashley Wilson claims his "16MP, 1.71MB file became 1.32MB in JPEGMini, whereas Guetzli only managed to get it to 1.51MB".

niutech commented Mar 27, 2017

Could you please add JPEGmini to the comparison?

One commenter Ashley Wilson claims his "16MP, 1.71MB file became 1.32MB in JPEGMini, whereas Guetzli only managed to get it to 1.51MB".

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Mar 27, 2017

@niutech who cares? it's a proprietary software, so it's irrelevant.

magicgoose commented Mar 27, 2017

@niutech who cares? it's a proprietary software, so it's irrelevant.

@acdha

This comment has been minimized.

Show comment
Hide comment
@acdha

acdha Mar 27, 2017

@magicgoose If JPEGMini gets significantly better results I'm sure a lot of people would like to see an OSS solution become more competitive

acdha commented Mar 27, 2017

@magicgoose If JPEGMini gets significantly better results I'm sure a lot of people would like to see an OSS solution become more competitive

@thehelp1

This comment has been minimized.

Show comment
Hide comment
@thehelp1

thehelp1 Mar 28, 2017

thehelp1 commented Mar 28, 2017

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Mar 28, 2017

@acdha yeah that's true, being more competitive is always good. Regardless of whether there exists some non-free software which is claimed to be more powerful.
What bothers me is the claim that some non-free software is better, without any evidence. It looks like a regular marketing job, and IMO it has no place in an issue tracker of a free software project.

magicgoose commented Mar 28, 2017

@acdha yeah that's true, being more competitive is always good. Regardless of whether there exists some non-free software which is claimed to be more powerful.
What bothers me is the claim that some non-free software is better, without any evidence. It looks like a regular marketing job, and IMO it has no place in an issue tracker of a free software project.

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Mar 28, 2017

I'd like to remind everyone that any benchmark that only looks at file sizes is essentially invalid. Any such benchmark can simply be won by the worst codec that lowers image quality more than all other tools.

You can say one codec is better than another if it wins on both file size and quality.

But also be careful with "close enough" quality. In JPEG file size grows exponentially with increase in quality, so even differences in quality that are so small that you'd say "looks the same to me", can significantly vary file size, and not because the codec is technically better, but because the settings that were used or re-encoding process allowed it to lose more humanly-imperceptible information.

kornelski commented Mar 28, 2017

I'd like to remind everyone that any benchmark that only looks at file sizes is essentially invalid. Any such benchmark can simply be won by the worst codec that lowers image quality more than all other tools.

You can say one codec is better than another if it wins on both file size and quality.

But also be careful with "close enough" quality. In JPEG file size grows exponentially with increase in quality, so even differences in quality that are so small that you'd say "looks the same to me", can significantly vary file size, and not because the codec is technically better, but because the settings that were used or re-encoding process allowed it to lose more humanly-imperceptible information.

@kornelski kornelski referenced this issue Apr 1, 2017

Closed

Compare MOZJPG #160

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Apr 1, 2017

In another issue @pflacson posted a link to a direct comparison: https://css-ig.net/articles/guetzli

Note about the comparison: it uses DSSIM metric, but Guetzli deliberately optimizes for its own Butteraugli metric. It definitely succeeds optimizing for its own metric, so in the end it result depends on what measurement you believe is more accurate (and neither of them is perfect).

kornelski commented Apr 1, 2017

In another issue @pflacson posted a link to a direct comparison: https://css-ig.net/articles/guetzli

Note about the comparison: it uses DSSIM metric, but Guetzli deliberately optimizes for its own Butteraugli metric. It definitely succeeds optimizing for its own metric, so in the end it result depends on what measurement you believe is more accurate (and neither of them is perfect).

@jmespadero

This comment has been minimized.

Show comment
Hide comment
@jmespadero

jmespadero Apr 4, 2017

Thanks for bringing here that comparative, Kornel.
I have been testing for a week with a large corpus of images, and my conclusions are similar. In my opinion, the better butteraugli distance achieved with guetzli does not pay the incredible amount of resources it needs to encode a file.
It is true that with guetzli you can easily beat the filesize of jpeg files created with imagemagick, but if you have in mind to reduce the size of a big set of personal images, you should better read about the parameters of mozjpeg and achieve the same (visual) result in x200 speed.

Also, I must say that this week guetzli has received a set of enhancements that have notably reduced the resources (memory to encode an image went from 300 to 125 kBytes bytes per pixel), so it is good to keep an eye on it.

jmespadero commented Apr 4, 2017

Thanks for bringing here that comparative, Kornel.
I have been testing for a week with a large corpus of images, and my conclusions are similar. In my opinion, the better butteraugli distance achieved with guetzli does not pay the incredible amount of resources it needs to encode a file.
It is true that with guetzli you can easily beat the filesize of jpeg files created with imagemagick, but if you have in mind to reduce the size of a big set of personal images, you should better read about the parameters of mozjpeg and achieve the same (visual) result in x200 speed.

Also, I must say that this week guetzli has received a set of enhancements that have notably reduced the resources (memory to encode an image went from 300 to 125 kBytes bytes per pixel), so it is good to keep an eye on it.

@robryk

This comment has been minimized.

Show comment
Hide comment
@robryk

robryk Apr 4, 2017

Collaborator

@jmespadero Small correction: it's 125 bytes per pixel, not KB per pixel.

Collaborator

robryk commented Apr 4, 2017

@jmespadero Small correction: it's 125 bytes per pixel, not KB per pixel.

@jmespadero

This comment has been minimized.

Show comment
Hide comment
@jmespadero

jmespadero Apr 4, 2017

True. Receive one more time my congratulations for the set of optimizations and enhancements you have done recently.

jmespadero commented Apr 4, 2017

True. Receive one more time my congratulations for the set of optimizations and enhancements you have done recently.

@vinhlh

This comment has been minimized.

Show comment
Hide comment
@vinhlh

vinhlh Apr 5, 2017

I've tested with a larger set: 500 product images, this is the result:

❯❯❯ ./generate_size_values.sh                                                                                                                                                             
original 79925 kB
guetzli_images 55584 kB, 30% smaller (with guetzli)
jpegtran_images 60094 kB, 24% smaller (with jpegtran -optimize)
moz_images 55325 kB, 30% smaller (with mozjpeg -optimize)
guetzli_prog_images 40178 kB, 49% smaller (with guetzli then mozjpeg -optimize)

vinhlh commented Apr 5, 2017

I've tested with a larger set: 500 product images, this is the result:

❯❯❯ ./generate_size_values.sh                                                                                                                                                             
original 79925 kB
guetzli_images 55584 kB, 30% smaller (with guetzli)
jpegtran_images 60094 kB, 24% smaller (with jpegtran -optimize)
moz_images 55325 kB, 30% smaller (with mozjpeg -optimize)
guetzli_prog_images 40178 kB, 49% smaller (with guetzli then mozjpeg -optimize)
@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Apr 5, 2017

@vinhlh what about quality metrics?

magicgoose commented Apr 5, 2017

@vinhlh what about quality metrics?

@vinhlh

This comment has been minimized.

Show comment
Hide comment
@vinhlh

vinhlh Apr 5, 2017

@magicgoose

  • guetzli with default quality (95).
  • jpegtran/ mozjpeg is lossless, only use -optimize parameter.

Note: tested on a c4.2xlarge instance, it takes 66 minutes for guetzli compression.

vinhlh commented Apr 5, 2017

@magicgoose

  • guetzli with default quality (95).
  • jpegtran/ mozjpeg is lossless, only use -optimize parameter.

Note: tested on a c4.2xlarge instance, it takes 66 minutes for guetzli compression.

@magicgoose

This comment has been minimized.

Show comment
Hide comment
@magicgoose

magicgoose Apr 5, 2017

@vinhlh and what's the point of this comparison?

magicgoose commented Apr 5, 2017

@vinhlh and what's the point of this comparison?

@swrobel

This comment has been minimized.

Show comment
Hide comment
@swrobel

swrobel Jul 3, 2017

@vinhlh I'm a bit late to this discussion but I looked at your image comparisons and the mozjpeg colors seem to be significantly different to me. I do not see any difference with the guetzli ones.

swrobel commented Jul 3, 2017

@vinhlh I'm a bit late to this discussion but I looked at your image comparisons and the mozjpeg colors seem to be significantly different to me. I do not see any difference with the guetzli ones.

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 3, 2017

If color is significantly off in the entire image it's almost always not due to compression, but merely due to embedded color profile being interpreted differently.

Try converting images to sRGB or double-check that the same profile is embedded in the same way by all tools you compare.

kornelski commented Jul 3, 2017

If color is significantly off in the entire image it's almost always not due to compression, but merely due to embedded color profile being interpreted differently.

Try converting images to sRGB or double-check that the same profile is embedded in the same way by all tools you compare.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

I ran another test with +2k user images from a forum:

original                        326598 kB
jpegtran_images                 306822 kB, 6% smaller
moz_images                      292494 kB, 10% smaller
guetzli_prog_jpegtran_images    270136 kB, 17% smaller
guetzli_images                  250229 kB, 23% smaller
guetzli_prog_moz_images         239275 kB, 26% smaller

Slind14 commented Jul 29, 2017

I ran another test with +2k user images from a forum:

original                        326598 kB
jpegtran_images                 306822 kB, 6% smaller
moz_images                      292494 kB, 10% smaller
guetzli_prog_jpegtran_images    270136 kB, 17% smaller
guetzli_images                  250229 kB, 23% smaller
guetzli_prog_moz_images         239275 kB, 26% smaller
@kornelski

This comment has been minimized.

Show comment
Hide comment

kornelski commented Jul 29, 2017

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

Yes, I read that. I do not see a quality loss between guetzli vs guetzli + moz. The only diff I see is between original and guetzli at sharp edges.

Slind14 commented Jul 29, 2017

Yes, I read that. I do not see a quality loss between guetzli vs guetzli + moz. The only diff I see is between original and guetzli at sharp edges.

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 29, 2017

Again, please read the linked article, and see the example which shows 40% difference in compression efficiency and no obvious quality loss — by doing nothing.

kornelski commented Jul 29, 2017

Again, please read the linked article, and see the example which shows 40% difference in compression efficiency and no obvious quality loss — by doing nothing.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

So what does it matter? Images are looked at by humans, if there is no visible difference for the human eye..

Slind14 commented Jul 29, 2017

So what does it matter? Images are looked at by humans, if there is no visible difference for the human eye..

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

Btw. in that example the quality difference is visible at the edges, the autor of the post messed up the file description. The one on the right has sharper edges.

Slind14 commented Jul 29, 2017

Btw. in that example the quality difference is visible at the edges, the autor of the post messed up the file description. The one on the right has sharper edges.

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 29, 2017

"no visible difference" is a fine goal, but your benchmark measures a different thing than what you think.

"smaller with no visible difference" does not actually decide which codec is better, because it does require a Pareto improvement. Even summing up file sizes is subject to Simpson's paradox.

Basically, it's easy to get statistics wrong. Codec comparison is a tricky 2D problem, and it's tempting to reduce it to a 1D problem, but that reduction is harder than it seems. I can make any codec win this way.

I've made a presentation about this that describes the fallacies in depth:
https://ldnwebperf.org/sessions/snake-oil-in-image-compression-how-to-ensure-youre-getting-best-quality-and-file-size/

kornelski commented Jul 29, 2017

"no visible difference" is a fine goal, but your benchmark measures a different thing than what you think.

"smaller with no visible difference" does not actually decide which codec is better, because it does require a Pareto improvement. Even summing up file sizes is subject to Simpson's paradox.

Basically, it's easy to get statistics wrong. Codec comparison is a tricky 2D problem, and it's tempting to reduce it to a 1D problem, but that reduction is harder than it seems. I can make any codec win this way.

I've made a presentation about this that describes the fallacies in depth:
https://ldnwebperf.org/sessions/snake-oil-in-image-compression-how-to-ensure-youre-getting-best-quality-and-file-size/

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

I never made those claims, its also not my comparison, see previous comments. I just ran it with my inventory because the prev. test was run on product images with solid color backgrounds, all encoded the same way... so there was no mix of different encoded images, qualities, sizes, content...

The goal for me is to cut file size/storage requirements without sacrificing (human visible) quality. This needs to work on thousands of different images, so it can't just work on a specific kind.
I don't see why running a few different setups on a big inventory and doing human eye comparison would be wrong in this case.

Slind14 commented Jul 29, 2017

I never made those claims, its also not my comparison, see previous comments. I just ran it with my inventory because the prev. test was run on product images with solid color backgrounds, all encoded the same way... so there was no mix of different encoded images, qualities, sizes, content...

The goal for me is to cut file size/storage requirements without sacrificing (human visible) quality. This needs to work on thousands of different images, so it can't just work on a specific kind.
I don't see why running a few different setups on a big inventory and doing human eye comparison would be wrong in this case.

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 29, 2017

The method you've used would have shown similar results if you've used the same codec, any JPEG codec, twice.

It shows that re-encoding and small variations in quality make a big difference. It says nothing about relative performance of codecs. Getting smaller files by re-encoding them at quality you're OK with is absolutely fine, but posting that in a thread about "comparison with mozjpeg" makes misleading claims.

kornelski commented Jul 29, 2017

The method you've used would have shown similar results if you've used the same codec, any JPEG codec, twice.

It shows that re-encoding and small variations in quality make a big difference. It says nothing about relative performance of codecs. Getting smaller files by re-encoding them at quality you're OK with is absolutely fine, but posting that in a thread about "comparison with mozjpeg" makes misleading claims.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

#10 (comment)
#10 (comment)

So you mean, if I run guetzli again (guetzli + guetzli), I end up with a comparable result as to guetzli + moz progressive?

Slind14 commented Jul 29, 2017

#10 (comment)
#10 (comment)

So you mean, if I run guetzli again (guetzli + guetzli), I end up with a comparable result as to guetzli + moz progressive?

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 29, 2017

I mean that a simple method of "re-encoded a bunch of images, got smaller files, and quality is almost the same" is so inaccurate that it will show you a "improvement" even if you compare the same codec with itself. Since a codec obviously can't be better than itself, it shows that the method is misleading.

It's a perfectly valid thing to do if you want to have smaller files. It's just not a codec benchmark.

Take a bunch of random files, re-encode with mozjpeg at q 90.
Re-encode the images with mozjpeg at q 85. You will get result that can be described as "the only difference is at sharp edges".
Re-encode the images with mozjpeg at q 83. You will get even smaller files! And compared to the previous batch there will be no visible difference to human eyes.

Because lossy codecs can produce basically any file size they want, looking at just at the file size is not enough.

Because in JPEG relationship between quality and file size is close to exponential, differences in quality smaller than human can see create file size differences big enough to seem important. It doesn't mean technical improvement in the codec, it means quality was set higher than necessary.

kornelski commented Jul 29, 2017

I mean that a simple method of "re-encoded a bunch of images, got smaller files, and quality is almost the same" is so inaccurate that it will show you a "improvement" even if you compare the same codec with itself. Since a codec obviously can't be better than itself, it shows that the method is misleading.

It's a perfectly valid thing to do if you want to have smaller files. It's just not a codec benchmark.

Take a bunch of random files, re-encode with mozjpeg at q 90.
Re-encode the images with mozjpeg at q 85. You will get result that can be described as "the only difference is at sharp edges".
Re-encode the images with mozjpeg at q 83. You will get even smaller files! And compared to the previous batch there will be no visible difference to human eyes.

Because lossy codecs can produce basically any file size they want, looking at just at the file size is not enough.

Because in JPEG relationship between quality and file size is close to exponential, differences in quality smaller than human can see create file size differences big enough to seem important. It doesn't mean technical improvement in the codec, it means quality was set higher than necessary.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 29, 2017

Lets just say, I re-encode all images with q90, since I do not have the original raw file, the result will be worse than doing the same with the original file. It could even be that the file size increases because it was encoded with a smaller quality before.

Slind14 commented Jul 29, 2017

Lets just say, I re-encode all images with q90, since I do not have the original raw file, the result will be worse than doing the same with the original file. It could even be that the file size increases because it was encoded with a smaller quality before.

@tomByrer

This comment has been minimized.

Show comment
Hide comment
@tomByrer

tomByrer Jul 30, 2017

One can reduce filesizes for JPEG that is not lossy using MozJPEG. This is exactly what ImageOptim does (unless you turn on lossy, which is not default).

@Slind14 usually filesizes go down in recompess-lossy. I have not seen it go up.

tomByrer commented Jul 30, 2017

One can reduce filesizes for JPEG that is not lossy using MozJPEG. This is exactly what ImageOptim does (unless you turn on lossy, which is not default).

@Slind14 usually filesizes go down in recompess-lossy. I have not seen it go up.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Jul 30, 2017

Slind14 commented Jul 30, 2017

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 30, 2017

Guetzli is lossy. It just claims to be smart about how much lossy it is.

kornelski commented Jul 30, 2017

Guetzli is lossy. It just claims to be smart about how much lossy it is.

@jyrkialakuijala

This comment has been minimized.

Show comment
Hide comment
@jyrkialakuijala

jyrkialakuijala Aug 11, 2017

Collaborator

@Slind14 Yes, lossy. Guetzli tries to be "psychovisually lossless" at default settings. However, the psychovisual model is not perfect and some observable loss may occur. Generally the worst artefacts in a Guetzli image should be less visible than with other JPEG codecs at similar byte budgets.

Collaborator

jyrkialakuijala commented Aug 11, 2017

@Slind14 Yes, lossy. Guetzli tries to be "psychovisually lossless" at default settings. However, the psychovisual model is not perfect and some observable loss may occur. Generally the worst artefacts in a Guetzli image should be less visible than with other JPEG codecs at similar byte budgets.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Aug 11, 2017

So, the best way would be to re-encode images with a "medium-high" quality level (e.g. 80) then pick these if they are +10% smaller than the original, if not, pick the original?
Do you recommend moz for this? What would you use for png's?

Slind14 commented Aug 11, 2017

So, the best way would be to re-encode images with a "medium-high" quality level (e.g. 80) then pick these if they are +10% smaller than the original, if not, pick the original?
Do you recommend moz for this? What would you use for png's?

@romainmenke

This comment has been minimized.

Show comment
Hide comment
@romainmenke

romainmenke Aug 11, 2017

@Slind14

If you have critical visual quality requirements you should start from a lossless source image. If you don't have that, there is no point in using Guetzli IMHO.

If you do have a lossless source image Guetzli will be a good fit to compress the image while minimising visual quality loss.

If however you have bandwidth restrictions and don't care about quality there are compressors out there that are better suited than Guetzli.

In no case does it ever make sense to compress an image multiple times. Doing so will have an effect, but never one that can't be achieved by choosing the right compressor in the first place and using the correct setting.

There will always be other ways to get smaller images or higher quality images. Guetzli only attempts to find a good compromise without requiring you to choose the best setting for each image.

romainmenke commented Aug 11, 2017

@Slind14

If you have critical visual quality requirements you should start from a lossless source image. If you don't have that, there is no point in using Guetzli IMHO.

If you do have a lossless source image Guetzli will be a good fit to compress the image while minimising visual quality loss.

If however you have bandwidth restrictions and don't care about quality there are compressors out there that are better suited than Guetzli.

In no case does it ever make sense to compress an image multiple times. Doing so will have an effect, but never one that can't be achieved by choosing the right compressor in the first place and using the correct setting.

There will always be other ways to get smaller images or higher quality images. Guetzli only attempts to find a good compromise without requiring you to choose the best setting for each image.

@Slind14

This comment has been minimized.

Show comment
Hide comment
@Slind14

Slind14 Aug 11, 2017

I do not have the source image every time. Images are from users, I'm looking for a way to improve mobile loading (save bandwidth) and storage requirements.

Slind14 commented Aug 11, 2017

I do not have the source image every time. Images are from users, I'm looking for a way to improve mobile loading (save bandwidth) and storage requirements.

@romainmenke

This comment has been minimized.

Show comment
Hide comment
@romainmenke

romainmenke Aug 11, 2017

In that case I would not look into running Guetzli multiple times but analyse if the benefit of Guetzli (better visual quality) is worth it. If your application/users don't care about a few artefacts it would make sense to choose a compressor which is capable of much smaller file sizes.

romainmenke commented Aug 11, 2017

In that case I would not look into running Guetzli multiple times but analyse if the benefit of Guetzli (better visual quality) is worth it. If your application/users don't care about a few artefacts it would make sense to choose a compressor which is capable of much smaller file sizes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment