Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimizing Keka for smaller size on disk #149

Closed
MaxPower85 opened this Issue Mar 14, 2018 · 33 comments

Comments

Projects
None yet
3 participants
@MaxPower85
Copy link

MaxPower85 commented Mar 14, 2018

Here are more optimized versions of icons from Keka's beta.

Compressed file.tar.gz

Also... I recommend that you use ditto --hfsCompression on Keka.app to copy it to DMG when you are packaging it for the DMG that you are putting on the website (you don't have to do it for the AppStore version... all AppStore apps get compressed with HFS+ compression once installed), instead of just using a compressed DMG (you can still use the compressed DMG, but have the files inside use HFS+ compression, so the app would be compressed with HFS+ compression when the user moves it to the Applications folder)... that would save a few MBs on disk too.

With these optimized icons and HFS+ compression, Keka.app takes just 19MB on disk (instead of 53.5MB)... and there's still room to save more space... since the icon for gz and gzip is the same and the icon for lz and lzip is the same, when the icon appears twice for similar formats, just symlink it.

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 14, 2018

I thought that when a HFS+ compressed file/app is copied the compression is undone? Is this not the case when copying from a DMG?

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 14, 2018

@gingerbeardman
It depends from how someone copied it... if you just drag&drop it to another HFS+ or APFS partition (or a DMG), it stays compressed... if you copy it to your clipboard then you paste it from the clipboard, then it loses the compression.

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 14, 2018

That’s great info. I’m currently working on an app that manages HFS+ compression for users. Let me know if you want to test it. (sorry to derail thread)

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 14, 2018

Here are more optimized versions of icons from Keka's beta.

@MaxPower85 can you tell me what changes you've made to them so I can replicate it? Great work!

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 14, 2018

since the icon for gz and gzip is the same and the icon for lz and lzip is the same, when the icon appears twice for similar formats, just symlink it.

@MaxPower85 they should be already symlinks... It seems that in the build process Xcode follows the symlink and copies the original.

screen shot 2018-03-14 at 17 33 34

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 14, 2018

I'm making some tests with the HFS+ compression for the dmg. Actually compressing the app results in a slightly bigger dmg, but the app itself seems to be smaller. I'm getting mixed results when checking the app size. Only du -sh Keka.app reports that it is smaller.

By the way @MaxPower85 I did try the drag & drop and the copy/paste and booth respected the compression.

@aonez aonez added this to the 1.1.0 milestone Mar 14, 2018

@aonez aonez added the enhancement label Mar 14, 2018

@aonez aonez self-assigned this Mar 14, 2018

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 14, 2018

Same here. Duplicate or copying to another volume resulted in decompression.

“Get Info” shows you true “on disk” size.

screen shot 2018-03-14 at 17 41 05

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 14, 2018

“Get Info” shows you true size.

Im getting "53.5 MB on disk" even if du -shreports it is 44MB. Working on 10.11.6 El Capitan... will try now on 10.13.

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 14, 2018

It can take a short while to update the true size.

also mdls Keka.app

shows

_kMDItemDisplayNameWithExtensions  = "Keka.app"
…
kMDItemLogicalSize                 = 53484070
kMDItemPhysicalSize                = 44675072
…
kMDItemVersion                     = “1.1.0-beta.7"
@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 15, 2018

@aonez
About optimizing icons... I just used some free icns optimizing app from AppStore that makes an optimized version on drag&drop... but how it optimizes was not very configurable and it didn't support dragging&dropping multiple items at the same time.

So I did some more searching now to find a better way...
Take a look at this: https://github.com/avl7771/createicns

It allows you to extract PNGs from an .icns file, so you can optimize them with a PNG optimizer of your choice and configure the optimizations how you want... and then you can use createicns to make an .icns file from those PNGs and it keeps the optimizations when it creates the .icns file (the iconutil that comes with macOS doesn't keep the optimizations when it creates an .icns file). With a simple bash script you could run it on multiple items.

About HFS+ compression... I see now that copy & paste from Finder also seems to respect the compression... I guess when I tested copying it before that maybe the size on disk didn't update right away or something like that.

@gingerbeardman
About an app to manage HFS+ compression... see if you can find some way for users to avoid unnecessary writes to disk if you are making an app for that... so not just something that would monitor a folder for the user copying uncompressed files to it... maybe some Finder extension to copy a file using ditto --hfsCompression would be good... although the best way would be if there was a way to compress new files before they are written to the disk, but I'm not sure if that can be done without having the whole uncompressed file in RAM first (like if you mounted a RAM disk as a directory, so files written to it would go to RAM and then automatically copied with ditto --hfsCompression by some app that monitors that directory to some other directory on disk), which obviously wouldn't be very practical for very large files.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 16, 2018

@MaxPower85 I already use an script to create the icons so I'm testing createicns right away, because I've already tried to do some reduction in the PNG intermediate files but the ICNS where anyway big. Thanks!

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 16, 2018

@MaxPower85 did the test with amazing results. Did not got as fas as your tests, I've got 533KB instead of your 435KB, using ImageOptim. The colours of the original design are well kept with createicns.

Here a test, containing your version, the original version and the version I've got with optimizations:
Test.zip

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 17, 2018

Ok I'm done with PNG optimization... I suppose @MaxPower85 you've used pngquant as recommended in createicns. I'm using the binaries that ImageOptim uses, which are lossless, that's why the final icon is bigger.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 17, 2018

@MaxPower85 I'm having problems with the icons generated with createicns (avl7771/createicns#1). Your icons do not have this issue, can you let me know what PNG optimizer did you used? Tried with pngquant and got the same issue.

I'm gonna leave the icons uncompressed right now.

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 17, 2018

First, try using pngquant with these settings pngquant -v --speed 1 --quality 95-100 -f file.png -o file.png (that means that you will only accept the quality settings of 95 or higher... or you can set it to 98-100 if you only want to accept quality settings of 98 or higher... and it's a verbose mode, so it will tell you how many shades it found and which quality settings is using for them... it tries to examine the images and it will not reduce the image to 256 colors mode if the quality loss would be below of what you would accept... so, with those settings, it will not reduce images to 256 color mode if there are thousands and thousands of different shades in different colors like with the Siri's icon because that would be a really noticeable visual difference... but if there are only a few hundred different shades in the original image, then reducing it to 256 colors probably wouldn't be very noticeable, especially with dithering between 2 shades that are almost the same)

After pngquant, use zopflipng -y -m file.png file.png (it makes PNGs use compression... zopfli is an implementation of DEFLATE and PNG format supports compression, but PNGs created by various apps are often uncompressed).

With those two, I've managed to reduce the size of that same .icns file even more... it still looks pretty good, but now it has about 350KB :)

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 17, 2018

If you want something with a GUI to examine how would some image look when reduced to 256 colors (or maybe even less if the original image doesn't have too many shades) and would it look better with dithering or without... you can use ImageAlpha (although it's not very practical to use it for many files, since it opens a new window for every file... but you can see how something would look and then just use a bash script with pngquant if it looks acceptable)...

It seems that your icons look good enough even with only 128 shades (since they don't have various different colors in the same icon) and with 256 shades I'm not sure if I can tell any difference between it and the originals... and I have a Retina Mac... so don't worry about using pngquant.

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 17, 2018

BTW... there seems to be an issue with images for some sizes in the original .icns files (those included with Keka beta 7)... some seem noticeably darker, like some sizes were using different color profiles... like, if you open the 7z.icns file in Preview, look at the images number 6, 8 and 10 and compare them to other images.

You can also see the icons change their shade in Finder if you go to View > Show Status Bar and you change the size of icons to see how would they look at different sizes.

But the .png files that you included separately when you wrote about that issue you had with that _7.icns, those don't seem to have that issue... so it seems that the issue is that something was done differently with some of those PNG files when you created the original .icns file.

If you are using some optimizer, make sure that the same exact settings were used on all of those PNGs when you are creating the .icns file... don't use something with a GUI if you aren't sure what settings it is using when it optimizes all images and what exactly is doing with every image... and it would be a good idea to use the verbose mode.

I'm writing a script that you should use... I'm testing it now to make sure that everything is OK.

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 18, 2018

Here is the script... first version... maybe I'll add more features to it later: https://github.com/MaxPower85/icnsOptimize

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 18, 2018

But the .png files that you included separately when you wrote about that issue you had with that _7.icns, those don't seem to have that issue... so it seems that the issue is that something was done differently with some of those PNG files when you created the original .icns file.

The current beta icons are not optimized, just the PNGs I've shared and iconutil. That's why I've told you that createicns respects the colors way better, the icons should be "darker" or more vivid.

Thanks for the script! I've already done one that uses pngcrush, optipng and zopflipng, all lossless and get the icons to half the size, I think it's more that acceptable. I can share it if you like.

But to have this optimization benefits I need to use creations and fix the issue avl7771/createicns#1.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 18, 2018

So just to clarify a little more, the icons are originally in Sketch format (done by the great @ralts00), a sample design and all the possible layers and colors. Using a custom script I'm creating all Sketch variants, then converting them (with sketchtool) to PNG (no optimization, just an export). Then all those PNGs are converted using iconutil (where the colors get messed).

With your ideas I've added a new intermediate script that optimized those PNGs and converted them to icons using createicns instead of iconutil.

So no GUI used. Just used a GUI to perform the first tests of your idea.

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 18, 2018

I found a workaround to use createicns without those issue... I'll make some changes to my script a bit later to use that workaround until they can fix that bug...

Here is the workaround...

Create the .icns file with iconutil -c icns then extract the PNGs from it with readicns (even though it would then extract the 16x16 and 32x32 sizes as "icon_data_il32", "icon_data_info", "icon_data_is32", "icon_data_l8mk" and "icon_data_s8mk" files... don't remove those from the folder), then you can optimize the PNG files that were extracted without issues and then create the icns file with createicns... it will recognoze those "icon_data_il32", "icon_data_info", "icon_data_is32", "icon_data_l8mk" and "icon_data_s8mk" files and include them... and the final .icns file doesn't seem to have issues then.

But to make sure that those 16x16 and 32x32 images go through the optimization (so the colors would remain consistent)... and since you can't use PNG optimizers on those "icon_data_il32", "icon_data_info", "icon_data_is32", "icon_data_l8mk" and "icon_data_s8mk" files, before you create the icon with iconutil -c icns, make sure that all files go through the same optimization process as they would go after you extract the files with readicns.

pngquant -v --speed 1 --quality 95-100 -f file.png followed by zopflipng -y -m file.png file.png seems to have pretty good and consistent results for your icons and it doesn't seem to cause that issue that the shade of icon changes for different sizes like the icons that you included with Keka Beta 7.

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 18, 2018

BTW... if some optimizer is making changes to the color profile that some image uses, if the new color profile is different, to keep colors the same it would have to make changes to the image to readjust the colors... keeping the image the same while only changing the color profile can noticeably affect the visual quality of the image because with a different color profile it could look washed out or oversaturated or something like that (and depending from how big the difference was between those color profiles, the visual difference can even be very noticeably in some cases), even if they call it "lossless" because they aren't changing the actual image.

To avoid having images that look noticeably different after optimization, it's more important that the PNG optimizer makes adjustments to the image to keep colors visually similar after removing the color profile, even if that's not "lossless"... that can look better and more similar to the original than if they keep the image as it is and call it "lossless" while only removing the color profile.

If you want to keep the image as it is for some reason, then make sure that the color profile isn't touched also.

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 19, 2018

It seems that 16x16 and 32x32 non-Retina sizes from .icns files created by iconutil -c icns aren't actually PNGs... il32, is32, l8mk and s8mk seems to be icon formats that .icns files used for the Classic Mac OS 8.5.

It seems that iconutil -c icns converts those sizes to the PackBits format, so when you try to extract different icon sizes with readicns from an .icns file that was created by iconutil -c icns, instead of getting PNG images for 16x16 and 32x32 non-Retina sizes, you get those other files like "icon_data_il32", "icon_data_info", "icon_data_is32", "icon_data_l8mk" and "icon_data_s8mk" instead... and if you have those files in your .iconset folder, createicns would include them in the .icns file it creates.

That explains why I didn't see any issues with icons when I was only extracting the PNGs with readicns and 16x16 and 32x32 non-Retina sizes stayed in the original format.

For now, it seems that the best solution would be to use that workaround, to make sure that 16x16 and 32x32 non-Retina sizes get converted to that format that Classic Mac OS 8.5 uses.

Although, based on what articles say, .icns format should allow PNGs for 16x16 and 32x32 non-Retina sizes too on OS X 10.7 and newer... articles say that icp4 and icp5 formats in an .icns file are for PNGs at 16x16 and 32x32... and createicns seems to label them correctly... but for some reason, it seems like macOS still expects that PackBits format for those sizes.

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 19, 2018

readicns from an .icns file that was created by iconutil -c icns, instead of getting PNG images for 16x16 and 32x32 non-Retina sizes, you get those other files...

This is right now the best way! Using that already. Thanks for all your research @MaxPower85!

pngquant -v --speed 1 --quality 95-100

This just gives me the warning image degradation MSE=4.621 (Q=84) exceeded limit of 1.454 (95) and does not apply the reduction. Really I prefer lossless quality even if it weights a little more, more than happy with the pngcrush, optipng and zopflipng combination.

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 19, 2018

I once used ImageOptim and ImageAlpha to bring down the size of a Universal iOS app with retina images. Unoptimised it was 12MB and optimised it was 3.5MB!

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 19, 2018

Here a candidate to be the beta 8. HFS+ compression and icons optimized. From 52 to 23 MB. Not bad at all. The app icon is not optimized in this version: Keka-1.1.0-beta.8.dmg.zip

Let me know if you see any issue!

Thanks a lot @MaxPower85 and @gingerbeardman 💯 👍

@MaxPower85

This comment has been minimized.

Copy link
Author

MaxPower85 commented Mar 19, 2018

Don't forget the Keka_Legacy.icns... it has 2.4 MB on disk, even though it isn's very colorful... even Siri's icon with all those colors and gradients in different shades that have to be smooth has less (Siri's icon has 1.8 MB).

In ImageAlpha, even in 64 color mode with dithering when you compare the Keka_Legacy icon with the original (you have Show Original button to quickly compare), there seems to be very little difference and it looks almost the same if you don't zoom in to examine the details more closely... with 128 colors, even if you zoom in, it seems that is mostly the same and that only some shades change slightly, mostly the semi-transparent shades around the edges, but it still looks good and it only seems slightly noticeable if you zoom in and switch between it and the original... even on MBP Retina... and you can double that to 256 colors.

For Siri's icon even 256 color mode looks very noticeably different, even without having to zoom-in and switch between modes to compare, because of how many shades it has.

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 19, 2018

what version of afsctool did you use and with what settings? @aonez

I am running yesterday's release and it squeezes to 21MB

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 20, 2018

what version of afsctool did you use and with what settings?

@gingerbeardman I used the bundled ditto. Will try afsctool. Tried. Will use it instead of ditto 👍

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 20, 2018

@MaxPower85 I did not optimized Keka_Legacy.icns because it will not be part of the 1.1.0 version. The new app icon is optimized and has more color variety. But I'm waiting for the final release to use it.

You can get an idea of the new icon in Keka.app/Contents/Resourcestab_KekaM.png 😛

@aonez aonez added the package label Mar 21, 2018

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 21, 2018

It is curious that the more compressed with HFS+ the app is, the less compression the DMG package achieves. But for ~400 KB more on the package weight, having 10+ MB savings when the app is copied on the disk is worth it.

No compression:
screen shot 2018-03-21 at 09 44 29
ditto HFS+ compression:
screen shot 2018-03-21 at 09 44 31
afsctool level 9 HFS+ compression:
screen shot 2018-03-21 at 09 44 33

@gingerbeardman

This comment has been minimized.

Copy link
Contributor

gingerbeardman commented Mar 21, 2018

it's a bit like zipping a zipping, the second time there's not much gain

@aonez

This comment has been minimized.

Copy link
Owner

aonez commented Mar 21, 2018

I'm closing this one as a success!

  • Icons reduced in size
  • Correctly using symlinks on icons (no more duplicates)
  • Images reduced in size
  • HFS+ compression (afsctool level 9) on DMG package app

Thanks again @gingerbeardman and @MaxPower85!

@aonez aonez closed this Mar 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.