Losslessly compress all tiles

[Leland]

Reduces file sizes by >30% on average, reducing total file size by 6.5mb.

Pngcrush, Zopfli, PNGOUT and Advpng were used to compress.

[kevingranade]

What does it do to load speed?

[ZhilkinSerg]

What does it do to load speed?

"However, remember that decompression is still the same speed, and totally safe" (see https://blog.codinghorror.com/zopfli-optimization-literally-free-bandwidth/ and https://developers.googleblog.com/2013/02/compress-data-more-densely-with-zopfli.html).

Also https://superuser.com/questions/741011/do-more-compressed-png-images-take-longer-to-load.

[Leland]

Decompression speeds should be similar, - perhaps marginally increased (we're talking nanoseconds most likely) and the net result should be faster load speeds as file size is much more important in image loading than compression scheme.

This is for mobile apps, and thus different hardware, but the principle is the same:

Decoding speed appears to be correlated to image file size more than anything else (most likely savings on byteswapping are negligible compared to additional disk I/O and extra data to decompress.)

https://imageoptim.com/tweetbot.html

And another one: https://www.cocoanetics.com/2011/10/avoiding-image-decompression-sickness/

[kevingranade]

The answer you're looking for is "I don't know", the articles pointing out that it helps sometimes are irrelevant. I'll see when I can test it, but no idea when I can get around to it.

[Leland]

Hah – thought those articles were fairly conclusive. Will work on finding more assertive ones.

[kevingranade]

I'm convinced that it's faster in theory, so more articles won't help. I just want a simple test that demonstrates that it doesn't actually regress. I need to double-check when tile loading happens, but it might be sufficient to simply do something like:
configure cataclysm to use one of the tilesets
time ./cataclysm
hit the requisite keys to enter a game
hit the requisite keys to exit the game
check the output of time
repeat for each tileset, and for before/after.

[codemime]

I like the change, but it will tend to regress since the tilesets are occasionally updated.

[Leland]

@codemime correct – was almost tempted to not compress the Chesthole tilesets. The other tilesets are very rarely updated.

However, my thinking is that a future regression from an update will have minimal impact and I'm more than happy to compress again at certain points in the future.

Also @kevingranade I've been looking into this a lot more and my thinking was erroneous – there will be no additional "decompression" time. To the computer, these files are rendered exactly the same as the original files, they just happen to be smaller.

PNGs themselves are compressed files, and the performance of decompressing them into raw pixels is only relative to the filesize. Which is to say, the speed of rendering them to the screen will only increase as filesize goes down. The algorithm used to get to that smaller filesize is irrelevant.

[Coolthulhu]

It should probably be automated.
Best if it was an optional (defaulting to yes if necessary tools are detected) step in make bindist. Then we could have the binary distro always use the smaller files without manual re-compressing.

[Leland]

@Coolthulhu I will say that my initial goal in this was to free up room in the repo for adding in the music created in the original CDDA contest. As I understand, it was removed due to space constraints on the CI server.

[kevingranade]

@Leland I think a better way to handle the space constraints is to make the data files available outside the existing build path. Also the music files are just gigantic all on their own :/

[Leland]

Made #21690 for tracking tile compression automation; this PR should be a net positive for the time being, and ready to merge – assuming @kevingranade's good with my breakdown of compression logic above.

[kevingranade]

hahahahaha!!!!.... whew
Are you ready for this? (times in ms)
tl;dr ~60% faster!!!!!!!!

Tileset	before	after	difference
gfx//BlockheadTileset/blockheadtiles.png	68738	25622	43116
gfx//ChestHole16Tileset/fallback.png	83480	2966	80514
gfx//ChestHole16Tileset/tiles24.png	83871	919	82952
gfx//ChestHole16Tileset/tiles.png	145250	23332	121918
gfx//ChestHole32Tileset/fallback.png	158360	9051	149309
gfx//ChestHole32Tileset_iso/fallback.png	169289	74067	95222
gfx//ChestHole32Tileset_iso/tiles48.png	10854	5364	5490
gfx//ChestHole32Tileset_iso/tiles.png	201099	122479	78620
gfx//ChestHole32Tileset/tiles48.png	106521	3333	103188
gfx//ChestHole32Tileset/tiles.png	250300	83373	166927
gfx//ChestHole32Tileset/tree.png	80772	7751	73021
gfx//ChestHoleTileset/fallback.png	26897	5567	21330
gfx//ChestHoleTileset/tiles32.png	3115	2729	386
gfx//ChestHoleTileset/tiles.png	70712	44648	26064
gfx//ChestHoleTileset/tree.png	4482	3817	665
gfx//DeonTileset/deontiles.png	25422	14569	10853
gfx//HitButton_iso/fallback.png	11470	4560	6910
gfx//HitButton_iso/HitButton_iso.png	7737	5012	2725
gfx//HoderTileset/hodertiles.png	6878	4109	2769
gfx//MShock24TilesetModded/fallback.png	80641	23303	57338
gfx//MShock24TilesetModded/tiles.png	289673	32318	257355
gfx//MShock32TilesetModded/fallback.png	63024	33339	29685
gfx//MShock32TilesetModded/tiles.png	73969	71667	2302
gfx//RetroASCIITileset/retroasciitiles.png	18200	3569	14631
gfx//RetroDaysTileset10/retrodaysfallback10.png	103381	1619	101762
gfx//RetroDaysTileset10/retrodaystiles10.png	53724	1653	52071
gfx//RetroDaysTileset20/../RetroDaysTileset10/retrodaysfallback10.png	4152	1592	2560
gfx//RetroDaysTileset20/../RetroDaysTileset10/retrodaystiles10.png	3883	1933	1950
gfx//ThuztorTileset@/thuztortiles@.png	24294	4940	19354
gfx//TsuTileset/tsutiles.png	17631	3760	13871

[kevingranade]

BTW, here's my benchmark code:

diff --git a/src/cata_tiles.cpp b/src/cata_tiles.cpp
index ec9ce84170..443cef15fb 100644
--- a/src/cata_tiles.cpp
+++ b/src/cata_tiles.cpp
@@ -316,12 +316,17 @@ static void apply_color_filter(SDL_Surface_Ptr &surf, void (&pixel_converter)(pi
         }
     }
 }
+#include <chrono>
 
 int cata_tiles::load_tileset(std::string img_path, int R, int G, int B, int sprite_width, int sprite_height)
 {
+    auto start = std::chrono::high_resolution_clock::now();
     /** reinit tile_atlas */
     SDL_Surface_Ptr tile_atlas( IMG_Load( img_path.c_str() ) );
-
+    auto end = std::chrono::high_resolution_clock::now();
+    long diff = std::chrono::duration_cast<std::chrono::microseconds>( end - start ).count();
+    printf( "Loaded %s in %ld microseconds\n", img_path.c_str(), diff );
+    
     if(!tile_atlas) {
         throw std::runtime_error( std::string("Could not load tileset image at ") + img_path + ", error: " +
                                   IMG_GetError() );

[kevingranade]

Android players are gonna LOVE you.

[kevingranade]

Side note, I don't trust these numbers to be at all accurate, they're jittery as hell, but it's a massive improvement so measuring more carefully is pointless.

[ZhilkinSerg]

Now we gonna need to revive #19165 and/or #19412 and the game will be quicker than roadrunner.