overflow/image corruption in octree with png256 + large pixel output

[artemp]

see the large image here:

http://dbsgeo.com/tmp/wms_image_corruption.png

fixed with advice from artem:

#!diff
Index: include/mapnik/octree.hpp
===================================================================
--- include/mapnik/octree.hpp   (revision 1659)
+++ include/mapnik/octree.hpp   (working copy)
@@ -86,11 +86,11 @@

                bool is_leaf() const { return count == 0; }
                node * children_[8];
-               unsigned reds;
-               unsigned greens;
-               unsigned blues;
-               unsigned count;
-               unsigned count2;
+               boost::uint64_t  reds;
+               boost::uint64_t  greens;
+               boost::uint64_t  blues;
+               boost::uint64_t  count;
+               boost::uint64_t  count2;
                byte  children_count;
                byte  index;
          };

Although with this patch white background + png256 output (image/png8 mime) turns same color as processed_p.shp

compare:

normal png:

http://tile2.dbsgeo.com/?layers=__all__&bbox=18.497972,-72.479639,18.572645,-72.247381&service=WMS&request=GetMap&version=1.3.0&srs=EPSG:4326&format=image/png&styles=&crs=EPSG:4326&height=10000&width=12000

to:

png256/above patch:

http://tile2.dbsgeo.com/?layers=__all__&bbox=18.497972,-72.479639,18.572645,-72.247381&service=WMS&request=GetMap&version=1.3.0&srs=EPSG:4326&format=image/png8&styles=&crs=EPSG:4326&height=10000&width=12000

[artemp]

[mar_rud] Yes, in this case there were so many pixels of similar color, that unsigned type overflowed. In worse case it could happened with more than 16M pixels of near white color, that have assigned single palette index. For square image it is 4000x4000 image with mostly near white background.

As for color deformation with 64bit type, png256 is rather fast and mostly enough, then perfect in all cases. In this case changing
{{{
const static unsigned MIN_LEVELS = 3;
}}}
to 2 would help, but as I remember, in some cases it could also lead to loosing rare colors, i.e. single symbol in distinct color could disappear, so I left it with 3 value.

I'm going to look into octree.h to find some ways of improvement, maybe considering color variance when searching for node to reduce, but with better palette, image size will increase, not mentioning cpu time.

I just did simple test with converting 1312x1136 image to png256. "convert" from imagemagick got 208kB file in 2.2s, and simple program using mapniks functions got 152kB file in 0.4s. Smaller file is because less optimized palette, but in this case there weren't any visible differences.

On some faster machine and mentioned 12000x10000, 28MB png file imagemagick got: 14MB (45s, 1.3GB RAM), mapniks octree: 6.4MB (13s, 590MB RAM), but visible color difference and 9MB with MIN_LEVELS=2.

[artemp]

[mar_rud] Attached patch includes mentioned fix from artem and some improvements in octree.h to first reduce nodes that introduce smallest color error, instead of ones with least pixels.

I checked it with my test images I often use for png256 and it generally produces slightly better image (closer colors to original) with only <5% file size increase. Also for mentioned extreme 120MPix image it produces correct colors (image size: 9.2MB instead of 6.4MB), so I've committed to 0.7.1-dev and trunk (r1661, r1662).

[artemp]

[springmeyer] Great stuff. I recompiled on the server I experienced the issue on and r1661 fixed both of the color issues. fantastic.