Extreme memory consumption on big images

[GoogleCodeExporter]

I used this image 
http://www.profiphotos.com/blog/photos/articles/Untersberg/Panorama-BG.jpg (it 
is just random big file from Internet). It have 12830 x 3206, in JFIF 1.02 
format and have 9MB. Uncompressed to PNG it have 46MB.

When trying to convert it to WEBP, webpconv consumes about 1.5GB (gigabytes) of 
RAM, severly slowing my computer (it is 2GB RAM, so started significantly 
swaping).

Config: 32-bit, i386, pentium-m 1.7GHz, Debian Linux/GNU unstable, kernel 
2.6.36-rc5. gcc 4.5.1.

Original issue reported on code.google.com by witold.b...@gmail.com on 1 Oct 2010 at 8:39

[GoogleCodeExporter]

I actually needed to kill this process. So probably peak memory usage is even 
higher than this reported 1.5GB.

Original comment by witold.b...@gmail.com on 1 Oct 2010 at 8:40

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

could it be related to the size of the look-ahead encoding queue?

Original comment by s...@google.com on 2 Nov 2010 at 11:00

[GoogleCodeExporter]

Confirmed (marking as accepted)
On 32 bit windows, larger images simply fail.  They work on 64 bit linux.
A 64 bit build of the tools is one option.
Can you look into where all the memory is going?

Original comment by fbarch...@chromium.org on 11 Nov 2010 at 6:17

• Changed title: Extreme memory consumption on big images
• Changed state: Accepted

[GoogleCodeExporter]

Issue 27 has been merged into this issue.

Original comment by vikaas.a...@gmail.com on 15 Nov 2010 at 12:25

[GoogleCodeExporter]

Original comment by vikaas.a...@gmail.com on 23 Nov 2010 at 10:54

[GoogleCodeExporter]

'cwebp' is now working with big sources.

The following command shows an average usage of 62M of RAM, with
a peak of ~120M during the initial image loading.

> convert Panorama-BG.jpg panorama.png
> cwebp panorama.png -o panorama.webp

Saving file 'panorama.webp'
3174900 bytes Y-U-V-All-PSNR 40.09 42.64 43.65   40.89 dB
block count:  intra4: 86592
              intra16: 74610  (-> 46.28%)
              skipped block: 5814 (3.61%)
bytes used:  header:            537  (0.0%)
             mode-partition: 469740  (14.8%)
 Residuals bytes  |segment 1|segment 2|segment 3|segment 4|  total
  intra4-coeffs:  | 1371591 |  443680 |  143034 |   36780 | 1995085  (62.8%)
 intra16-coeffs:  |   37346 |  135198 |  132952 |   32433 |  337929  (10.6%)
  chroma coeffs:  |  190608 |  118558 |   50117 |   12299 |  371582  (11.7%)
    macroblocks:  |      35%|      37%|      22%|       5%|  161202
      quantizer:  |      32 |      25 |      19 |      15 |
   filter level:  |       5 |       0 |       0 |       0 |
------------------+---------+---------+---------+---------+-----------------
 segments total:  | 1599545 |  697436 |  326103 |   81512 | 2704596  (85.2%)

Original comment by s...@google.com on 25 Feb 2011 at 2:42

• Changed state: Fixed

[GoogleCodeExporter]

does this also allow 16k x 16k on 32 bit systems?

Original comment by fbarch...@google.com on 25 Feb 2011 at 10:27