image: decode / resize into an existing buffer

[nigeltao]

The image/foo packages currently provide foo.Decode functions that allocate a new image
buffer. It would be nice, especially when decoding a moving image, to decode into an
existing buffer.

Also, it would be nice to be able to resize an image during (and not after) decoding,
which can obviously allocate a smaller buffer but can also be faster. Apparently
Imagemagick can do this for JPEGs, and
https://groups.google.com/forum/#!topic/golang-nuts/13Gr_AmBAKY claims a 50% speedup for
generating a thumbnail.

The two concerns are superficially separate, but the API (if any) to do this should
probably consider both use cases.

[gopherbot]

Comment 1 by richard.scothern:

To see this in action, install imagemagick and run the following commands.  'convert' is
an imagemagick command line tool for image transformations.  This command resizes a
3264x2448 jpeg to 800x800. 
$ time  convert balloon02.jpg -colorspace RGB -resize 128x128 -auto-orient
balloon02-128x128.jpg
real    0m0.165s
user    0m0.628s
sys 0m0.020s
If you profile with perf, you will note that 60% of the time is spent decoding, and ~20%
resizing.  Here, a 'size hint' is passed to the jpeg decoder to inform it that the
output will be no larger than 128x128:
$ time  convert balloon02.jpg -jpeg:size=128x128 -colorspace RGB -resize 128x128
-auto-orient balloon02-128x128.jpg
real    0m0.078s
user    0m0.040s
sys 0m0.032s
Note that the first example completes in 165ms and the second in 78ms.  The output
images are practically identical.  In reality, JPEG decoding time in GO is linear with
the input image size, but with hinting it becomes constant.
Imagemagick is by no means a fast image processing library.  vips
(http://www.vips.ecs.soton.ac.uk/) is far more performant and offers the same feature.

[rsc]

Comment 2:

Labels changed: added release-none.

[wathiede]

Comment 3:

I am attempting to solve the JPEG specific "faster resize on load" in an unorthodox way
here:
https://camlistore.org/r/1217
By piping to the djpeg command which can scale on load according to:
http://jpegclub.org/djpeg/
Another feature that my be nice to lump into this thinking is how to handle the
progressive JPEG case, as touched on here:
http://golang.org/src/pkg/image/jpeg/scan.go#L252

[gopherbot]

Comment 4 by richard.scothern:

Interesting.  What is 'scale on load'?
Do you have performance numbers?

[wathiede]

Comment 5:

'Scale on load' means that djpeg ignores some of the DCT data while decompressing the
image.  For a 256x256 image with a scaling factor of 2, djpeg will decompress the image
directly into 128x128.  This is probably what the size hint enables in imagemagick. 
This reduces the peak memory usage (128*128*3 instead of 256*256*3) and also gives you a
smaller input image to your resize function, assuming your goal is to generate an image
that isn't exactly 1/2, 1/4 or 1/8 the original size.  The cr2_test.go benchmarks in the
linked changelist test just that.  They take an CR2 image you provide, which has a large
thumbnail in it (5616 x 3744 for the numbers below), decode and resize to 128x128.  It
compares the time for decoding and resizing entirely in Go with BenchmarkStdlib, and
also decompressing by piping to djpeg with a sample factor of 1, 2, 4, and 8 before
resizing down to 128x128.  The results are:
$  go test -test.bench 'Stdlib|Djpeg'                                                   
                                       
PASS
BenchmarkStdlib        1        1677506024 ns/op
BenchmarkDjpeg1        1        1426977483 ns/op
BenchmarkDjpeg2        5         428295667 ns/op
BenchmarkDjpeg4       10         156096810 ns/op
BenchmarkDjpeg8       20          81281665 ns/op
ok      camlistore.org/pkg/images       13.014s
Comparing the results of BenchmarkStdlib vs BenchmarkStdlib is basically comparing the
pure-Go CR2 thumbnail decoder versus piping JPEG bytes to the djpeg subprocess and
reading and parsing the PNM data returned.
The BenchmarkDjpeg[248] cases are the more interesting ones.  Assuming your target image
size is smaller than 1/2, 1/4 or 1/8 of your original image size, you can see
significant improvements in resize time.

[griesemer]

Comment 6:

Labels changed: added repo-main.

[yonderblue]

Having the existing decoders support decoding into an interface with At(x,y,color) would be great.

[kalexmills]

This is very necessary for gamedev. Those of us using https://ebitengine.org could make use of this.

[m8rge]

Guys, I've translated part of djpeg to go, covered it with tests https://github.com/m8rge/go-scaled-jpeg. Borrowed some code from go
Baseline jpeg are have memory and speed increase. Progressive jpeg case not covered.