image, image/draw: add interfaces for using RGBA64 directly #44808
Description
I have an application that involves calculating image data at extremely high resolutions, sometimes on the scale of gigapixels, then resampling the results to smaller sizes. Using package x/image/draw to perform the resampling gives excellent results, but rather slowly. One significant reason for this is that every call to At allocates, because the result type is an interface. A quick benchmark shows that this is responsible for 80% of the allocations in my application.
Package x/image/draw already contains specialized fast paths for many of the raw image types in package image. However, because my images are calculated by histogramming sometimes trillions of iterations of a chaotic process, I cannot use those types to accumulate the image. I also cannot copy to such a type prior to rescaling, because the images I'm working with already occupy most of the memory I have available.
In order to resolve this, I propose to create a standardized interface in package image that allows image processing routines to define fast paths that can avoid unnecessary allocations. The interface would be as follows:
package image
// RGBA64Image is an image with pixels that can be transformed directly to
// color.RGBA64.
type RGBA64Image interface {
// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
// equivalent to calling At(x, y).RGBA() and converting the resulting
// 32-bit values to color.RGBA64, but it may avoid allocations from
// converting concrete color types to the color.Color interface type.
RGBA64At(x, y int) color.RGBA64
Image
}Additionally, I propose that a corresponding interface be added to package image/draw (with an alias in x/image/draw):
package draw
// RGBA64Image is an Image with a SetRGBA64 method to set the color of a
// single pixel. It is like Image, but using it may mitigate allocations from
// converting concrete color types to the color.Color interface.
type RGBA64Image interface {
SetRGBA64(x, y int, c color.RGBA64)
Image
}I propose that the image types Alpha, Alpha16, CMYK, Gray, Gray16, NRGBA, NRGBA64, Paletted, and RGBA be extended to implement both of these interfaces (noting that RGBA64 already does), and that NYCbCrA, Uniform, and YCbCr be extended to implement the former.
With these interfaces, image processing algorithms in image/draw, x/image/draw, and elsewhere can implement generic fast paths, rather than only having fallbacks for image.Image that force at least an allocation per pixel. An experiment using a modified version of x/image/draw to implement such a fast path for Kernel.Scale shows that resampling an image from 1280x1280 to 128x128 goes from 3296775 to 32775 allocations per operation, a 100x improvement, with approximately a 3x speed improvement. In particular, the primary culprit method in my application disappears completely from the memory profile.
I originally proposed to add an interface with RGBA64At to x/image/draw specifically, rather than to the standard library, because the interface allocations are most impactful when processing extremely large images, and resampling seems like the most likely operation for them. However, I've since talked with others who have had serious performance issues using image/draw for color transformations on many images, and @nigeltao suggested in a comment that these interfaces be added to the standard library.