Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image, image/draw: add interfaces for using RGBA64 directly #44808

Open
zephyrtronium opened this issue Mar 5, 2021 · 13 comments
Open

image, image/draw: add interfaces for using RGBA64 directly #44808

zephyrtronium opened this issue Mar 5, 2021 · 13 comments

Comments

@zephyrtronium
Copy link
Contributor

@zephyrtronium zephyrtronium commented Mar 5, 2021

I have an application that involves calculating image data at extremely high resolutions, sometimes on the scale of gigapixels, then resampling the results to smaller sizes. Using package x/image/draw to perform the resampling gives excellent results, but rather slowly. One significant reason for this is that every call to At allocates, because the result type is an interface. A quick benchmark shows that this is responsible for 80% of the allocations in my application.

Package x/image/draw already contains specialized fast paths for many of the raw image types in package image. However, because my images are calculated by histogramming sometimes trillions of iterations of a chaotic process, I cannot use those types to accumulate the image. I also cannot copy to such a type prior to rescaling, because the images I'm working with already occupy most of the memory I have available.

In order to resolve this, I propose to create a standardized interface in package image that allows image processing routines to define fast paths that can avoid unnecessary allocations. The interface would be as follows:

package image

// RGBA64Image is an image with pixels that can be transformed directly to
// color.RGBA64.
type RGBA64Image interface {
	// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
	// equivalent to calling At(x, y).RGBA() and converting the resulting
	// 32-bit values to color.RGBA64, but it may avoid allocations from
	// converting concrete color types to the color.Color interface type.
	RGBA64At(x, y int) color.RGBA64
	Image
}

Additionally, I propose that a corresponding interface be added to package image/draw (with an alias in x/image/draw):

package draw

// RGBA64Image is an Image with a SetRGBA64 method to set the color of a
// single pixel. It is like Image, but using it may mitigate allocations from
// converting concrete color types to the color.Color interface.
type RGBA64Image interface {
	SetRGBA64(x, y int, c color.RGBA64)
	Image
}

I propose that the image types Alpha, Alpha16, CMYK, Gray, Gray16, NRGBA, NRGBA64, Paletted, and RGBA be extended to implement both of these interfaces (noting that RGBA64 already does), and that NYCbCrA, Uniform, and YCbCr be extended to implement the former.

With these interfaces, image processing algorithms in image/draw, x/image/draw, and elsewhere can implement generic fast paths, rather than only having fallbacks for image.Image that force at least an allocation per pixel. An experiment using a modified version of x/image/draw to implement such a fast path for Kernel.Scale shows that resampling an image from 1280x1280 to 128x128 goes from 3296775 to 32775 allocations per operation, a 100x improvement, with approximately a 3x speed improvement. In particular, the primary culprit method in my application disappears completely from the memory profile.


I originally proposed to add an interface with RGBA64At to x/image/draw specifically, rather than to the standard library, because the interface allocations are most impactful when processing extremely large images, and resampling seems like the most likely operation for them. However, I've since talked with others who have had serious performance issues using image/draw for color transformations on many images, and @nigeltao suggested in a comment that these interfaces be added to the standard library.

@gopherbot gopherbot added this to the Proposal milestone Mar 5, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Mar 10, 2021
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 10, 2021

@nigeltao
Copy link
Contributor

@nigeltao nigeltao commented Mar 11, 2021

Note that the RGBAAt(x, y int) (r, g, b, a uint32) proposal conflicts with the existing func (p *RGBA) RGBAAt(x, y int) color.RGBA in the standard library's package image.

Perhaps add some interfaces to the standard library (and make its concrete image types implement them):


package image

type ImageV2 interface {
    Image
    // AtDotRGBA(x, y) is equivalent to At(x, y).RGBA() but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    AtDotRGBA(x, y int) (r, g, b, a uint32)
}

// or perhaps

type ImageV2 interface {
    Image
    // RGBA64At(x, y) is equivalent to At(x, y).RGBA(), after packing RGBA's
    // four uint32 return values into one color.RGBA64, but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    RGBA64At(x, y int) color.RGBA64
}

package draw

type ImageV2 interface {
    Image
    // SetRGBA64(x, y, c) is equivalent to Set(x, y, c) but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    SetRGBA64(x, y int, c color.RGBA64)
}

golang.org/x/image/draw obviously gets:

type ImageV2 = draw.ImageV2

Yeah, the names are horrible and inconsistent, but we're constrained by Go 1 backwards compatibility...

@nigeltao
Copy link
Contributor

@nigeltao nigeltao commented Mar 11, 2021

@zephyrtronium
Copy link
Contributor Author

@zephyrtronium zephyrtronium commented Mar 11, 2021

@nigeltao
I had forgotten about image.RGBA.RGBAAt. Considering this, the RGBA64At(x, y int) color.RGBA64 interface does seem more appropriate.

Reading through it again, package image also has somewhat of a convention that would suggest a better name. The concrete image types are RGBA, Gray16, &c. whereas the interface types are Image and PalettedImage. So, it would be natural to add RGBA64Image.

Unless there are objections, tomorrow I will retitle and change this proposal to adding

// RGBA64Image is an image with pixels that can be transformed directly to
// RGBA64.
type RGBA64Image interface {
	// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
	// equivalent to calling At(x, y).RGBA() and converting the resulting
	// 32-bit channels to color.RGBA64, but it may avoid allocations from
	// converting concrete color types to the color.Color interface type.
	RGBA64At(x, y int) color.RGBA64
	Image
}

to package image, updating the concrete image types to implement it, adding a corresponding RGBA64Image interface to image/draw and x/image/draw, updating the concrete image types in package image to implement these interfaces, and implementing fast paths in image/draw and x/image/draw using these types.

@nigeltao
Copy link
Contributor

@nigeltao nigeltao commented Mar 11, 2021

Sounds good to me.

With the new fast paths, I'd only bother with:

  • "image.RGBA64Image on draw.RGBA64Image"

and not e.g.

  • "image.Image on draw.RGBA64Image" or
  • "image.RGBA64Image on draw.Image"
@zephyrtronium zephyrtronium changed the title proposal: x/image/draw: add interface for providing pixel RGBA directly proposal: image, image/draw: add interfaces for using RGBA64 directly Mar 12, 2021
@naisuuuu
Copy link

@naisuuuu naisuuuu commented Apr 12, 2021

I was about to submit a separate proposal to add additional fast paths to x/image/draw for more resampling destination types. This proposal would solve that and many more similar problems in a much more elegant way. Really hoping to see it come to fruition!

@nigeltao
Copy link
Contributor

@nigeltao nigeltao commented Apr 16, 2021

@ianlancetaylor as above (March 12), this proposal (affecting the std image package) Sounds Good To Me.

What's the formal process from here? Do I upgrade this issue from Proposals/Incoming to Proposals/Active?

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 18, 2021

Change https://golang.org/cl/311129 mentions this issue: image: add RGBA64Image interface

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 18, 2021

@nigeltao No, that move will be done when the proposal review committee gets to this issue.

I'll add it to the list to consider in the next meeting.

@rsc
Copy link
Contributor

@rsc rsc commented Apr 21, 2021

Seems OK. Adding to minutes.

@rsc
Copy link
Contributor

@rsc rsc commented Apr 21, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals Apr 21, 2021
@rsc rsc moved this from Active to Likely Accept in Proposals Apr 28, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Apr 28, 2021

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc
Copy link
Contributor

@rsc rsc commented May 5, 2021

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc moved this from Likely Accept to Accepted in Proposals May 5, 2021
@rsc rsc changed the title proposal: image, image/draw: add interfaces for using RGBA64 directly image, image/draw: add interfaces for using RGBA64 directly May 5, 2021
@rsc rsc modified the milestones: Proposal, Backlog May 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants