New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Summary of the wide gamut support on Canvas #4167

Open
fserb opened this Issue Nov 9, 2018 · 21 comments

Comments

7 participants
@fserb
Contributor

fserb commented Nov 9, 2018

This is an IDL summary of the new proposed interface to support wide gamut on Canvas (half-float + extended SRGB)

2D Context

enum CanvasPixelFormat {
  "uint8", // default
  "float16",
};

dictionary CanvasRenderingContext2DSettings {    
  boolean alpha = true;    
  CanvasPixelFormat pixelFormat = "uint8";
};

dictionary CanvasContextCreationAttributesModule {    
  // Canvas 2D attributes    
  boolean alpha = true;  // Also used for WebGL.    
  CanvasPixelFormat pixelFormat = "uint8";
};

ImageData createImageData(unsigned long sw, unsigned long sh, 
    ImageDataColorSettings imageDataColorSettings);

ImageData createImageData(ImageDataArray data, unsigned long sw, unsigned long sh, 
    optional ImageDataColorSettings imageDataColorSettings);

ImageData

enum ImageDataStorageType {
  "uint8", // default
  "uint16",
  "float32",
};

typedef (Uint8ClampedArray or Uint16Array or Float32Array) ImageDataArray;

dictionary ImageDataColorSettings {
  ImageDataStorageType storageType = "uint8";
};

[
  Constructor(unsigned long sw, unsigned long sh, 
    optional ImageDataColorSettings imageDataColorSettings),
  Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, 
    optional ImageDataColorSettings imageDataColorSettings),
  Exposed=(Window,Worker)] interface ImageData {
  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  readonly attribute ImageDataArray data;
  
  ImageDataColorSettings getColorSettings();
};

ImageBitmap

enum ImageBitmapPixelFormat { "default", "uint8" };

dictionary ImageBitmapOptions {
  ImageBitmapPixelFormat imagePixelFormat = "default";
};

convertToBlob

enum ImagePixelFormat {
    "uint8", // default
    "uint16",
};

dictionary ImageEncodeOptions {
    DOMString type = "image/png";
    unrestricted double quality = 1.0;
    ImagePixelFormat pixelFormat = "uint8";
};
                                                                                
[HTMLConstructor]
interface HTMLCanvasElement : HTMLElement
{
  Promise<Blob> convertToBlob(optional ImageEncodeOptions options);
};
@fserb

This comment has been minimized.

Contributor

fserb commented Nov 9, 2018

@fserb

This comment has been minimized.

Contributor

fserb commented Nov 12, 2018

Some clarification notes, that came up while talking to @annevk:

  • when a float16 canvas is created, we default to extended-SRGB color space
  • we need a defined behavior for putImageData of float16 to uint8: clamp from 0 to 1.
  • there's no known primitive in web platform to specificy this float16 backing storage. We are up to suggestions of better alternatives.
@jrmuizel

This comment has been minimized.

jrmuizel commented Nov 12, 2018

I haven't really followed all of the previous discussion about this kind of extension. Can you explain how this fits into the context of w3ctag/design-reviews#315, #299 and https://github.com/WICG/canvas-color-space/blob/master/CanvasColorSpaceProposal.md?

@fserb

This comment has been minimized.

Contributor

fserb commented Nov 12, 2018

This is a summary of the IDL changes proposed by w3ctag/design-reviews#315.

This is also a subset of the original proposal, that included color spaces specification and that was discussed on #299. So those two are a superset of what's being proposed here.

@jrmuizel

This comment has been minimized.

jrmuizel commented Nov 12, 2018

Can you provide the use cases that this solves along with the rationale for the decisions made relative to other options?

@fserb

This comment has been minimized.

Contributor

fserb commented Nov 14, 2018

The high level goal is to allow canvases to take advantage of full color gamut available on display devices.
Example use cases:

  • support applications that are wide gamut sensitive like medical imaging
  • add a first step of support towards better photo editing / graphics on the web

A fully formed color space spec would include this (wide gamut) plus color spaces support (P3, rec2020, etc... and maybe a SRGB mode that doesn't do linear blending of non-linear values).

That said, it is our understanding that even that fully formed spec would need a wide-gamut mode that is backward compatible with the current SRGB support. This spec is this minimal color space support (extended SRGB) and wide gamut support on canvas and canvas export API.

@jrmuizel

This comment has been minimized.

jrmuizel commented Nov 14, 2018

Do you envision the entire canvas pipeline running at float16 when CanvasPixelFormat is set to "float16"?

What is the functionality that those applications need from canvas? Naively, I can't think of much overlap between the canvas api and the needs of a photo editor.

@fserb

This comment has been minimized.

Contributor

fserb commented Nov 15, 2018

I'm not 100% sure what you mean by "the entire canvas pipeline", but if you mean the backing storage for both 2D and WebGL and the drawing commands for 2D, then yes.

The main 2D commands that are impacted by this, as it stands now, are drawImage of wide gamut content (video, images, etc) and rendering of gradients.

In general, for any type of drawing (and some game) application color fidelity (and representing colors outside of sRGB) is a big issue.

Granted, the case strictly for 2D is a bit reduced, although we do have cases of people doing WebGL -> 2D canvas for interface, in which we would like to preserve the quality of the rendered image.

@jrmuizel

This comment has been minimized.

jrmuizel commented Nov 15, 2018

CoreGraphics doesn't seem to support a 16bit float pixel format. Do you know what Apple's feedback on this proposal is?

@ccameron-chromium

This comment has been minimized.

ccameron-chromium commented Nov 16, 2018

There was general consensus about this being the way forward at 1-year-ago's TPAC. There is definitely support in macOS for half-float IOSurfaces through the whole pipeline (I haven't locally tried in CoreGraphics myself)

@fserb

This comment has been minimized.

Contributor

fserb commented Nov 16, 2018

(cc @smfr, I think)

@smfr

This comment has been minimized.

smfr commented Nov 28, 2018

CoreGraphics doesn't seem to support a 16bit float pixel format. Do you know what Apple's feedback on this proposal is?

CG does support half-float pixel formats on both its GPU-accelerated and non-accelerated code paths, so we support this proposal.

@annevk

This comment has been minimized.

Member

annevk commented Nov 28, 2018

But what kind of backing store would you use that can be exposed to JavaScript? JavaScript doesn't have float16 so you'd have to do conversion at the boundary, no?

@kenrussell

This comment has been minimized.

Contributor

kenrussell commented Nov 29, 2018

@annevk per the IDL above, Float32Array will be used to represent the pixels during getImageData and putImageData operations when the backing store is float16. These aren't the fastest way to use the canvas APIs anyway, so a conversion is fine. WebGL already uses Float32Array for uploading to float16 format textures and reading back float16 format framebuffers.

@svgeesus

This comment has been minimized.

svgeesus commented Nov 29, 2018

@fserb wrote:

when a float16 canvas is created, we default to extended-SRGB color space

Where is the extended sRGB space defined? What is the transfer function? What is the range of the function? In other words, what do values less than 0 or greater than 1.0 (255/255, in 8bits) mean?

@annevk

This comment has been minimized.

Member

annevk commented Nov 30, 2018

@kenrussell what is the fastest way? And why don't we add Float16Array if we have several use cases?

@annevk

This comment has been minimized.

Member

annevk commented Nov 30, 2018

By the way, related to @svgeesus's question, are we ensuring consistency with CSS and SVG somehow?

@kenrussell

This comment has been minimized.

Contributor

kenrussell commented Nov 30, 2018

@annevk the fastest Canvas 2D APIs are the drawing operations (drawImage, rectangle/line/text drawing), not per-pixel manipulation (getImageData/putImageData).

Adding Float16Array to JS would require a large amount of hand-tuned assembly language to be added to every JS engine in existence. float16 is a native data type on GPUs but not on today's CPUs, and is handled with a library in every other language. I think it should be handled with a library in JS, too, and such libraries have already been written: https://github.com/petamoriken/float16 .

@svgeesus

This comment has been minimized.

svgeesus commented Dec 4, 2018

@annevk

By the way, related to @svgeesus's question, are we ensuring consistency with CSS and SVG somehow?

I plan to, yes; primarily by following/commenting on this and making sure that CSS Color 4 and canvas stay aligned wrt Wide Color Gamut and High Dynamic Range.

@annevk

This comment has been minimized.

Member

annevk commented Dec 5, 2018

@kenrussell thanks, could you perhaps also address #4167 (comment)? And how would out-of-range values be handled, e.g., when going from float32 to float16?

@svgeesus that's great to hear. CSS Color 4 also seems like the ideal place to describe the WCG/HDR abstract model (the mapping of float16 values to colors, if you will).

@kenrussell

This comment has been minimized.

Contributor

kenrussell commented Dec 6, 2018

@fserb is the best person to describe the extended sRGB color space; that's outside my domain.

float32 values can be defined in the spec as being converted to float16 internally with similar rules to how double (float64) is converted to float (float32) in C/C++: https://stackoverflow.com/questions/16737615/how-is-floating-point-conversion-actually-done-in-cdouble-to-float-or-float .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment