Add a layer and filter interface in the 2D canvas

[graveljp]

This adds new beginLayer and endLayer functions to open and close layers
in the canvas. While layers are active, draw calls operate on a separate
texture that gets composited to the parent output bitmap when the layer
is closed. An optional filter can be specified in beginLayer, allowing
effects to be applied to the layer's texture when it's composited its
parent.

Fixes #8476

• At least two implementers are interested (and none opposed):
  • Chromium
  • Gecko
  • WebKit
• Tests are written and can be reviewed and commented upon at:
  • https://github.com/web-platform-tests/wpt/tree/master/html/canvas/element/layers
  • https://github.com/web-platform-tests/wpt/tree/master/html/canvas/offscreen/layers
• Implementation bugs are filed:
  • Chromium: crbug.com/1396372
  • Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1866551
  • WebKit: https://bugs.webkit.org/show_bug.cgi?id=259494
• MDN issue is filed: …

(See WHATWG Working Mode: Changes for more details.)

---

/acknowledgements.html ( diff )
/canvas.html ( diff )
/index.html ( diff )
/infrastructure.html ( diff )

[Kaiido]

Impressive work on this.
I still have some concerns about the layer rendering state being picked from the global settings instead of being defined along the filter as a parameter, and about the "when a canvas is presented" concept, but otherwise it all seems to make sense, I think.

Also, I've been a bit quick over the CanvasFilters part as I suppose it's mostly what had already been reviewed on #6763 by Aaron.

[Kaiido]

"presented" is unfortunately a bit unclear as a concept, should it be clearly defined?

[graveljp]

I agree, this has actually caused me some confusion as I was trying to understand how the canvas gets consumed by the event loop. I added a "concept-canvas-presented" definition above to clarify this.

[Kaiido]

Great.
Hopefully if/when the "update the rendering" steps are rewritten to finally match the implementations we'll be able to add a line about that in there too?
My only remaining concern about this would be about other canvas consumers that don't go through the "check the usability of the image" algo. On the top of my head I can only think of the mediacapture-fromelement (canvas.captureStream()), where they never defined when the canvas frame should be moved to the stream.
I guess having this definition would help them to specify the apparent implementation which seems to be to wait for the next "update the rendering". (c.f. w3c/mediacapture-fromelement#94 and linked).

[Kaiido]

Given that reading operations do throw if a layer is open, this leaves only the ill-defined "when the canvas is presented" case that could reach it, right?

[graveljp]

That was the intent. But I removed this section. The CanvasRenderingContext2D now has a top level output bitmap and a current output bitmap. The top level output bitmap can always be presented to the user even when layers are opened. When layers are opened, draw operations are performed in a separate bitmap via current output bitmap.

[Kaiido]

So this doesn't restore any of the bitmaps, right? This means that if we have something like

ctx.globalAlpha = 0.2;
ctx.beginLayer();
ctx.fillRect(0, 0, 50, 50);

The rectangle will keep getting more opaque every time the canvas is "presented". Which is... every frame while it's visible on the page? Or every time the page has to be repainted? Or something else?

[graveljp]

This section has been removed.

The idea was that when presenting a frame, all the layers would have been closed, the draw calls would have been rasterized, and then, all layers would have been re-opened. This algorithm was describing the logic for re-opening the layers. Only the states were to be re-populated, not the draw calls. On later frame, only new draw calls would get drawn.

This was all replaced with an alternative strategy where unclosed layers are never presented. Instead, only the draw calls up to the first beginLayer() gets rasterized. The content of the layer is preserved for the next frame, where layers can be closed and rasterized as a whole.

[Kaiido]

This should probably exclude the canvas layer state.

[graveljp]

This one is tricky to phrase. Let me know what you think.

[domenic]

Will/should PaintRenderingContext2D also get this mixin?

[graveljp]

It should probably. Since it's spec'ed in a different place, I can file a ticket to track this separately. Would that make sense?

[tuankiet65]

Resetting a context should also close/discard all currently opened layers right?

[graveljp]

Yes. But I think that's already covered in step 3 and 4 below:

"To reset the rendering context to its default state:
....
3 - Clear the context's drawing state stack
4 - Set the context's layer-count to zero."

Isn't it?

[Kaiido]

But that should be done first. Here canvas's bitmap in step 1 may still be an alias to an open layer, and stating clearly what happens to pending layers may be a good thing too.

[graveljp]

I was interpreting "canvas's bitmap" to be the "canvas element's bitmap" (for element canvas), which is different than the "context output bitmap". For instance, in section 4.12.5.1.1, we have:
"The top level output bitmap of a rendering context, when it has one, is always just an alias to a canvas element's bitmap."

But you are right that it whould be made clear that the "context's current output bitmap", which changes when layers are opened, should be restored to point to the canvas element bitmap.

[tuankiet65]

I interpret step 3 and 4 as: literally clear the drawing state stack and not unwinding the stack (like say, if the drawing state stack is a stack variable, just call some clear() method to clear it), and set the layer-count variable to 0 without unwinding opened layers. I think for clarity purposes, it'd be helpful to explicitly specify it like "close all opened layers", or "while layer count is not zero, perform the steps in endLayer()" or even "discard all opened layers".

[graveljp]

The thing is, closing all layers is redundant if you are to clear the state stack and reset all states to their default value. In practice, implementations would not close all layers, compositing up all layer outputs and applying filters, just to discard everything in the end. The current spec already uses the step 3 (clear the context's drawing state stack). It's not saying "call restore() in a loop until the state stack is empty.

Maybe I can add a non-normative note saying that these steps have the effect of closing all layers?

[Kaiido]

There would be an observable difference though: flushing the layers (or calling endLayer) would update the origin-clean flag of the canvas bitmap, while if you don't something like the below snippet would not taint the canvas:

ctx.beginLayer()
ctx.drawImage(crossOriginImage, x, y)
ctx.reset()

I'm not quite sure what security risks this would involve (since the cross-origin resource would technically never have touched the bitmap), but that would be observable.

(Ps: Testing on the current Canary's implementation the above snippet does taint the canvas, so somehow at least part of the flushing might be done?)

[graveljp]

I added a bullet point specifying that the current output bitmap has to be reset to point to the top level output bitmap.

@Kaiido, thanks for pointing out the origin-clean issue. You are correct that closing all layers would give a different result than just resetting the state stack.

The reset() spec should actually say that origin-clean must be set to true. The context does reset the origin-clean flag if the canvas is reset by a size change:

“The flag can be reset in certain situations; for example, when changing the value of the width or the height content attribute of the canvas element to which a CanvasRenderingContext2D is bound, the bitmap is cleared and its origin-clean flag is reset.”
https://html.spec.whatwg.org/multipage/canvas.html#security-with-canvas-elements

[Kaiido]

Oh, I never noticed that sentence before, but I agree with your reading.
FWIW, this is not interoperable today, only Chrome does reset the flag when changing the canvas size. Can probably be left for another issue.

[graveljp]

I uploaded a commit addressing all review comments. I also made these important changes:

• beginLayer now accepts CSS filter strings.

• The CanvasRenderingContext2D now has a "top level output bitmap" and a "current output bitmap". The "top level output bitmap" can always be presented to the user, even when layers are open. The content of layers only gets rasterized to the "top level output bitmap" when all layers are closed. This is done by having layers replace and restore the "current output bitmap" to which context draws.

[graveljp]

It should probably. Since it's spec'ed in a different place, I can file a ticket to track this separately. Would that make sense?

[graveljp]

Layers behave the same as other draw calls for {alpha: false} contexts. The layer content and the layer itself is alpha-blended up to the point where it gets written to the top level output bitmap. I updated the paragraph below to clarify this, and added WPT tests validating this (https://crrev.com/c/5006000).

[graveljp]

This section has been removed.

The idea was that when presenting a frame, all the layers would have been closed, the draw calls would have been rasterized, and then, all layers would have been re-opened. This algorithm was describing the logic for re-opening the layers. Only the states were to be re-populated, not the draw calls. On later frame, only new draw calls would get drawn.

This was all replaced with an alternative strategy where unclosed layers are never presented. Instead, only the draw calls up to the first beginLayer() gets rasterized. The content of the layer is preserved for the next frame, where layers can be closed and rasterized as a whole.

[Kaiido]

They would be applied to filters applied via beginLayer.

That's one more surprising behavior and I don't see where it would come from in the current PR.
In #9537 (comment) you stated that it's how CSS and SVG work, but there, it's not only the filters that are affected by the transform, but the whole coord system. For instance in your example the default font-size has also been scaled by 3. If I'm not mistaken, the equivalent for layers would be to have the current transform affect how the whole layer is rendered, not just the filter.

I'll look at spec'ing and implementing BeginLayerOption with alpha, compositeOperation, filter and shadowXXX.

To clarify my personal opinion further, I believe we should have either BeginLayerOption, either have the context's current rendering states affect the layer's rendering, but not both. Having both will inevitably lead to confusion as to which states will apply, in which order, additive or replacing, etc.

[graveljp]

That's one more surprising behavior and I don't see where it would come from in the current PR.

You are right that the current PR doesn't fully correctly this.

To avoid resampling layer's output, which would be slow and lower image quality, the current PR keeps the transform global and doesn't reset it when entering layers. Thus, draw calls write pixels at the position they will have in the final raster. Because there is no concepts of layer-local transforms and because ctx.filter doesn't observe the current transform, it follows that ctx.filter never observes the transform irrespective of whether it's nested in a layer. What's missing is for the spec to mention that the transform should apply to the filter in BeginLayerOption.

I think that in the above, the surprising behaviour is that context filters inside a layer are not affected by the transform in the parent layer. When scaling a whole layer, the content of the layer in all it's details should scale the same way and that should include ctx.filter. Maybe we need to add a concept of parent transform, which would be applied to ctx.filter and ctx.shadowXXX. If we did this, then indeed BeginLayerOptions would be equivalent to the context state plus a save/restore pair. BeginLayerOptions would then be just a syntactic sugar allowing a safer and nicer usage pattern.

You have mentioned the idea of resetting the transform inside layer. It's an interesting idea, but I don't know how we could spec that without requiring layer resampling. It looks like we would need to keep a global transform regardless, to draw pixels at their intended final position, and keep an "artificial" layer-local transform for the only purpose of having getTransform return a layer-local matrix. This adds complexity and disaligns the API with how the canvas really works, for little added value.

[graveljp]

Making the transform layer-local (resetting it when we enter a layer) might have another advantage. If the transform is always global, calling setTransform from within a layer is really strange.

If a non-invertible matrix is used, nothing can get drawn to the canvas:

ctx.scale(0, 0);
ctx.fillRect(0, 0, 10, 10);  // Draws nothing.

This should apply to layers too:

ctx.scale(0, 0);
ctx.beginLayer();
  ctx.fillRect(0, 0, 10, 10);
ctx.endLayer()  // Draws nothing.

But what happens if setTransform is called from within the layer? If the transform is global, we have:

ctx.scale(0, 0);  // Non-invertible matrix.
ctx.beginLayer();  // The whole layer is non-rasterizable.
  ctx.fillRect(0, 0, 10, 10);  // Can't be drawn, the transform is still non-invertible.
  ctx.setTransform(1, 0, 0, 1, 0, 0);  // Restores the matrix to identity.
  ctx.fillRect(0, 0, 10, 10);  // Unclear what happens here.
ctx.endLayer();

If the transform is layer-local, we have:

ctx.scale(0, 0);  // Non-invertible matrix.
ctx.beginLayer();  // The whole layer is non-rasterizable.
  ctx.fillRect(0, 0, 10, 10);  // Can be drawn in the layer (but implementations could optimize away if wanted).
  ctx.setTransform(1, 0, 0, 1, 0, 0);  // No-op, the layer transform is already identity.
  ctx.fillRect(0, 0, 10, 10);  // Same: can be drawn.
ctx.endLayer();  // Draws nothing to the top level output bitmap.

I think we could spec that by maintaining a parent transform (already needed for ctx.filter), a layer transform, and then having all draw calls use parent transform x layer transform to transform strokes and images.

Would that work?

[graveljp]

Hi all,

I uploaded a few updates to this PR following to the above discussions. The changes since the last update include:

• ctx.filter and the current transformation matrix are now part of the the layer rendering states. This means that they are both reset to their default value when entering a layer.
• As before, filters and shadows ignore the layer's current transform, but they are now impacted by the transforms of parent layers. This means that if you scale a layer containing a shadowed shape, the shape and the shadow will scale together.
• I have removed the BeginLayerOptions dictionary, meaning that beginLayer() no longer has a parameter. Filters can now only be specified via ctx.filter.

The current transformation matrix change was tricky to do. On the one hand, shadows and filters are no longer global, so we need to keep track of the coordinate spaces in which they are drawn. On the other hand, we have concepts like the current default path which stores coordinates pre-transformed with with the global matrix. To make everything work, the spec now defines three different types of matrices:

• The current transformation matrix: the matrix local to the current layer. This is the matrix manipulated by the transforms API.
• The parents transformation matrix: the multiplication of the current transformation matrix of all ancestors of the current layer. This is the matrix applied to the shadows and filters.
• The total transformation matrix: the multiplication of all matrices, equivalent to multiplying parents and the current transforms. This is the matrix applied to shapes, images and paths.

To apply the shadow on pre-transformed shapes like the ones using the current default path, I added an algorithm for applying a filter or shadow "in layer coordinate space".

Please review these latest changes and let me know if there's any issue.

[Kaiido]

I still haven't had enough time for a thorough review, but regarding the new total transformation matrix property, instead of being an actual object that is set, it's live and always the result of the multiplication of both other matrices, right?
Seems a bit unusual to my non-editor's eyes, but I'll let @domenic confirm it can work like this.

Otherwise at a glance it looks good (modulo some typos), but I'll try to take the time to think of all the edge cases.

[graveljp]

regarding the new total transformation matrix property, instead of being an actual object that is set, it's live and always the result of the multiplication of both other matrices, right?

That's the idea. In the PR, I wrote:

The <span>total transformation matrix</span> [...] is always equals to the results of
multiplying the <span>parents transformation matrix</span> with the <span>current transformation
matrix</span>.

Otherwise at a glance it looks good (modulo some typos), but I'll try to take the time to think of all the edge cases.

Thanks!