Implement Layer Masking

[Jose-Moreno]

Issue Summary

Note: The implementation of this feature would be [dependent on task #533 ]

I see the need for two types of masking to cover most use cases:

• Alpha layer-based (compositing) masking -> Proposed name: Stencil Mask
• Individual Layer masking -> Proposed name: Cutout Mask

Alpha based masking

In Krita they also have a separate mode called "alpha inheritance" which is used like "Clipping Masks" in Photoshop. While presentation varies both rely in the compositing result of the alpha values from the "clipping mask" that is inherited to the "clipped" layers.

The problem is that clipping masks in programs like Photoshop allow you to see the original mask, whereas regular masks (alpha based) should only allow the masked element to be seen.

The problem with how alpha compositing is implemented in Krit or Synfig is that you require a layer group to define inheritance scope, however I think that a layer mask should be able to define it's own scope by parenting, basically allowing a layer marked as a mask to choose it's own children, that is, which object layers inherit its alpha.

Flash / animate CC works converting the layer marked as mask into a "parent" or "layer group" styled layer where any layer that is nested will inherit the alpha from the mask layer, however it's main problem was that you had to "lock" the parent and chicld layer to see the masking effect in the editor. Often times you want to have a quick toggle for this.

Krita has the alpha inheritance property which makes a layer that has the "inherit alpha" mode toggle activated becomes a "masked" element. In turn those elements that are not "inheriting alpha" become the masks themselves, so the advantage of this is that you can have multiple, separate, layers become a mask. The problem with this approach is that you always have to have the "mask" elements visible, so if you want only to see the masked element there needs to be an additional approach

In Photoshop a "clipping" group or clipping layer functions the same but in a nested way (The top layer is the CLIP, the nested layers are "CLIPPED" or inheriting the alpha)

In Synfig you use the blending modes (onto / straight onto / alpha over , etc) along the cut out tool which creates a freehand mask when used over any layer entity (in synfig every graphic object is a "layer" btw) what happens in the background though is that a vector shape is created on top of the currently selected layer, this shape is given a blending mode of "Alpha Over" renamed as "mask" and then both layers are "encapsulated" (older terminology) into a Layer Group.

Individual Layer masking

In PS and Krita you can have individual layer masks / transparency masks that will only affect the current layer. This is useful when you are iterating over a design but only need the masked element to be show, not the mask. This is usually accomplished with greyscale value maps, where black hides the masked element and white is see-through

Pencil2D Masking

From the previous examples I see the following requirements:

Alpha layer-base masking

Each layer should:

• Have a toggle to activate / deactivate the masking state
• Have the ability to parent multiple children i.e a single layer mask can effect multiple normal layers
  • Have the ability to parent the children of other masks. i.e each mask layer can have the same children to affect
• Have the masking state scope affect only the parented children instead of layers in the whole canvas
• Allow to add the result of alpha based compositing i.e the masking effect, to any other non-mask layer e.g having more layers with the mask mode enabled will expand the masking effect so it behaves as a global mask within the layer parenting scope

For Pencil2D we'd need a way to make any layer have a mask mode property, allowing it to select which elements can be "masked" instead of making a mask on the whole canvas.

Also consider that allowing vector layers to have this capability would remove the need to have a dedicated, separate, masking tool when you can use a vector layer as a mask, allowing itself to be animated and changed due to their vector nature. Of course for animated vector shapes, both b-spline representation and vector interpolation would need to be implemented to be fully usable in that regard.

Individual layer masking

This one would be heavily based on greyscale value maps. To mask the layers you would use selections or pixel / vector painting. Also it's important to consider the possibility of having vector masks made with a future polyline / bezier curve tool as well.

Ideally the "layer mask" would be an associated but separate appended layer that would only work as a mask. If bitmap layers were used, the pixels would be mapped to greyscale values only, no color. If vector layers were used color would be also mapped to greyscale values. If a selection was used to invoke the "cut out mask" function, the selection would create a vector layer with absolute fully opaque (white) / fully transparent (black) values.

[MrStevns]

It's a feature that's difficult to make simple in usage and understandable without making the workflow involved feel more complex, like with the above applications.

While Adobe Flash is old and hasn't changed much, their way of doing masks works well and in favour of our current way of handling layer management in the timeline.
https://www.youtube.com/watch?v=8GHoAFeTYQc

Photoshop and Krita are not designed around being animation software, it's just stuff that's been integrated down the line and thus they have a use-case for a layer stack, which imo. we do not.

We need to come up with a solution that doesn't add more widgets, nor makes the timeline feel more complex to interact with.

When that's said I agree that layer masking should happen, we just gotta figure out the appropriate way to implement it for pencil2D :)

[Jose-Moreno]

@candyface I agree and it's the reason why I took more time to show several "references" to other software. I forgot to include Moho but it's similar to Synfig since you have to encapsulate two layer entities and make one be the mask and the other one(s) the masked element(s).

I also agree that from all the options, the most "intuitive" in relation to how Pencil2D layer management works is that of Flash's masking, which is also present in TVPaint and to some degree in Harmony; just make a shape in another layer and use that to mask the layer you want to effect. To me that's always been a super easy way to use masks. However since this way of masking is mostly alpha based masking or "stencil" masking and we have to use another shape to either filter or "cut out" a piece in the other elements, we're dependent on another layer always.

The previous thought is what prompted me to suggest the other kind of masking which would be mostly for single layers. I mean you can't use a stencil on itself, so having a way to mask a single layer is also important, however we would not be able to have it animated, and making it animated would still depend on another layer or image sequence to be loaded on the widget thing... So if we can only have stencils "ala" Flash I won't complain at all to be honest Harmony uses their node filter thing which basically plugins other layers so it's the same thing in the end.