Clipping Mask Column (Vector based)

[manongjohn]

This PR re-enables the Mask column feature found originally in Digital Video's "The TAB Pro" software which converts any Vector column into a real time Clipping Mask for the column that is immediately above/right of it. There is no need to enable Preview mode to use it.

Clipping masks normally works with Panels -> Export . This also provides the initial attempt at rendering clipping masks via the Fx rendering logic (Render -> Preview/Render).

Enable the option by going into the Additional column settings dropdown:

• Clipping Mask by default will cut out everything outside the masked area (Matte In)
• Invert Mask will cut out everything inside the masked area (Matte Out)
• Render Mask will render the image in the column as well as act as a clipping mask for the next column (Visible Matte In/Out)

Clipping Mask columns are currently indicated by a small icon in the upper right corner of the thumbnail area that will take the color of the Filter color if enabled.

Notes:

• Masking is based on vector fills, not lines.
  • The vector lines, themselves, are not included as part of the clipping logic.
  • The clipping is actually based on the center of the line, if the line has any thickness.
  • Opacity has no impact to how clipped images look
• Clipping masks will work against Vector, Smart Raster, Raster and Sub-scene columns.
• Mesh columns will not block Clipping Masks if between the mask and the clipped column.
• Multiple clipping mask columns against the same column are allowed.
  • They need to be next to each other (see image above)
• Clipping mask columns can be animated as normal.

[artisteacher]

This is great - such a helpful and long awaited feature.

So far, I've found a glitch where vector text loses the fill when used as a clipping mask:
The layer being masked:

The mask:

The result:

[artisteacher]

Sub-scenes look like they are masked/clipped in T2D but it renders without the masking.

Love that it works when using the plastic tool on the mask layer. It would be even better if the mesh layer's default location/layer order didn't interfere with the masking. It's not a big deal moving it, but I could see this being an area that causes confusion.

[artisteacher]

Actually, it looks like the stroke is excessively thick on text at smaller sizes, so some fonts look like they are solid fills, but are mostly stroke at size 70 or etc. This is behaving properly but doesn't look like you'd expect based on how the text looks. Personally I think the line thickness for text is excessive and doesn't usually look the best. I often use the selection tool's pump widget to fix the appearance.

I did get a crash trying to type on a vector layer with clipping mask turned on. The first time I typed and created a letter it was fine, but then I switched fonts, typed a letter, and clicked to create the text and T2D crashed. I had also recently deleted a mesh layer.
mask-text-crash.txt

[manongjohn]

Unfortunately I have to redo the render logic as I've discovered an issue with how it's generating the image to clip. It pre-renders an image including all other column images in the frame, not just the clipped column. This becomes an issue when Fx are applied to any of the images, especially the clipped column.

[manongjohn]

Got the original issues I was seeing worked out. Currently works for the 3 main level types. It will ignore the mesh column if it's between the level and the clipping mask.

Trying to get it to work for sub-scenes and column based fx (zeraryfx), like checkerboard etc. I have it working for sub-scenes (not included in PR yet) but it doesn't render with any fx applied to the individual columns. Unfortunately I will likely need to rewrite my logic again :(

Feel free to play around with it for a while and see if any other issues pop up.

[manongjohn]

Ok. I've revised the logic so it now renders sub-scenes correctly.

Reviewing the various column based zeraryFx, I've decided to hold off on making any of them maskable and will move forward with what I have.

For those who have time to try it out, please let me know of any issues.

[artisteacher]

Loving the updates! This is amazing. I do think I am getting more crashing than normal when navigating the Timeline. I'll have more crash reports later from school, but here's one to start:
timeline-crash.txt

[manongjohn]

Added a fix to an issue with undo/redo of mask settings. Squashed the commits down to 1 and updated the PR's description in preparation to merge.

If anyone wants to do a little more testing/sanity check, please do. Will likely merge in a day or so.

[beeheemooth]

I'm getting the following problem: after the collapse of the columns, I go inside the sub-scene and when I try to exit the sub-scene, the program freezes. Sometimes when exiting a sub-scene succeeds, for a moment I see the masked columns flicker. I am attaching the scene file.

mask_test.zip

[beeheemooth]

I disabled the masks in the column settings and entering / exiting the sub-scene happens without problems

[beeheemooth]

I tried creating masks inside sub-scene. The problem reoccurs when I make a mask for the fingers column.

[manongjohn]

Can't seem to reproduce it

[artisteacher]

I'm getting a crash on open for the action build.

Seems to be yet another library issue.

[manongjohn]

Seems to be yet another library issue.

Strange. Does this mean the nightlies are broken for you also? If so, i wonder when it broke.

[manongjohn]

I'm getting the following problem: after the collapse of the columns, I go inside the sub-scene and when I try to exit the sub-scene, the program freezes. Sometimes when exiting a sub-scene succeeds, for a moment I see the masked columns flicker. I am attaching the scene file.

Found a memory leak in vector drawing logic that is likely causing this. I get a crash when I quickly and repeatedly go in/out of sub-scene. On your system it seems you lock up instead of crashing.

I don't think this bug specific to masks logic, though. You might have just managed to create a vector scenario the makes the issue somewhat repeatable. I will try and fix it with this PR.

I'm getting a crash on open for the action build.

I think I see why the macOS action builds are failing. When putting together the dmg file, my script suddenly appears to be failing to update the library paths in some dylib files to point to the version within Tahoma2D.app. Since the dylib path hasn't changed, it's trying to find it on the system and failing.

[manongjohn]

Ok...I think I finally got this working right now.

I've thrown away the hack-ish logic I had and have hopefully integrated it properly into the existing Render logic. It should handle all the animations, parenting and fx correctly now.

Please play with it. It's still up in the air if it will be included in 1.4, but I think it's a lot closer to being ready for release.

[artisteacher]

With the latest update + macOS, I've just noticed two issues so far:

1. Caching with preview - if you change the mask to be inverted or etc., the viewer's preview doesn't recalculate, though the render is fine
2. Using the plastic tool on the clipping mask looks like it works, but doesn't render/preview properly - it is either blank, or only displays the mask (if render mask is checked)

[manongjohn]

@artisteacher Both issues should now be fixed.

[manongjohn]

I know there have been a few people playing with this and I believe I've fixed any issues mentioned here or on Discord. I haven't heard anything since my last fix, so I'm hoping this is a good thing? 😁

Unless someone shows me something that I feel is a potential issue, causing users wasted time for something not working right, I'm strongly inclined to merge this sometime tomorrow.

Let me know.

[beeheemooth]

@manongjohn It looks like the Tahoma2D Windows Build/Windows archive (pull_request) does not contain all the files. Here's what I unpacked:

[manongjohn]

@beeheemooth Thanks for letting me know. I've forced the Windows version to rebuild and see if it corrects itself.

[manongjohn]

Seems there is an issue with meshes applied to subscenes which have masks inside them. The Viewer doesn't display the results of the masked column, though it does show in Preview/Render correctly.

Investigating, though I'm not sure I will hold up including this in 1.4 because of it since the workaround would be to turn on Preview.

[artisteacher]

I'm still seeing the preview not updating when the clipping mask settings are changed - ie. invert turned on. On frames that weren't originally calculated, it looks fine. On frames that were, they don't update.

[manongjohn]

@artisteacher I just downloaded and tested the macOS artifact in my VM and it does update the Preview when I change Invert or Render clipping mask options. Could you be running an older artifact?

[artisteacher]

It must have been an older artifact, the preview issue is definitely fixed now.

[artisteacher]

I did just find, that if there is a mask in the sub-scene, it is ignored when creating a mesh for the sub-scene:
Inside the sub-scene:

When creating a mesh:

As you can see, the mesh seems to be factoring in the full clipped level, rather than just what should actually be visible. I also went back in & erased part of the raster level to be sure that was the case & not a recurrence of the rectangular mesh thing that happens on some systems.

[manongjohn]

I did just find, that if there is a mask in the sub-scene, it is ignored when creating a mesh for the sub-scene

Oops. Seems my last change only applied to Windows and Linux builds. Unfortunately when I tried to extend my changes to macOS builds, it failed to work. Need to find a separate solution for macOS.

[artisteacher]

That figures! Funny that Linux is different.

[manongjohn]

@artisteacher Should be fixed now. MacOS has special handling for meshes on sub-scenes for some reason and wasn't handling masking at all, but I figured out what was missing and extended the Windows/Linux fix to it and works now. Let me know if you still have problems with it.

[artisteacher]

Yes, it fixes it. Thank you!

[manongjohn]

Thanks for verifying. I have a lot more confidence in merging this new feature in for 1.4.....so I guess I will. :)

Thanks to all for your patience as I worked to get this feature right and especially to those who helped with testing to make sure it is.

[Dixxi0509]

This PR re-enables the Mask column feature found originally in Digital Video's "The TAB Pro" software which converts any Vector column into a real time Clipping Mask for the column that is immediately above/right of it. There is no need to enable Preview mode to use it.

Clipping masks normally works with Panels -> Export . This also provides the initial attempt at rendering clipping masks via the Fx rendering logic (Render -> Preview/Render).

Enable the option by going into the Additional column settings dropdown:

• Clipping Mask by default will cut out everything outside the masked area (Matte In)
• Invert Mask will cut out everything inside the masked area (Matte Out)
• Render Mask will render the image in the column as well as act as a clipping mask for the next column (Visible Matte In/Out)

Clipping Mask columns are currently indicated by a small icon in the upper right corner of the thumbnail area that will take the color of the Filter color if enabled.

Notes:

• Masking is based on vector fills, not lines.

  • The vector lines, themselves, are not included as part of the clipping logic.
  • The clipping is actually based on the center of the line, if the line has any thickness.
  • Opacity has no impact to how clipped images look

• Clipping masks will work against Vector, Smart Raster, Raster and Sub-scene columns.

• Mesh columns will not block Clipping Masks if between the mask and the clipped column.

• Multiple clipping mask columns against the same column are allowed.

  • They need to be next to each other (see image above)

• Clipping mask columns can be animated as normal.

I like this idea!