[unity] Unity 2017 SpriteMask support

[pharan]

Unity 2017.1 introduces SpriteMask.
https://docs.unity3d.com/2017.1/Documentation/ScriptReference/SpriteMask.html
SpriteMask is a component that can mask UnityEngine.Sprites in a certain arrangement in the scene graph.

Example of SpriteMask used on UnityEngine.Sprites:

Under investigation:

Q: Does it require new Shader code on the masked object? Does it require component code?
A: SpriteRenderers that use a Spine/Skeleton shader Material seem to be able to be masked.

Q: Can it mask MeshRenderers? (it can mask Particles. so maybe.)
Q: How is its mask scope determined by code?

[pharan]

Doesn't look good so far. The system/API seems pretty enclosed at the moment.

[ZimM-LostPolygon]

It's entirely possible. The masking system uses stencil buffer:
https://forum.unity3d.com/threads/sprite-mask-inner-workings.492410/#post-3211401
Basically, you'll need to modify the shader, that's it. Not sure if this should be included into the spine-unity runtime itself, though.

[pharan]

Thanks for the info! Super helpful.
Also, feel free to chime in on the other Unity 2017 stuff! Any help is appreciated, if you happen to be poking around those areas.

What's weird is that (in the above gif) I was using the unmodified Spine/Skeleton shader on a SpriteRenderer and the masking still functioned correctly. How did that work?

[ZimM-LostPolygon]

Perhaps Unity is doing some broken magic internally if no Stencil section is defined in the shader?

[pharan]

I suppose, under the hood, stencil properties you type in ShaderLab are really just instructions for the UnityEngine's C++ side to call the appropriate graphics library stencil function calls. Maybe there's no reason why their components can't just call those stencil functions regardless of what the ShaderLab code says. No magic.

Anyway... I just added this part Sergi mentioned:

Stencil {
	Ref 1  //Customize this value
	Comp Equal //Customize the compare function
	Pass Keep
}

to the Spine/Skeleton shader.
And it worked.

The Ref 1 is an odd magic number though. He says to customize it, but customize it based on what?

At the very least, we can just share this in a thread in the forums, with the link and description of what changes were made, so the user can apply the same changes to their own shaders. We'll judge the need to include it in the runtime from there.

[pharan]

I'll update this later with a version that exposes the stencil properties as material inspector fields.
https://gist.github.com/pharan/eb758e46630ad7ae2234227480922c21

---

Done

But the shader needs a material inspector to nicely show the compare function.

[ZimM-LostPolygon]

@pharan Problem with this shader is that this way, it will always do the stencil test. There is no way to turn it off except not having the Stencil block. So two shaders and materials are required to cover everything - one with stencil test, and one without.

[pharan]

Oh, definitely. I had no intention of officially adding the stencil test to the default shader in the package.

I was just doing it to poke at stuff and see how the thing works.

[HaraldCsaszar]

Related forum bugreport / request:
Correct masking behaviour of SkeletonGraphic with Mask, but not with RectMask2D:
http://esotericsoftware.com/forum/SkeletonGraphic-and-RectMask2D-9023

[HaraldCsaszar]

Issue with RectMask2D identified as separate issue, now filed separately under #1283.