Can't render into Anti-aliased texture when back buffer uses stencil buffer

[PrimaryFeather]

Problem Description

It was noticed that when using a mask in Starling, objects with a FragmentFilter suddenly disappear if they have antiAliasing enabled.

• AIR SDK: 27.0.0.128
• Flash Player (Projector, Debugger): 27.0.0.187
• macOS 10.13.1
• Affects both AIR and Flash Player apps
• On iOS, the problem is a little different (see below)
• Not yet tested on Android.
• Adobe Tracker issue AIR-4198507
• This problem was first reported in this Starling issue.

Steps to Reproduce

You can reproduce this with the following simple Starling code:

var victim:Quad = new Quad(50, 50, 0xff);
victim.x = victim.y = 200;
victim.filter = new FragmentFilter();
// victim.filter.antiAliasing = 2;
addChild(victim);

var quad:Quad = new Quad(200, 200, 0xff0000);
addChild(quad);

var mask:Quad = new Quad(200, 200, 0xff00);
mask.rotation = Math.PI / 4;
quad.mask = mask;

Run the code as it is: two objects will show up. Then activate anti-aliasing on the filter and run it again: now the second object will disappear. It's visible only during the very first frame, then it's gone.

Starling's filters draw into a texture, passing antiAliasing property is passed to 'context.setRenderToTexture. Otherwise, the antiAliasing` setting does not change any of Starling's logic. Thus, I would conclude that Stage3D has a problem when stencil masks are used while drawing anti-aliased to a texture.

BTW: on iOS, the behavior is different: the code from above displays correctly, but if you animate victim, the output becomes strange. To reproduce, append this code:

Starling.juggler.tween(victim, 10, { rotation: Math.PI });

Known Workarounds

Starling developers can use the filter.resolution property instead of filter.antiAliasing. That makes Starling simply render the filter to a bigger texture, scaling it down manually.