Stencil tests are automatically disabled every frame inside draw loops

[inaridarkfox4231]

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

p5.js version

1.8.0

Web browser and version

Chrome

Operating system

Windows

Steps to reproduce this

Steps:

1. Enable the stencil test in setup().
2. I don't enable the stencil test, inside draw loop.
3. When drawing with a stencil inside a drawloop, the stencil is not taken into account.

Snippet:

function setup() {
  createCanvas(400, 400, WEBGL);
  noStroke();

  const gl = this._renderer.GL;

  // enable
  gl.enable(gl.STENCIL_TEST);
}

function draw(){
  background(0);

  const gl = this._renderer.GL;
  //gl.enable(gl.STENCIL_TEST);

  // initialize 0
  gl.clear(gl.STENCIL_BUFFER_BIT);

  // set 1
  gl.stencilFunc(gl.ALWAYS, 1, ~0);
  gl.stencilOp(gl.KEEP, gl.REPLACE, gl.REPLACE);

  fill(255, 0, 0);
  plane(200, 400);

  // if stencil value is 1, draw plane
  gl.stencilFunc(gl.EQUAL, 1, ~0);
  gl.stencilOp(gl.KEEP, gl.KEEP, gl.KEEP);

  fill(0, 0, 255);
  plane(400, 200);

  noLoop();
}

version 1.7.0

version 1.8.0

Of course, if you enable the stencil test inside the draw loop, you will get the desired drawing result.

However, some users may not like it when the stencil test is disabled regardless of their intentions.

[inaridarkfox4231]

DEMO page (p5.Editor): DEMO

Execution point of disable: https://github.com/processing/p5.js/blob/main/src/webgl/p5.RendererGL.js#L930

The commit in which this specification was introduced: Implement clip() to shapes #6306

[ImRAJAS-SAMSE]

Hi @inaridarkfox4231,

I’ve been looking into this issue and traced it back to PR #6306, which introduced clip(). The problem seems to be that the stencil test (gl.STENCIL_TEST) is automatically disabled each frame inside p5.RendererGL.js

Root Cause

• The clip() implementation uses the stencil buffer and disables it at the start of each frame.
• This unintentionally overrides the user’s manual enabling of gl.STENCIL_TEST in setup(), even when clip() is not used.
• The expected behavior is that if a user enables gl.STENCIL_TEST in setup(), it should remain enabled unless explicitly disabled.

Proposed Fix
The issue occurs because gl.STENCIL_TEST is automatically disabled at the start of each frame, unintentionally overriding the user’s manual enabling of the stencil test. This behavior was introduced in the clip() implementation.

To fix this, I propose modifying p5.RendererGL.js to preserve the stencil test state across frames. Instead of forcefully disabling it, we will check whether it was enabled before and restore its state accordingly. This will ensure that users who enable gl.STENCIL_TEST in setup() do not have it unexpectedly disabled in draw().

Let me know your thoughts!

[davepagurek]

I think that's generally the approach we want. The nuance will be in how exactly we record the previous value, because ideally we don't want to be calling gl.isEnabled() too often, as in the past these sorts of calls have been pretty slow. A workaround that I've used at work is to track when someone changes state using something like this:

this.isStencilTestOn = false;

// Record state when enabling/disabling
const prevEnable = this.drawingContext.enable;
this.drawingContext.enable = (key) => {
  if (key === this.drawingContext.STENCIL_TEST) {
    this.isStencilTestOn = true;
  }
  return prevEnable.call(this, key);
};

const prevDisable = this.drawingContext.disable;
this.drawingContext.disable = (key) => {
  if (key === this.drawingContext.STENCIL_TEST) {
    this.isStencilTestOn = false;
  }
  return prevEnable.call(this, key);
};

// Return the cached value if we try to access it via getEnabled
const prevGetEnabled = this.drawingContext.getEnabled;
this.drawingContext.getEnabled = (key) => {
  if (key === this.drawingContext.STENCIL_TEST) {
    return this.isStencilTestOn;
  }
  return prevGetEnabled.call(this, key);
};

...so that we can call getEnabled without having to actually get updated state from the GL context, since we've kept a variable in js.