If anyone is interested in taking this on, the way I've calculated screen-space positions out of WebGL coordinates in the past is by doing this:

// Assuming `this` is a `RendererGL`:
const toClipSpace = new DOMMatrixReadOnly(this.uPMatrix.mat4).multiply(
  new DOMMatrixReadOnly(this.uMVMatrix.mat4)
);
const clip = toClipSpace.transformPoint(new DOMPoint(0,0,0));
const screenX = (clip.x / clip.w + 1) / 2 * this.width;
const screenY = (clip.y / clip.w + 1) / 2 * this.height;

i would like to contribute

Thanks @Gaurav-1306! I'll assign it to you. Let me know if I can help clarify anything!

@davepagurek
in this code, I think the values of x and y are wrong, and solving that could remove the error(for textoutput()). So after the condition check that the renderer is 3d or not what should I do? should i make a new function that calculates the correct x and y position(using your suggestion for 3d and checking what is wrong with 2d) or do I have to change this._renderer.uMVMatrix.mat4 and this.drawingContext.getTransform() to find the correct value of x and y.

hey @davepagurek

//gets position of shape in the canvas
p5.prototype._getPos = function (x, y) {
  const untransformedPosition = new DOMPointReadOnly(x, y);

  if (this._renderer.isP3D) {
    const toClipSpace = new DOMMatrixReadOnly(this._renderer.uPMatrix.mat4)
      .multiply(
        new DOMMatrixReadOnly(this.uMVMatrix.mat4)
      );
    const clip = toClipSpace.transformPoint(new DOMPoint(0, 0, 0));
    const transformedX = (clip.x / clip.w + 1) / 2 * this.width;
    const transformedY = (clip.y / clip.w + 1) / 2 * this.height;

    return { x: transformedX, y: transformedY };
  } else {
    const currentTransform = this.drawingContext.getTransform();
    const { x: transformedX, y: transformedY } = untransformedPosition
      .matrixTransform(currentTransform);

    console.log('the diff');
    console.log(transformedX, transformedY);

i tried solving the problem using this, but i am unable to do it. can you help!

That looks like a good start! It looks like the function currently returns some strings for the location:

so maybe rather than returning an object with an x and y position, your code should be something like this, where both branches set transformedX and transformedY, to be used later on when returning a string:

//gets position of shape in the canvas
 p5.prototype._getPos = function (x, y) {
   const untransformedPosition = new DOMPointReadOnly(x, y);
+  let transformedX = 0;
+  let transformedY = 0;

   if (this._renderer.isP3D) {
     const toClipSpace = new DOMMatrixReadOnly(this._renderer.uPMatrix.mat4)
       .multiply(
         new DOMMatrixReadOnly(this.uMVMatrix.mat4)
       );
     const clip = toClipSpace.transformPoint(new DOMPoint(0, 0, 0));
-     const transformedX = (clip.x / clip.w + 1) / 2 * this.width;
+     transformedX = (clip.x / clip.w + 1) / 2 * this.width;
-     const transformedY = (clip.y / clip.w + 1) / 2 * this.height;
+     transformedY = (clip.y / clip.w + 1) / 2 * this.height;

-     return { x: transformedX, y: transformedY };
   } else {
     const currentTransform = this.drawingContext.getTransform();
-     const { x: transformedX, y: transformedY } = untransformedPosition
+     { x: transformedX, y: transformedY } = untransformedPosition
       .matrixTransform(currentTransform);

     console.log('the diff');
     console.log(transformedX, transformedY);

We can close this issue , textOutput and gridOutput works perfectly with the latest version of p5.js <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.10.0/p5.js"></script>

Updated example with v1.10

The label now correctly describes the square's position as the bottom right! That's awesome!

Unfortunately, the sphere is still not described, and now the label says that the square takes up 15625% of the canvas, whereas before it said 6%.

Thank you for the update @calebfoss . We can Identify 2 issues here. Unreported shapes and inaccurate percentage. Please take a look at my code . It seems seems all 3D shapes are not reported by the function.

function setup() {
  createCanvas(400, 400, WEBGL);
  textOutput(LABEL);
}

function draw() {
  background(220);
  sphere();
  circle(50, 50, 100);
  square(100, 100, 100);
}

@Forchapeatl, exactly! 3D shape descriptions were never implemented. That's what I was referring to here:

When using WEBGL mode, textOutput() and gridOutput() only produce descriptions of 2D shapes

In this accessibility features proposal, I go into more detail on the future of these functions.

I'm glad you're looking into improving the accessibility features!