pgl.readPixels depth is always returning 0 #2771

Closed
nakednous opened this Issue Aug 10, 2014 · 6 comments

Comments

Projects
None yet
4 participants
@nakednous

pgl.readPixels is always returning 0. However, it should be a value ranging in [0..1](near and far plane respectively) depending on the object's depth under pixel (when using the GL_DEPTH_COMPONENT flag as mentioned here). Below you find a sketch which requires proscene2 reproducing the problem. The Scene.pointUnderPixel is overriden for debugging purposes (the function is the same but it just prints the returned depth value). Please run it and click over any box to see it.

import java.nio.FloatBuffer;

import remixlab.dandelion.core.*;
import remixlab.dandelion.core.Constants.*;
import remixlab.dandelion.geom.*;
import remixlab.proscene.*;

MyScene scene;
Box [] boxes;

public void setup() {
  size(640, 360, P3D);
  scene = new MyScene(this);
  scene.setRadius(150);
  scene.showAll();  
  boxes = new Box[50];
  for (int i = 0; i < boxes.length; i++)
    boxes[i] = new Box(scene);

  scene.setKeyboardShortcut('z', Scene.KeyboardAction.RESET_ANCHOR);
  scene.setMouseClickBinding(Target.EYE, LEFT, 1, ClickAction.ZOOM_ON_PIXEL);
  scene.setMouseClickBinding(Target.EYE, RIGHT, 1, ClickAction.SHOW_ALL);
  scene.setMouseClickBinding(Target.EYE, Event.CTRL, RIGHT, 1, ClickAction.ANCHOR_FROM_PIXEL);
}

public void draw() {  
  background(0);
  for (int i = 0; i < boxes.length; i++)      
    boxes[i].draw();
}

class MyScene extends Scene {
  public MyScene(PApplet p) {
    super(p);
  }

  @Override
  protected Vec pointUnderPixel(Point pixel) {
    PGraphicsOpenGL pggl;
    if (pg() instanceof PGraphicsOpenGL)
      pggl = (PGraphicsOpenGL) pg();
    else
      throw new RuntimeException("pg() is not instance of PGraphicsOpenGL");
    float[] depth = new float[1];
    PGL pgl = pggl.beginPGL();
    pgl.readPixels(pixel.x(), (camera().screenHeight() - pixel.y()), 1, 1, PGL.DEPTH_COMPONENT, PGL.FLOAT, FloatBuffer.wrap(depth));
    pggl.endPGL();
    Vec point = new Vec(pixel.x(), pixel.y(), depth[0]);
    //debug
    println("depth[0] seems to be always 0 (no matter if an object is picked): " + depth[0]);
    point = unprojectedCoordinatesOf(point);
    return (depth[0] < 1.0f) ? point : null;
  }
}

public class Box {
  Scene scene;
  public InteractiveFrame iFrame;
  float w, h, d;
  int c;

  public Box(Scene scn, InteractiveFrame iF) {
    scene = scn;
    iFrame = iF;
    iFrame.setGrabsInputThreshold(25);
    setSize();
    setColor();
  }

  public Box(Scene scn) {
    scene = scn;
    iFrame = new InteractiveFrame(scn);
    setSize();
    setColor();    
    setPosition();
  }

  public void draw() {
    draw(false);
  }

  public void draw(boolean drawAxes) {
    pushMatrix();
    iFrame.applyWorldTransformation();

    if (drawAxes)
      scene.drawAxes(max(w, h, d)*1.3f);
    noStroke();
    if (scene.grabsAnyAgentInput(iFrame))
      fill(255, 0, 0);
    else
      fill(getColor());
    //Draw a box    
    box(w, h, d);

    popMatrix();
  }

  public void setSize() {
    w = random(10, 40);
    h = random(10, 40);
    d = random(10, 40);
  }

  public void setSize(float myW, float myH, float myD) {
    w=myW; 
    h=myH; 
    d=myD;
  }  

  public int getColor() {
    return c;
  }

  public void setColor() {
    c = color(random(0, 255), random(0, 255), random(0, 255));
  }

  public void setColor(int myC) {
    c = myC;
  }

  public Vec getPosition() {
    return iFrame.position();
  }  

  public void setPosition() {
    float low = -100;
    float high = 100;
    iFrame.setPosition(new Vec(random(low, high), random(low, high), random(low, high)));
  }

  public void setPosition(Vec pos) {
    iFrame.setPosition(pos);
  }

  public Quat getOrientation() {
    return (Quat)iFrame.orientation();
  }

  public void setOrientation(Vec v) {
    Vec to = Vec.subtract(v, iFrame.position()); 
    iFrame.setOrientation(new Quat(new Vec(0, 1, 0), to));
  }
}

@codeanticode codeanticode self-assigned this Aug 11, 2014

@codeanticode codeanticode added the opengl label Aug 11, 2014

@nakednous

This comment has been minimized.

Show comment
Hide comment
@nakednous

nakednous Sep 12, 2014

Here you can find a simpler sketch reproducing the issue without the proscene dependency:

import java.nio.FloatBuffer;

void setup() {
  size(100, 100, P3D);
}

void draw() {
  translate(58, 48, 0);
  rotateY(0.5);
  box(40);
}

PVector pointUnderPixel(int x, int y) {
  PGraphicsOpenGL pggl = (PGraphicsOpenGL)g;
  float[] depth = new float[1];
  PGL pgl = pggl.beginPGL();
  pgl.readPixels(x, height - y, 1, 1, PGL.DEPTH_COMPONENT, PGL.FLOAT, FloatBuffer.wrap(depth));
  pggl.endPGL();
  return new PVector(x, y, depth[0]);
}

void mousePressed() {
  print(pointUnderPixel(mouseX, mouseY));
  println(" z should give the depth value under the pixel, but it's always 0 no matter where mouse has being pressed");
}

Here you can find a simpler sketch reproducing the issue without the proscene dependency:

import java.nio.FloatBuffer;

void setup() {
  size(100, 100, P3D);
}

void draw() {
  translate(58, 48, 0);
  rotateY(0.5);
  box(40);
}

PVector pointUnderPixel(int x, int y) {
  PGraphicsOpenGL pggl = (PGraphicsOpenGL)g;
  float[] depth = new float[1];
  PGL pgl = pggl.beginPGL();
  pgl.readPixels(x, height - y, 1, 1, PGL.DEPTH_COMPONENT, PGL.FLOAT, FloatBuffer.wrap(depth));
  pggl.endPGL();
  return new PVector(x, y, depth[0]);
}

void mousePressed() {
  print(pointUnderPixel(mouseX, mouseY));
  println(" z should give the depth value under the pixel, but it's always 0 no matter where mouse has being pressed");
}
@JakubValtar

This comment has been minimized.

Show comment
Hide comment
@JakubValtar

JakubValtar Sep 13, 2014

Contributor

Right now, Processing does not support depth and stencil reads on multisampled frame buffers. To make it work, you have to disable multisampling with noSmooth().

Implementing this is trivial, but I guess the memory and performance overhead would be considerable, @codeanticode?

Contributor

JakubValtar commented Sep 13, 2014

Right now, Processing does not support depth and stencil reads on multisampled frame buffers. To make it work, you have to disable multisampling with noSmooth().

Implementing this is trivial, but I guess the memory and performance overhead would be considerable, @codeanticode?

@benfry

This comment has been minimized.

Show comment
Hide comment
@benfry

benfry Sep 13, 2014

Member

It's not something we'd want to enable by default (due to the overhead, and otherwise we'd have API for it), but is there a way we can enable a workaround for advanced users? Or, is there anything preventing advanced users from implementing this?

This isn't technically a "bug" per se since we don't claim to support anything inside being/endPGL().

Member

benfry commented Sep 13, 2014

It's not something we'd want to enable by default (due to the overhead, and otherwise we'd have API for it), but is there a way we can enable a workaround for advanced users? Or, is there anything preventing advanced users from implementing this?

This isn't technically a "bug" per se since we don't claim to support anything inside being/endPGL().

@JakubValtar

This comment has been minimized.

Show comment
Hide comment
@JakubValtar

JakubValtar Sep 13, 2014

Contributor

We can easily make a hint which would set a flag and trigger reinitialization of frame buffer the same way smooth() does. I have time to do this today.

Contributor

JakubValtar commented Sep 13, 2014

We can easily make a hint which would set a flag and trigger reinitialization of frame buffer the same way smooth() does. I have time to do this today.

JakubValtar added a commit to JakubValtar/processing that referenced this issue Sep 16, 2014

New hint for depth reading (see #2771)
Use hint(ENABLE_DEPTH_READING) to enable depth reading while using
multisampling.

JakubValtar added a commit to JakubValtar/processing that referenced this issue Sep 16, 2014

New hint for depth reading (see #2771)
Use hint(ENABLE_DEPTH_READING) to enable depth reading while using
multisampling.

JakubValtar added a commit to JakubValtar/processing that referenced this issue Sep 16, 2014

New hint for depth reading (see #2771)
Use hint(ENABLE_DEPTH_READING) to enable depth reading while using
multisampling.
@JakubValtar

This comment has been minimized.

Show comment
Hide comment
@JakubValtar

JakubValtar Sep 16, 2014

Contributor

@benfry Please check my branch for suggested changes: 77bf726
Not sure how @codeanticode likes this, it adds a lot of code into PGL. However, it would be nice to extend it later so it can provide depth texture. Personally, I would use depth texture a lot in depth of field and other fancy shaders.

Example code:

void setup() {
  size(100, 100, P3D);
  hint(ENABLE_DEPTH_READING); // enables depth reading for underlying PGraphicsOpenGL
}

void draw() {
  translate(58, 48, 0);
  rotateY(0.5);
  box(40);
}

PVector pointUnderPixel(int x, int y) {
  PGL pgl = beginPGL();
  float z  = pgl.getDepthValue(x, y);
  endPGL();
  return new PVector(x, y, z);
}

/* // this also works
PVector pointUnderPixel(int x, int y) {
  PGraphicsOpenGL pggl = (PGraphicsOpenGL)g;
  float[] depth = new float[1];
  PGL pgl = pggl.beginPGL();
  pgl.readPixels(x, height - y, 1, 1, PGL.DEPTH_COMPONENT, PGL.FLOAT, FloatBuffer.wrap(depth));
  pggl.endPGL();
  return new PVector(x, y, depth[0]);
}
*/

void mousePressed() {
  print(pointUnderPixel(mouseX, mouseY));
}

Edit: please ignore the three references above, I had some mess up here...

Contributor

JakubValtar commented Sep 16, 2014

@benfry Please check my branch for suggested changes: 77bf726
Not sure how @codeanticode likes this, it adds a lot of code into PGL. However, it would be nice to extend it later so it can provide depth texture. Personally, I would use depth texture a lot in depth of field and other fancy shaders.

Example code:

void setup() {
  size(100, 100, P3D);
  hint(ENABLE_DEPTH_READING); // enables depth reading for underlying PGraphicsOpenGL
}

void draw() {
  translate(58, 48, 0);
  rotateY(0.5);
  box(40);
}

PVector pointUnderPixel(int x, int y) {
  PGL pgl = beginPGL();
  float z  = pgl.getDepthValue(x, y);
  endPGL();
  return new PVector(x, y, z);
}

/* // this also works
PVector pointUnderPixel(int x, int y) {
  PGraphicsOpenGL pggl = (PGraphicsOpenGL)g;
  float[] depth = new float[1];
  PGL pgl = pggl.beginPGL();
  pgl.readPixels(x, height - y, 1, 1, PGL.DEPTH_COMPONENT, PGL.FLOAT, FloatBuffer.wrap(depth));
  pggl.endPGL();
  return new PVector(x, y, depth[0]);
}
*/

void mousePressed() {
  print(pointUnderPixel(mouseX, mouseY));
}

Edit: please ignore the three references above, I had some mess up here...

@codeanticode

This comment has been minimized.

Show comment
Hide comment
@codeanticode

codeanticode Aug 4, 2015

Member

Fixed with #3527

Member

codeanticode commented Aug 4, 2015

Fixed with #3527

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment