Finalize AR library functionality and API

[codeanticode]

Version 4.1-beta1 incorporates the AR library developed during GSOC 2018. Even though it is functional, at this moment it only allows using a single anchor from the first available plane detected by AR Core. Some simple API should be added to allow users to select plane/anchor points, etc.

[codeanticode]

@benfry I have been implementing a tentative minimalist AR API in PApplet around ARCore, following up initial work from last year's GSOC. The ARCore API is significantly more complex than GVR, with several classes for trackable surfaces and point clouds in real space, user-defined anchors associated to the trackables, pose transformations, etc., so this was trickier that implementing the VR library.

I'm also pinging @stalgiag, who's is leading the XR library for p5js.

My goal was to find a small number of methods that would be enough to use Processing's drawing API to create AR scenes easily and without the need of any additional classes in the API. These methods should support at least two basic elements in AR: trackables and anchors. ARCore tracks planes and point clouds, but I restricted the API in AR library to support only planes, as I believe they cover most situations. Users cannot create trackables, since they are detected and handled automatically by ARCore. Anchors are essentially positions fixed in relation to an existing trackable surface. One way of thinking of anchors is as matrix transformations that the user pushes as desired. Anchors can be created and deleted by users, and creating anchors from a screen touch is important since it allows interaction with the AR scene.

So, I came up with the following methods:

• int trackableCount()
  Returns the number of trackable planes.

• int trackableId(int i)
  Returns a unique id of the i-th trackable plane (an integer greater than 0). Unique ids are needed because trackables are created and destroyed automatically by ARCore as the camera image changes, so the positional index will correspond to different planes during the runtime of the sketch.

• int trackableType(int i)
  Returns the type of the i-th trackable plane. ARCore is able to discrimiante if a plane is pointing upwards (i.e.: a floor or a tabletop surfave), pointing downwards (eg.: the ceiling) or vertical (walls). This information is important for the user to decide how to handle the geometry attached to a particular trackable.

• int trackableStatus(int i)
  Returns the status of the i-th trackable plane. The idea was to combine several situations while provide non-redundant values, right now they are: CREATED (the status in the frame when a plane has been detected), UPDATED (if the plane's has changed in any way in the last frame), TRACKING (if the plane is being tracked), PAUSED (if the tracking is paused, but could resume anytime), STOPPED (if the tracking has stopped and will never resume, in this case the plane will be removed at the end of the frame). In particular, a status of either CREATED or UPDATED implies that the plane is being tracked.

• boolean trackableSelected(int i, int mx, int my)
  Returs true if the i-th plane is being selected by a touch at screen position (mx, my)

• float[] getTrackablePolygon(int i)
  Returns the list of (x, z) coordinates that define the boundary polygon of the i-th trackable plane. These coordinates are defined in the local coordinate system attached to the plane. The y axis is always perpendicular to the trackable planes, that's the reason why the coordinates only include x and z (y being zero). With this call, a new float array is created every time.

• float[] getTrackablePolygon(int i, float[] points)
  Same as before, but it accepts an existing float array that will be used to store the coordinates. The array will be resized if is not large enough, and is also returned as the result.

• PMatrix3D getTrackableMatrix(int i)
  This returns the transformation matrix associated to the i-th trackable plane. This matrix is needed to draw the plane correctly in world coordinates. It creates a new matrix with every call.

• PMatrix3D getTrackableMatrix(int i, PMatrix3D target)
  Same as above, but allows to reuse an existing matrix object.

• int anchorCount()
  Return the number of anchors

• int anchorId(int i)
  Returns the id of the i-th anchor (an integer greater than 0).

• int anchorStatus(int i)
  Returns the status of i-th anchor: TRACKING, PAUSED, or STOPPED. Stopped anchors should be deleted by the user.

• createAnchor(int i, float x, float y, float z)
  Creates a new anchor associated to the i-th trackable plane, at position (x, y, z) in trackable coordiantes. It returns the id of the new anchor.

• int createAnchor(int mx, int my)
  Creates a new anchor associated to the trackable plane selected with touch (mx, my). It returns the id of the new anchor, and zero if the touch hits no plane.

• void deleteAnchor(int i)
  Deletes the i-th anchor.

• PMatrix3D getAnchorMatrix(int i)
  Returns the transformation matrix associated to the i-th anchor, creating a new matrix object in every call.

• PMatrix3D getAnchorMatrix(int i, PMatrix3D target)
  Same as above, but reusing the provided matrix object.

• void anchor(int i)
  Applies the transformation of i-th anchor.

This is a total of 14 functions (counting the overloaded versions as one), so not a small number, but given the complexity of the ARCore API, is not too bad. I tried to keep them as "low-level" functions that users could apply to build their own custom AR classes. For advanced users, it is also possible to access the underlying ARCore objects and use ARCore's API directly.

With these functions, one can create simple AR scenes with several objects attached to trackable planes and touch points with minimal amount of auxiliary variables:

import processing.ar.*;

float[] points;
PMatrix3D mat;
float angle;
int oldSelAnchor;
int selAnchor;

void setup() {
  fullScreen(AR);
  mat = new PMatrix3D();
}

void draw() {
  // The AR Core session, frame and camera can be accessed through Processing's surface object
  // to obtain the full information about the AR scene:
//    PSurfaceAR surface = (PSurfaceAR) getSurface();
//    surface.camera.getPose();
//    surface.frame.getLightEstimate();

  // No background call is needed, the screen is refreshed each frame with the image from the camera

  lights();   // lighting is automatically corrected to reflect light intensity in the real space

  if (mousePressed) {
    // Create new anchor at the current touch point, save old anchor id to delete
    oldSelAnchor = selAnchor;
    selAnchor = createAnchor(mouseX, mouseY);
  }

  // Draw objects attached to each anchor
  for (int i = 0; i < anchorCount(); i++) {
    int id = anchorId(i);
    if (oldSelAnchor == id) {
      deleteAnchor(i);
      continue;
    }

    int status = anchorStatus(i);
    if (status ==  PAR.PAUSED || status == PAR.STOPPED) {
      if (status == PAR.STOPPED) deleteAnchor(i);
      continue;
    }

    pushMatrix();
    anchor(i);

    if (selAnchor == id) {
      fill(255, 0, 0);
    } else {
      fill(255);
    }

    rotateY(angle);
    box(0.15);
    popMatrix();
  }

  // Draw trackable planes
  for (int i = 0; i < trackableCount(); i++) {
    int status = trackableStatus(i);
    if (status ==  PAR.PAUSED || status == PAR.STOPPED) continue;

    if (status == PAR.CREATED && trackableCount() < 10) {
      // Add new anchor associated to this trackable, 0.3 meters above it
      if (trackableType(i) == PAR.PLANE_WALL) {
        createAnchor(i, 0.3, 0, 0);
      } else {
        createAnchor(i, 0, 0.3, 0);
      }
    }

    points = getTrackablePolygon(i, points);

    getTrackableMatrix(i, mat);
    pushMatrix();
    applyMatrix(mat);
    if (mousePressed && trackableSelected(i, mouseX, mouseY)) {
      fill(255, 0, 0, 100);
    } else {
      fill(255, 100);
    }

    beginShape();
    for (int n = 0; n < points.length/2; n++) {
      float x = points[2 * n];
      float z = points[2 * n + 1];
      vertex(x, 0, z);
    }
    endShape();
    popMatrix();
  }

  angle += 0.1;    
}

Let me know if you have any comments.

[codeanticode]

An option to reduce the number of PApplet functions could be to put all the current trackables into a trackables array, similar to touches, so one could do:

  // Draw trackable planes
  for (int i = 0; i < trackables.length; i++) {
    int status = trackables[i].status;
    if (status ==  PAR.PAUSED || status == PAR.STOPPED) continue;

    if (status == PAR.CREATED && trackables.length < 10) {
      // Add new anchor associated to this trackable, 0.3 meters above it
      if (trackables[i].type == PAR.PLANE_WALL) {
        createAnchor(i, 0.3, 0, 0);
      } else {
        createAnchor(i, 0, 0.3, 0);
      }
    }
   
    pushMatrix();
    applyMatrix(trackables[i].matrix);
   
    ...

   points = trackables[i].points;
   // Or maybe put auxiliary functions, such as this one, inside the PAR class to further reduce the 
   // API in PApplet:
  // points = PAR.getPoints(trackables[i]);

  ...
  popMatrix();

Anchors are essentially a transformation matrix together with extra additional info, such its status, so I'm thinking if it would make sense to define a PAnchor class to get away with all the anchor functions I define previously, so that:

  // anchor attached to trackable trackIdx, with position (x, y, z) relative to that trackable 
  Anchor anchor = createAnchor(trackIdx, x, y, z);

  // anchor attached to trackable intersected by pointer position (mouseX, mouseY)
  Anchor anchor = createAnchor(mouseX, mouseY);
  ...
  
  println(anchor.status);
  applyMatrix(anchor.matrix);
  deleteAnchor(anchor);

Just some thoughts...

[codeanticode]

In the end, I went for an object-oriented API that's consistent with existing libraries (sound, video, etc). This eliminates the need of adding new API to PApplet, and makes handling trackables and anchors more intuitive (I hope). Commits d471ead through 5466625 implement the new AR API.

Essentially, now the AR libraries introduces three classes:

• ARTracker - At least one ARTracker object is needed in the sketch to handle all trackables (only planes at this point) detected by AR Core in the scene.

• ARTrackable - This class provides methods to query the status of a trackable surface, apply its associated pose transformation on Processing's camera, and ARTrackable objects are needed to attach anchors to.

• ARAnchor - This class represents a fixed position with respect to a trackable. The user needs to create them explicitly at any arbitrary position, or at the intersection with the touch pointer.

Also, the library supports a trackable event handling method that gets called when a new trackable gets created, to allow for specific initialization steps.

All classes are prepended by AR so users can import the AR Core classes (i.e.: Trackable and Anchor) without naming conflicts.

The following code demonstrates the entire API:

import processing.ar.*;

ARTracker tracker;
ARAnchor touchAnchor;
ArrayList<ARAnchor> trackAnchors;
float angle;

void setup() {
  fullScreen(AR);
  tracker = new ARTracker(this);
  tracker.start();
  trackAnchors = new ArrayList<ARAnchor>();
}

void draw() {
  // The AR Core session, frame and camera can be accessed through Processing's surface object
  // to obtain the full information about the AR scene:
//    ARSurface surface = (ARSurface) getSurface();
//    surface.camera.getPose();
//    surface.frame.getLightEstimate();

  lights();

  if (mousePressed) {
    // Create new anchor at the current touch point
    if (touchAnchor != null) touchAnchor.dispose();
    ARTrackable hit = tracker.get(mouseX, mouseY);
    if (hit != null) touchAnchor = new ARAnchor(hit);
    else touchAnchor = null;
  }

  // Draw objects attached to each anchor
  for (ARAnchor anchor : trackAnchors) {
    if (anchor.isTracking()) drawBox(anchor, 255, 255, 255);

    // It is very important to dispose anchors once they are no longer tracked.
    if (anchor.isStopped()) anchor.dispose();
  }
  if (touchAnchor != null) drawBox(touchAnchor, 255, 0, 0);

  // Conveniency function in the tracker object to remove disposed anchors from a list
  tracker.clearAnchors(trackAnchors);

  // Draw trackable planes
  for (int i = 0; i < tracker.count(); i++) {
    ARTrackable trackable = tracker.get(i);
    if (!trackable.isTracking()) continue;

    pushMatrix();
    trackable.transform();
    if (mousePressed && trackable.isSelected(mouseX, mouseY)) {
      fill(255, 0, 0, 100);
    } else {
      fill(255, 100);
    }
    beginShape();
    float[] points = trackable.getPolygon();
    for (int n = 0; n < points.length / 2; n++) {
      float x = points[2 * n];
      float z = points[2 * n + 1];
      vertex(x, 0, z);
    }
    endShape();
    popMatrix();
  }

  angle += 0.1;
}

void drawBox(ARAnchor anchor, int r, int g, int b) {
  anchor.attach();
  fill(r, g, b);
  rotateY(angle);
  box(0.15f);
  anchor.detach();
}

void trackableEvent(ARTrackable t) {
  if (trackAnchors.size() < 10) {
    float x0 = 0, y0 = 0;
    if (t.isWallPlane()) {
      // The new trackable is a wall, so adding the anchor 0.3 meters to its side
      x0 = 0.3f;
    } else if (t.isFloorPlane()) {
      // The new trackable is a floor plane, so adding the anchor 0.3 meters above it
      y0 = 0.3f;
    } else {
      // The new trackable is a floor plane, so adding the anchor 0.3 meters below it
      y0 = -0.3f;
    }
    trackAnchors.add(new ARAnchor(t, x0, y0, 0));
  }
}

[codeanticode]

After conversations with @stalgiag, decided to go ahead with this version of the API (the one described in the previous comment and implemented in d471ead, 95adb35, 61a9935, 2f4d446, 10b825b, c96efaa, ee72172, da9b631, 5466625) in the next version of the mode (4.1.0). It can be refined after getting feedback from users, and taking into account consistency with p5.xr once the status of AR in WebXR becomes more clear.