AR Image Marker Implementation Open Discussion ft. GSoC'23

[ranaaditya]

Introduction

Hi everyone,

This is an open discussion for implementation of AR Image Markers integration in processing-android as part of GSoC'23

Lets start with Augmented Images - Augmented Images in ARCore let you build AR apps that can respond to 2D images, such as posters or product packaging, in the user’s environment. You provide a set of reference images, and ARCore tracking tells you where those images are physically located in an AR session, once they are detected in the camera view.

Basically, using augmented images, you can turn a simple 2D image into an augmented image which can be recognized by your app and be then used to place a 3D model above it.

There are other specific criteria for augmented images, you can get those in the below referrence links.

Three ways to include AR Image Markers in an android application:

• Pure Static way - Add all the models and images before hand and not allow the user to add any more images/models.
  In this way we cannot add more than 1000 images dataset (thats cap from ARCore side not from our). With this small dataset for max 1000 images we cannot satisfy everyone's needs with the AR image marker usage. This was useful if we had limited set of user (like certain type of artists) but fortunately we don't have finite users. So according to me this will not be any useful for us.

• Static + dynamic way - Add all the models before hand but allow the users ability to add more images to identify the existing models.
  This Way would be useful if we know that usecases are limited to certain areas but we cannot define the usecases of this feature so adding some static image DB will not only confine the database but also usecases.

• Pure Dynamic way - Create empty database and let the user add models and images to their liking.
  This will be most suitable implementation for us as we will not be confined to some database but we can generalize the use of this feature. With this implemetation we can give full freedom to the artist to test their creativity and will be more useful for everyone.

References

• Gaurav's ideation about the ARImage Markers
• How Markerless ARImage marker works (static Image DB)
• Detailed explanation about ARImage Marker in android
• arcoreimg tool for generating image set from an image

What are your thoughts on this @codeanticode ?

GSoC Contributor: Gaurav Puniya

Thanks,
Aditya Rana

[SableRaf]

Thanks Aditya for the detailed proposal. I appreciate the thorough overview and possible implementation methods that you've presented.

Just to ensure we're all on the same page, let's define the scope of our discussion. We're primarily focusing on Augmented Reality applications that leverage the camera on an Android mobile device. The goal is to overlay 2D or 3D content over a specific image in the real world, as seen in augmented picture books such as the one pictured in the GIF below. Is my understanding correct?

Image: Adrien Mondot and Claire Bardainne

To clarify, are you considering the addition of a built-in library of 3D models to Processing for Android? From what I understand, based on Gaurav's Medium article, the main concern seems to be around how to handle the image marker database given ARCore's limitation of 1000 images per database. To confirm, is this limitation specific to each application?

I believe that Processing creators will want to load their own 3D models (or build geometry at runtime in OpenGL), and use their own image markers. This means that Processing's AR support should focus on facilitating the process of adding user's custom models and images, rather than providing a built-in database of models or images. Therefore the "Pure Dynamic way" seems to best align with the open-ended nature of Processing. We can definitely include a few smaller copyright-free 3D models in the examples for the library, but our main focus should be providing the tools necessary for users to incorporate their own work.

Let me know if I'm correctly understanding the scope of the discussion or if there's anything else that needs further clarification.

Best,
Raphaël

PS: While the idea of loading models from an online database, like Firebase, is intriguing, I think it falls a bit outside of the project's scope. That is, unless ARCore has some specific quirks that require a distinct method for loading files from a remote server.

[ranaaditya]

Hi @SableRaf , albosutely correct !

Your understanding is absolutely correct, I wanted to add other aspects also and the future optimization of this feature but wasn't sure to add it here now, so didn't add.

Yes, Image dataset is limited to 1000 Image dataset per application, its restriction from ARCore SDK itself I believe.

For our own original and copyrighted dataset I agree completely with this but this will be the next step after succesfully adding the Image Marker feature into processing-android and also out of scope of the GSoC'23 project.

Future changes we can do (not in this GSoC'23 project scope):

• add a full fledge librray of our oen original image marker dataset along with the models
• host it on fast CDN (firebase is free but too slow !)
• Add a image detection in the AR camera view of our project and with this we can detect the type of object in the environment
• From our dataset fetch only the required image datasets

This way artists need not to add their dataset into the PDE but this will be much tiring job to have every kind of dataset, still this is not in the scope of the current GSoC Project. Will add one more discussion in detail about it. I have plans for this as well !

Thanks,
Aditya Rana

[p4puniya]

I would like to add something here for future prospects.

What I learned after going through some AR apps was that they don't load all the images/3D-models from online database, instead they try to filter the models that might be needed by the User and store them locally.

For apps that used a fixed kind of 3D models( like interior design apps) they usually store the 3D models in local storage while only getting the material (learn more about materials here) from cloud to make the app faster.Thus making it a seamless experience for it's users.

Though, this method is more inclined towards the use of markerless AR, it can be used when adding the Cloud database functionality to Processing-AR as well.

[SableRaf]

I noticed the inclusion of the article "The Ultimate Guide to Markerless Augmented Reality" in your references. Since this piece is about markerless AR, it seems to fall outside the scope of our current discussion on image markers. Could you clarify this for me? Was this link intended?

However, this made me curious about other aspects of the ARCore API. Are you also considering incorporating features like plane detection, depth-based occlusion or face augmentation in this project?

[ranaaditya]

Hi @SableRaf , the referernce for the markerless AR was for the reading purpose only to understand why some need it but as i already clarify this cannot fullfill our usecase.

Just for reading purpose to understand how it works and why it will not be useful for us.

Thanks !

[SableRaf]

Hi @ranaaditya and @p4puniya,

I appreciate your ideas for the AR project, but I worry that a focus on the remote hosting of assets adds complexity that could make things more challenging than necessary for our user base.

In my opinion, the main focus should be on simplifying the process of loading user's custom models and image markers from the sketch folder (locally). The 1000 image-cap should be more than enough for most use cases.

I believe that an online database of image markers is superfluous and problematic. Artists will likely prefer to use their unique images, and having to download markers or textures could pose problems for those with limited bandwidth or in offline situations. Additionally, an online database would require consistent hosting and maintenance, which presents yet another operational concern.

I'm interested in hearing what Andres (@codeanticode) Colubri thinks about this.

Best,
Raphaël

[ranaaditya]

Yes @SableRaf I agree with you, that's why I didn't discuss the online db. Its not our main goal right now, our main goal is to have AR Image Markers in the processing-android and each user can have their own db which they need to add into the app as per their use.

I am also eager to hear from Andres, what he thinks about it !

Thanks,
Aditya Rana

[codeanticode]

Hello everyone, sorry for the delay in responding to this thread!

Very interesting points and references by @ranaaditya and @p4puniya, thank you guys.

After reading the Google's developer documentation on augmented images, and considering the discussion so far, my own opinion would be to simply use the local database functionality in ARCore, following the purely dynamic approach, where users would create the image database on the fly from the sketch by adding PImage objects to it.

Also, I think we should minimize the need of any new API, if my understanding of ARCore's AugmentedImage class is correct, we would be able to integrate image trackables into the current infrastructure of the AR library by adding a new trackable type alongside wall and floor, i.e.: image, so we can reuse the existing trackable functionality, so a sketch using image markers could look like this:

import processing.ar.*;

ARTracker tracker;

void setup() {
  fullScreen(AR);
  tracker = new ARTracker(this);
  PImage img = loadImage("example.png");
  tracker.addImage(img);
  tracker.start();
}

void draw() {
  for (int i = 0; i < tracker.count(); i++) {
    ARTrackable t = tracker.get(i);
    println("Trackable", i, t, t.isImage());
  }  
}

A cloud database would be cool, but as @SableRaf pointed out, this would introduce significant complications and extra maintenance. And advanced users should still be a able to implement their own remote marker databases using the Android API directly.

Hope that my comments are useful :-)

[ranaaditya]

Hi @codeanticode , yes we are also thinking about implementing similarly.

Thanks for the code snippet.

Thanks,
Aditya

[p4puniya]

I was thinking how to Add the Images in the AugmentedImageDatabase for ARMarkers, and there can be 2 ways:

1. Adding them one by one and writing boiler plate code for each image in PDE to add them to the AugmentedImageDatabase.
2. Making a database file using arcoreimg tool and adding that to the augmented image database.
   I personally prefer the second method, as it will be easier to index all the images. The only short coming being that the user needs to go through a few extra steps to make the database file. So, should I use the first method or the second to implement this?
   P.s. If we use the second method, once we are done with our project, we can make some tutorials for the user to make the image database using the CLI tool and add it to the project.

[codeanticode]

@p4puniya Thanks for pointing out these two options. Personally, I'd go for the first method since I think that no much boilerplate code would be needed in the sketch itself, other than loading the image into the tracker so it gets dynamically added to the internal database.

All the user would need to do in their sketch is:

tracker = new ARTracker(this);
tracker.addImage(img);

which should be handled inside the ARTracker class along the following lines, if my understanding of augmented images API in ARCore is correct:

public class ARTracker {
    ...
   AugmentedImageDatabase imgDB;
    
  public ARTracker(PApplet parent) {
    ...
    ARGraphics  arg = (ARGraphics)parent.g;
    imgDB = new AugmentedImageDatabase(arg.surfar.session);  // Might need to make session public or add a getSession() method.
  }

  void addImage(PImage img) {
    Bitmap bitmap = (Bitmap)img.getNative();
   int imgIndex = imgDB.addImage("name"); // User-provided or auto-generated name here, physical size argument seems optional

    // Re-set the session config with the updated image database
    Config config = new Config(session);
    config.setAugmentedImageDatabase(imgDB);
    session.configure(config);
  }
 ...
}

A new trackable type in ARGraphics for images should be implemented:

public class ARGraphics extends PGraphics3D {
  ...
  protected ArrayList<Plane> trackObjects = new ArrayList<Trackable>(); // replace trackPlanes with this

  protected HashMap<Trackable, Integer> trackIds = new HashMap<Trackable, Integer>(); // Use Trackable as the the key's type to it hand hold both Plane and AugmentedImage objects
  trackImages
  ...
  protected void updateTrackables() {
    Collection<Plane> planes = surfar.frame.getUpdatedTrackables(Plane.class);
     ...
    Collection<AugmentedImage> images = surfar.frame.getUpdatedTrackables(AugmentedImage.class);
    for (AugmentedImage image: images) {
      trackObjects.add(image);
    }
  }
  ...
  public int trackableId(int i) {
    return trackIds.get(trackObjects.get(i));
  }
  ...
}

There is some more logic that would be needed in ARGraphics and ARTrackable to support both planes and images seamlessly, in my opinion, the user should always get a ARTrackable object in their sketch and use it accordingly to its type, so a new method ARTrackable.isImage() could be added for this purpose.

Of course, all of this is just a suggestion, following the existing architecture of the AR library :-)

[p4puniya]

Okay, Please correct me if I am wrong. What I understood is that instead of creating a new java class file (AugmentedImageDatabase.java) you are suggesting to use the ARTracker.java and add new function to Load the dataset containing the reference images for Augmented Images. That's a brilliant idea. Also, for adding images to the database, I'm happy to go with your suggestion. Will try to implement this and share the code for code review soon.

[p4puniya]

@codeanticode Do you want me to replace this: protected ArrayList<Plane> trackPlanes = new ArrayList<Plane>();
with : protected ArrayList<Plane> trackObjects = new ArrayList<Trackable>();
If yes, then please note that Trackable cannot be converted to Plane. Plane just inherits a few methods from Trackable.

[p4puniya]

Also, for int imgIndex = imgDB.addImage("name"); I'm taking a string input from the user and using that for the addImage function, like this:

void addImage(String name, PImage img) {
    Bitmap bitmap = (Bitmap)img.getNative();
    int imgIndex = imgDB.addImage(name,bitmap); 
    Config config = new Config(session);
    config.setAugmentedImageDatabase(imgDB);
    session.configure(config);
  }

[codeanticode]

Yes, I think having a name argument in the addImage() method is a good idea. I don't know if those names are use for anything actually by ARCore...

About your observation regarding replacing Plane by Trackable as the key in the map tracked objects, I think it should be fine, since you can cast the Trackable object to the appropriate subclass when getting the item from the list, something like:

Trackable obj = trackObjets.get(idx);
if (obj instanceof Plane) {
  Plane plane = (Plane)obj;
  ...
}

[SableRaf]

Hey @p4puniya

Just caught up with your discussion with Andres, really interesting stuff! It's clear you're taking the user experience to heart when thinking about AR markers, which is just great.

Your conversation reminded me of a key aspect of Processing's design philosophy. The main idea is to make it easy to do simple things, but to also allow a more experienced programmer to do complicated things within the same language.

Sometimes, what seems like a great solution from an engineering perspective (think using a CLI tool, or loading assets from a CDN) can actually be daunting for a beginner. This is a tricky balance to achieve and there's no one-size-fits-all solution to make your designs beginner-friendly. However, keeping this goal in mind can help guide you in future design decisions for the project.

For further insights into what makes Processing unique, I recommend reading A Modern Prometheus by Ben Fry and Casey Reas (the creators of Processing).

[p4puniya]

Hi @codeanticode, I'm making the ARtrackables.isImage() method. I just wanted to discuss what would be it's exact use. I read that it will work to use both Image and planes. So should I make it as a boolean method which will return false in case we have a plane and true for Images?
I saw that in ARGraphics, there are specific IDs for each plane:

  static protected final int PLANE_FLOOR   = 0;
  static protected final int PLANE_CEILING = 1;
  static protected final int PLANE_WALL    = 2;
  static protected final int POINT         = 3;

And then these IDs are checked by specific functions in ARTrackable like this:

  public boolean isCeilingPlane() {
    int idx = g.trackableIndex(id);
    return g.trackableType(idx) == ARGraphics.PLANE_CEILING;
  }

So, following the above example, I can add a new Index for images in ARGraphics:
static protected final int IMAGE = 4;
And make isImage() method in ARTrackable() like:

public boolean isImage(){
    int idx = g.trackableIndex(id);
    return g.trackableType(idx)== ARGraphics.IMAGE;
  }

Am I correct? Also, are there any other changes that I need to make for tracking the Image?

[codeanticode]

Yes your code is correct? The idea of these methods is to allow users to check if the trackable object is of a specific type, so they can use or ignore it depending on the task.

[p4puniya]

Hi @codeanticode,
I've been facing an issue when running the addImage functionality:

System.err: An error occurred in ARCORE: Attempt to read from field 'long com.google.ar.core.Session.nativeWrapperHandle' 
on a null object reference in method 'void com.google.ar.core.Config.<init>(com.google.ar.core.Session)'

This is my current code:

public void addImage(String name, PImage img) {
    Bitmap bitmap = (Bitmap)img.getNative();
    int imgIndex = imgDB.addImage(name,bitmap); 
    Config config= new Config(session);
    
    config.setAugmentedImageDatabase(imgDB);
    session.configure(config);
  }

I tried the following to resolve this but it doesn't seem to work:

1. Using the default Session & Context of ARCore and initialising it to the config as done in the default AR Core apps. (error not resolved)
2. Initialising session using session= new Session(session) before passing it to the context. (I tried passing arg.surfar.session as well but still the same error.)
3. I saw that in arg.surfar.session was passed as a parameter instead of session when initialising the imgDB in ARTracker.
   Surfar is a ARSurface object (defined in ARGraphics) and we are passing it's session when defining imgDB. So I tried to do the same. Using arg.surfar.session , but still no luck.

I'm out of ideas here and can't seem to find a solution for this. Please help me resolve this issue.

[codeanticode]

Have you tried adding an image to an augmented image DB using an AR sample project from Google to see if you get the same or similar error?

[p4puniya]

Yes, The sample app instructs us to add a sample image to the ImageDB and I was able to replicate it without any errors.
Info on Codelabs:

Sample Replicated:

How the app added the image to the DB:

1. Load the image to the Bitmap:

private Bitmap loadAugmentedImageBitmap() {
    try (InputStream is = getAssets().open("default.jpg")) {
      return BitmapFactory.decodeStream(is);
    } catch (IOException e) {
      Log.e(TAG, "IO exception loading augmented image bitmap.", e);
    }
    return null;
  }

2. Setup the database and add the image to it.

private boolean setupAugmentedImageDatabase(Config config) {
    AugmentedImageDatabase augmentedImageDatabase;

    // There are two ways to configure an AugmentedImageDatabase:
    // 1. Add Bitmap to DB directly
    // 2. Load a pre-built AugmentedImageDatabase
    // Option 2) has
    // * shorter setup time
    // * doesn't require images to be packaged in apk.
    if (useSingleImage) {
      Bitmap augmentedImageBitmap = loadAugmentedImageBitmap();
      if (augmentedImageBitmap == null) {
        return false;
      }

      augmentedImageDatabase = new AugmentedImageDatabase(session);
      augmentedImageDatabase.addImage("image_name", augmentedImageBitmap);
      // If the physical size of the image is known, you can instead use:
      //     augmentedImageDatabase.addImage("image_name", augmentedImageBitmap, widthInMeters);
      // This will improve the initial detection speed. ARCore will still actively estimate the
      // physical size of the image as it is viewed from multiple viewpoints.
    } else {
      // This is an alternative way to initialize an AugmentedImageDatabase instance,
      // load a pre-existing augmented image database.
      try (InputStream is = getAssets().open("sample_database.imgdb")) {
        augmentedImageDatabase = AugmentedImageDatabase.deserialize(session, is);
      } catch (IOException e) {
        Log.e(TAG, "IO exception loading augmented image database.", e);
        return false;
      }
    }

    config.setAugmentedImageDatabase(augmentedImageDatabase);
    return true;
  }

I just provided the code from the code labs, maybe this could help.

[codeanticode]

Ok thank you, I was busy with other things but will look into it now.

[codeanticode]

I reviewed Gaurav's PR and made a number of changes to the augmented images work alongside the existing plane trackables. Everything is merged into main now. I did not tests extensively, but seems to work well, the photos below are from the Images example I added to the AR library:

@p4puniya @ranaaditya can you guys test some more to make sure all other functions in the AR library work as expected? I refactored the code a bit and I think everything should be the same but you never know :-)

[ranaaditya]

Sure @codeanticode , will test it on real devices along with other expected functionality as well.

Thanks for the help 🙂

[codeanticode]

Closing the discussion on this topic since all latest changes are merged into the main branch :-)