Support Images in NativeEngine.createImageBitmap

[ryantrem]

We need this as part of the XR image tracking work. Specifically, image tracking has a list of reference images:

https://github.com/BabylonJS/Babylon.js/blob/master/src/XR/features/WebXRImageTracking.ts#L22

These can either be strings (e.g. a url) or an ImageBitmap that has already been loaded. In the case of a string, the XR image tracking feature will use it as the source of an Image, load and decode that image, then pass the Image to Engine.createImageBitmap to create an ImageBitmap:

https://github.com/BabylonJS/Babylon.js/blob/master/src/XR/features/WebXRImageTracking.ts#L186-L197

Babylon Native already has a native polyfill for Image:

https://github.com/BabylonJS/BabylonNative/blob/master/Polyfills/Canvas/Source/Image.cpp

And this already supports loading from a string by setting the src property, and supports the onload callback, but it doesn't have the decode function today. That said, I believe the image decoding is done implicitly during load, so I think the first thing we need is:

• Add a decode function to the native Image polyfill that returns a Promise that is completed after onload is called.

Once that is done, then the XR image tracking code can pass the resulting Image into the call to createImageBitmap. Babylon Native already has a native implementation of createImageBitmap:

https://github.com/BabylonJS/BabylonNative/blob/master/Plugins/NativeEngine/Source/NativeEngine.cpp#L1511-L1542

However as is it only knows how to deal with the input arg being an ArrayBuffer. It needs to also be able to handle an Image being passed in. From a WebAPI standpoint, Image is basically an opaque object (you can't get direct access to the underlying image data buffer), which means whoever is implementing WebAPI needs to understand the internals of that opaque type. In the case of Babylon Native, it defines the Image type (through the native polyfill) and therefore understands the internal type. Given this, I think the next thing we need is:

• Have Image store the bimg::ImageContainer pointer in a property (perhaps _imageContainer or something) on the Image object, perhaps using the Napi::Pointer<bimg::ImageContainer>::Create(...) as seen elsewhere in NativeEngine.cpp.
• Have NativeEngine::CreateImageBitmap accept an Image object in addition to a raw ArrayBuffer. I think this really means just accept a generic JS object that has an _imageContainer property defined. Once CreateImageBitmap has the bimg::ImageContainer, the rest of the existing code that creates a BitmapImage from a bimg::ImageContainer should basically be the same.

This will allow NativeEngine::CreateImageBitmap accept an Image as input and return the corresponding ImageBitmap. Like Image, ImageBitmap is an opaque object from the WebAPI standpoint. I think this means that when that ImageBitmaps are passed from the JS code (as the XR session init arg, as the trackedImages), Babylon Native's NativeXR layer needs knowledge of the opaque ImageBitmap's internals. That is to say, it needs to read the data, width, and height properties off of the JS object to be passed down into the XR abstraction, and eventually passed on to ARKit/ARCore. This part is being handled though by another body of work that is more centrally about plumbing through image tracking support in that XR layer.

[CedricGuillemet]

As Image would be used by xr, canvas and engine, does it make sense to put its definition/implementation is Core Graphics?

[ryantrem]

Core Graphics feels wrong to me since we're basically talking about a common native defined JS object. I don't think Graphics even has a napi dependency. I wonder if maybe we need a new lib under Core, like Types or something like that. Even that feels a bit weird though, because conceptually it overlaps with polyfills. Is the idea that plugins should not be able to depend on polyfills maybe wrong? @bghgary thoughts?

In the short term, I still think maybe we can get away with just a non-enforced common contract (e.g. expect an _imageContainer property on the JS object or something like that, as I described above).

[bghgary]

I'm not sure. We are in the middle of adding a graphics context that has dependencies on napi, but it still might not make sense there. What do you think @syntheticmagus?

[Alex-MSFT]

createImageBitmap is now supported for internal operations via engine.createImageBitmap. There is possibly a minor follow up task if we want to support this externally.