Provide access to camera frame in "immersive-ar" mode

[antlai]

Since chrome implemented the new "immersive-ar" mode, and removed the "legacy-inline-ar" one, I can't figure out how to access the video frame. Before I was using readPixels in the gl frame buffer, but now compositing is done in the background. Mirroring is also optional (and not implemented) on mobile devices.

I'm aware of the long term effort to provide computer vision APIs, but it would be very useful to have a standard way to snapshot a video frame with a pose annotation, e. g., an extension to
ImageCapture.grabFrame(), even if performance/accuracy are not guaranteed, or an extra permission has to be given by the user.

[klausw]

Chrome doesn't currently support raw camera image access in "immersive-ar" mode. This is on the list of intended future features - https://github.com/immersive-web/administrivia/blob/master/modules.md mentions "camera alignment" as one of the proposed AR module items.

My understanding is that the future goal is to provide an opt-in feature (requiring separate user consent) that lets the application get camera images along with corresponding pose data, but the specifics are still in early discussion stages. In the meantime, it seemed preferable to use the more privacy-preserving separate compositing mode to avoid applications depending on nonstandard image access methods such as readPixels that weren't really intended to work in this way.

For reference, here's the computer vision work you mentioned, but that's going in a slightly different direction:

https://github.com/immersive-web/computer-vision
https://github.com/immersive-web/computer-vision/blob/master/video-effects.md

[asajeffrey]

cc @avadacatavra for privacy implications of access to raw camera data.

[klausw]

As a separate issue, see-through AR headsets typically use cameras in different ways than smartphone AR, for example the drawable FoV and the camera FoV are generally not the same, and there may be other restrictions such as only providing grayscale images. A future API should be able to take such properties into account to avoid platform-specific assumptions.

[antlai]

@klausw Thanks, great info!

it seemed preferable to use the more privacy-preserving separate compositing mode to avoid >applications depending on nonstandard image access methods such as readPixels that weren't really >intended to work in this way.

The basic requirement is "take a pic of your friend with the pink dragon", any model viewer will need something like that. A bit more ambitious goal, take a few pics annotated with pose so I can do, e.g., a 3D collage with them. A synchronous call that I can use in the requestAnimationFrame() callback could do both...

[klausw]

By "separate compositing mode" I just meant the current Chrome implementation of immersive-ar mode, where the drawn content gets composited with the camera image as a separate step.

"take a pic of your friend with the pink dragon" is an interesting use case. The user wants the picture, but there's no need for the JavaScript app or originating web site to have access to the camera pixels to do so. Currently, the user can save a picture of the composited AR scene by taking a screenshot using a non-browser platform method, i.e. power button long-press on Android. It sounds useful to offer "save a picture locally" as a WebXR feature separate from providing annotated camera frames due to its quite different privacy implications. If the user wants to share the resulting picture online, I think it would be cleaner to do this via standard picture upload as a separate step instead of giving the site direct access.

I think a 3D collage based on poses isn't currently possible using existing WebXR features, but that would be an example of a use case that would be enabled by the future extensions being discussed.

[ddorwin]

immersive-web/proposals#36 is related to the use case mentioned. There might also be some approaches for the this particular type of use case related to Web Share.

[antlai]

@klausw You are right, it makes sense to bypass JavaScript for privacy reasons in that use case. A "write frame to local file" followed by Web Share will do it, and it would be even easier if the local filename is visible to JavaScript and menus can be prepopulated.

@ddorwin Thanks, shall we close this as a duplicate? It may be worth to re-open the original proposal because this topic will keep on coming back...

[cwilso]

Closed as dupe of immersive-web/proposals#36 .