Request support for glasses-side preview/viewfinder during DAT app-controlled capture

[ktamulonis]

Summary

We are building an iOS app with the Meta Wearables DAT SDK for hands-free item capture. We can stream camera frames to the phone and trigger photo capture from the app, but we do not see a way to access or integrate the glasses-side preview/viewfinder that exists in the native wristband capture flow.

What works today

Using the public DAT iOS SDK, we are able to:

• connect to the glasses
• stream the camera feed to the phone
• display a live preview on the phone
• trigger app-controlled photo capture
• receive the captured image in the app and upload it to our backend

What seems to be missing

We do not see a public way to:

• show or control the glasses-side preview/viewfinder during app-controlled capture
• invoke the native preview-and-capture workflow from DAT
• give users an in-glasses framing experience while still using our own app capture pipeline

Why this matters

Our use case is hands-free item capture for e-commerce listing workflows.

The ideal user flow is:

1. user looks at an item
2. user centers the item in the glasses preview
3. user triggers capture hands-free (voice or wrist gesture)
4. image goes directly into our app/backend workflow

Phone-side preview is technically useful, but glasses-side framing is much more important for real-world use. Users need to be able to center the item they are photographing without relying on looking back at the phone.

Request

We would like one of the following:

• support for accessing the glasses-side preview/viewfinder during DAT app-controlled capture
• an API that allows apps to invoke or integrate with the native preview-and-capture workflow
• confirmation that this is not currently supported in the public DAT SDK

Additional context

We have already built a working prototype around DAT streaming + app-controlled capture on iOS. The main blocker to product rollout is the lack of access to the in-glasses framing experience that users already get in the native capture flow.

If helpful, we can share more detail about our workflow and prototype.

[OscarFalmer]

// Moved from Issues to Discussion as this is not a SDK's bug

Thanks for your interest! The DAT SDK does not currently offer the ability to display any information on the Meta Ray-Ban Display's display but the good news is that it will come later this year! The way the SDK currently works is that you can get Meta AI Glasses' video and audio streams to your phone and then you can output to glasses's speaker or simply the phone's display any information you need to share to the user. Please let me know if you have any other questions.

[worldjoe]

I wrote a facial recognition application for my Display glasses. https://github.com/worldjoe/FriendFinder
The need to constantly stream to the iOS companion application drains the battery considerably. It would be great if the toolkit had this API call that allowed the user to frame and take a picture and then send that photo to the companion application.

[worldjoe]

looks like a duplicate of #119 and #113