Device Access Toolkit v0.8 out! 🎉

[OscarFalmer]

Hi everyone,

We’re excited to announce the release of version 0.8 of the Meta Wearables Device Access Toolkit.

📌 Release Notes

• New Meta Glasses Support
• New Wifi Video Streaming Support for consistent video quality at high resolution settings
• Various APIs improvements

🔗 Check the iOS changelog and Android changelog to see full changelogs.
🔗 Check the full Documentation on Wearables Developer Center.
🔗 Post your projects on the new Meta Subreddit for AI glasses developers.

[r0n0]

Out of the box, on a GEN1 Wayfarer (with everything updated and rebooted):

• Video seems somehow broken in high resolution: after a couple of seconds, access units contains corrupted data (verified in hvc1, using own decoding chain)

Any clue ?

[vinidlidoo]

Wanted to confirm this on a Gen2 Ray-Ban Meta (looks like the earlier report here was Gen1). I'm on DAT SDK 0.8.0 with the on-glasses app at 0.8.0.31.0, requesting .high / hvc1.

WiFi definitely lifts the resolution; I'm getting a sustained 720×1280, up from the ~504×896 I was stuck at over BT-Classic, and frame delivery is smooth at ~30fps. So when it works, it's a big step up.

The problem is it freezes within seconds to a couple of minutes. When it does, the glasses are still delivering frames at ~30fps (the link looks fine), but VTDecompressionSession rejects every single one; OSStatus -12909 (kVTVideoDecoderBadDataErr) and -17694. So the hvc1 access units are coming through corrupt, same as r0n0 saw.

It doesn't seem tied to high res, at least on Gen2. I tried .medium and it still came through at 504×896 and corrupted the same way (continuous -12909).

The only thing that recovers it is a fresh IDR. Recreating the decompression session doesn't do anything; I had it rebuilding every ~0.5s through a 72-second freeze and not a single frame decoded, which lines up with the data being bad / a missing reference rather than the decoder getting wedged. Restarting the stream brings it back (that forces a keyframe), and a hard lockup needs a glasses power-cycle.

Couple of questions:

1. Any fix or 0.8.1 timeline for the WiFi hvc1 corruption?
2. Is there any way to request a keyframe/IDR from the glasses? I don't see anything in StreamConfiguration, but it'd let the app recover on its own without tearing the stream down.

[r0n0]

thanks Vinidlidoo for confirming my own experience! I was secretly hoping that I wasn't using the right calls sequence.
Experimenting with the sample app made me to suspect that setting the ".low" quality may have a positive impact on frame sanity. I need to experiment a bit more. I have a query to the meta dev team: 🙏🏻 please document a bit more the exact, and up to date, calls sequence to make wifi streaming working, because I am sure you had something working before delivering. I just want to reproduce it without having to guess from each new sample app version 🙏🏻

[r0n0]

I made more tests and I succeeded at 100% to start a sustainable stream with the low resolution settings and a 100% corrupted stream with medium and high.
The sample app has this "low" setting hard coded, maybe a start of explanation of insufficient QA coverage. I more and more feel I am doing QA for product developed with agentic 😭

[vinidlidoo]

@OscarFalmer is there a fix? I can't roll back the SDK version on the glasses to 0.7.0 it seems and corrupted hvc1 renders the dev work unusable.

[r0n0]

it would be nice, seeing the investment of both Meta and 3rd party devs, that at least one guy forth Meta dev team is assigned to check this list and answer questions.

[metadavithom]

Hi @r0n0 and @vinidlidoo

I've been trying this out myself, and I think there is an issue where we can drop frames when you are using bluetooth as the transport channel and have DAM enabled.

BTC transport & DAM will be the default behaviour if you update your app from the 0.7 sdk to 0.8 sdk, but we have moved to Wifi in our sample app so perhaps didn't catch this.

Can you test something? Disable DAM in your MWDAT configuration and let me know if it resolves the issue for you:

<key>MWDAT</key>
<dict>
...
  <!-- Enable the Device Access Toolkit App Model (DAM) -->
  <key>DAMEnabled</key>
  <false/>

Thanks for reporting, hopefully we can resolve this issue quickly.

David.

[kelvinharron]

Hi David, thanks for sharing this. I'm seeing the same behaviour as reported on the latest DAT and with this info.plist tweak set, .high and .hvc1 will work for maybe 30 seconds before frame delivery stalls. Without the info.plist, .high will fail after just a few frames on 1st gen. I see outputs in console with this (where Unknown profile bitstream I suspect is the culprit).

I0630 16:34:33.180108 7659660 WARPStreamClientService.cpp:59] [XMS_WARP] Hera.MediaQualityMonitor: [Hera.MediaQualityMonitorManager] streamId: 117, remoteNodeId: 117, streamType: VideoReceiver, duration: 0
I0630 16:34:33.180170 7659660 WARPStreamClientService.cpp:59] [XMS_WARP] Hera.MediaQualityMonitor: [Hera.MediaQualityMonitorManager] recv bitrate: 110138, recv fps: 2, dec fps: 0, dec time ms: 0, frz_dur: 0, frz_cnt: 0, tp_frz_dur: 17636, tp_frz_cnt: 23, avg_qpx10: -1, avg_vid_tp_delay: 89, video_recv_tp_delay_high_duration: 0, video_recv_tp_delay_extreme_duration: 0, dec key frames: 0
[hevc @ 0x11c5a6300] Unknown profile bitstream

I can have a reliable session with .low set but this isn't ideal for my use case. Do you have any suggestions what else to try?

[r0n0]

Hi again David, you mentioned that you moved the sample app to use Wifi transport, that is also what I'd like to do. From your message there is something we can do to select either BTC or Wifi transport. Could you tell us how to achieve that, as I missed the info in the (quite big) sample app code ?

many thanks in advance

[sourabh-nanoti]

Hello @r0n0 ,

Info copy.plist.copy.0.8.wifi.txt
Info copy.plist.copy.0.8.btc.txt
Here are my info.plist for wifi and btc respectively.
Also please add the following in Signing & Capabilities (entitlements) in xCode:
• Access Wi-Fi Information (com.apple.developer.networking.wifi-info)
• Hotspot Configuration (com.apple.developer.networking.HotspotConfiguration)

Thanks!

[vinidlidoo]

Thanks @sourabh-nanoti. Your plists got WiFi working on our Gen2, with one extra step worth
flagging: we also had to remove the ExternalAccessory wiring from the 0.7-era integration docs
(UISupportedExternalAccessoryProtocols + the external-accessory background mode) before the
transport would engage. The current CameraAccess sample has dropped those keys too, but the
integration guide still shows them.

Results at .high/.hvc1 (Gen2, glasses app 0.8.0.31.0): sustained 720×1280@30, smooth delivery.
The hvc1 corruption still appears but self-heals in ~2 s on the next keyframe, far milder than
the BTC+DAM freeze (72+ s, unrecoverable). Two integration notes: first frame takes ~20 s from
addStream (consent prompt + hotspot join), which seems to be on the longer side; and joining
the glasses' AP displaces the phone's WiFi, so the phone's internet drops to cellular for the
whole session.

That last point is a major blocker for apps that stream the camera out, so I've opened #233 requesting
a shared-infrastructure-network mode (plus transport select/observe API and documented WiFi
setup). Would appreciate any signal on whether that's on the roadmap.

[sourabh-nanoti]

Hello @vinidlidoo , Yeah i captured the caveats in #227 . 20 sec for the first frame seems too long. i have not experienced it. It takes around 5-7 sec for me. softAP setup takes that time and may not be same for each phone model. Glad that the config helped. Thanks. I will take a look at #233

[r0n0]

@metadavithom any news or suggestion ?

[metadavithom]

Hi @r0n0 - sorry for a delay, we've been digging into this.

We've identified this as a glasses firmware bug - it is an issue where i-frames are dropped occasionally by the glasses and not sent, if the packet size gets too large. So it happens under certain noisy environments and is much more common with .high streaming.

Unfortunately, we don't have a workaround for it, you can just reduce pressure with lower settings.
On top of that if you use .raw, our SDK decoder has a bug where it freezes on the missed frame and doesn't recover.

As @vinidlidoo reported, if you use .hvc1 yourself, then you should recover at the next keyframe. If your image is moving rather than completely still, then the trail frames will also heal faster than when static.

Therefore you have the following options and fixes coming:

1. Right now, switch to .hvc1 and decode yourself. You'll still get corruption but should recover at the next keyframe.
2. We will implement a fix for 0.8 which will make the decoder robust in the same way and support .raw, in a few weeks.
3. A fix to firmware should arrive as part of v128, this will be more than a month's time due to the firmware schedule.

Thanks for helping us find this and diagnose,

David.

[r0n0]

Hi @metadavithom,
Thanks for the detailed answer. We will wait for the fix impatiently. Meanwhile I will manage my own HVC1 queue 😅

Just a last request 🙏🏻: it would be nice to have more information about setting environnement for Wifi streaming (despite I have now time to check if the latest doc is up top date) because it is really not obvious right now. many thanks in advance