Live stream

The live stream sends the glasses' camera video and audio to the phone over the Wi-Fi Direct session. The phone asks for it on service 28 and service 51350. This page covers the start order, the messages, the frame and audio formats, the health events and the stop reasons.

Start order

The phone starts the glasses-side app on service 28, then configures the stream on service 51350. Everything runs over the Wi-Fi session once it is up.

1. The phone subscribes to device status over Wi-Fi, before it starts an app.
2. The phone sends svc28 type 2000 StartApp with field 1 set to the app id. The stream uses app id 3 (Livestream). Live AI uses 28.
3. The glasses answer with svc28 type 2001 and an error code in field 1.
4. The phone sends svc51350 type 27 SessionSettings.
5. The phone sends svc51350 type 5 ConfigureVideoStreamRequest, and the glasses answer with type 6.
6. The phone sends svc51350 type 3 ConfigureAudioStreamRequest, and the glasses answer with type 4.
7. The phone sends svc51350 type 7 ControlStreamRequest with action 1 (START), and the glasses answer with type 8.

Each request that expects an answer carries a nonce. The answer echoes it.

Message	Fields the phone sends	Answer
svc28 type 2000 StartApp	1 app id	svc28 type 2001, 1 error code
svc51350 type 27 SessionSettings	3 timers (1 doff timer, 2 inactivity timer, in seconds), 4 application type, 5 overrides	None
svc51350 type 5 ConfigureVideoStreamRequest	1 nonce, 2 VideoConfig, 3 enabled	Type 6, nonce and error
svc51350 type 3 ConfigureAudioStreamRequest	1 nonce, 2 AudioConfig, 3 enabled	Type 4, nonce and error
svc51350 type 7 ControlStreamRequest	1 nonce, 2 action (1 START, 2 STOP)	Type 8, nonce and error

Overrides (field 5) hold 1 bypass DON check, 2 media batching interval in ms, 3 allow frame capture and 4 allow OCR streaming. The Live stream sends application type 3.

Error codes in answers: 0 success, 1 internal error, 2 invalid parameter, 3 unsupported parameter, 4 glasses not worn, 5 busy. Service 28 start answers: 1 started, 12 already running, 5 unsupported app, 6 not enough battery, 7 thermal throttling, 9 start failed, 13 not enough priority, 15 denied by peak power.

On 5 GHz at 80 MHz the stream stops 6 to 15 s after it starts. See The 80 MHz firmware hang.

sequenceDiagram
    participant Phone
    participant Glasses
    Note over Phone,Glasses: Wi-Fi session up
    Phone->>Glasses: svc28 type 2000 StartApp
    Glasses-->>Phone: svc28 type 2001 error code
    Phone->>Glasses: svc51350 type 27 SessionSettings
    Phone->>Glasses: svc51350 type 5 ConfigureVideoStream
    Glasses-->>Phone: type 6 answer
    Phone->>Glasses: svc51350 type 3 ConfigureAudioStream
    Glasses-->>Phone: type 4 answer
    Phone->>Glasses: svc51350 type 7 ControlStream START
    Glasses-->>Phone: type 8 answer
    Glasses-->>Phone: type 1 parameter sets
    loop each frame
        Glasses-->>Phone: type 10 chunks, type 12 closes frame
        Glasses-->>Phone: type 9 audio chunks, type 11 closes packet
    end
    Glasses-->>Phone: type 21 health events
    Phone->>Glasses: svc51350 type 7 ControlStream STOP
    Glasses-->>Phone: type 8 answer
    Note over Phone,Glasses: Wi-Fi left, Bluetooth resumed

Loading

The diagram shows one frame loop. The glasses send frames continuously until the phone sends STOP or the glasses stop the stream on their own.

Stream messages

Service 51350 is media streaming control. See Service-catalogue.

Type	Name	From	Meaning
1	CodecConfigurationUpdateRequest	glasses	Parameter sets. Field 2 says which: 1 audio, 2 video. Field 3 holds the bytes.
2, 4, 6, 8	Configure and control answers	glasses	Nonce and error
3	ConfigureAudioStreamRequest	phone	Audio on or off, and its format
5	ConfigureVideoStreamRequest	phone	Video format and bitrate
7	ControlStreamRequest	phone	START (1) or STOP (2)
9	AUDIO_CAST_BUFFER	glasses	Audio chunk
10	VIDEO_CAST_BUFFER	glasses	Video chunk
11	AUDIO_CAST_BUFFER_END	glasses	Last audio chunk, carries data
12	VIDEO_CAST_BUFFER_END	glasses	Last video chunk, carries data
13	ControlStreamStopped	glasses	Field 2 holds the reason
21	SystemHealthEvent	glasses	Status, type and level
22	SupportedConfigurationUpdateRequest	glasses	Field 2 holds the highest quality the glasses offer now
23	UserEventNotification	phone	Field 2 set to 4 asks for a photo while streaming
27	SessionSettings	phone	See Start order
34, 35	PHOTO_CAST_BUFFER, _END	glasses	HEIC chunks of a photo
36	METADATA_CAST_BUFFER	glasses	Carries OCR only

Video format

The video is HEVC (mime video/hevc). Video comes as chunks of type 10. The last chunk of a frame is type 12, and it carries data too. The chunks of one frame are joined, then read as:

Bytes	Content
0 to 7	Presentation time in microseconds, unsigned, little-endian, on the glasses' clock
8 onward	One access unit of Annex-B HEVC: start codes, then NAL units

A frame is a keyframe when it has a NAL unit of type 16 to 21. Parameter sets (VPS 32, SPS 33, PPS 34) come on type 1 with field 2 set to 2. The SPS gives the picture size.

Audio format

Audio uses the same shape. Chunks of type 9 are closed by type 11. The first 8 bytes are the time, and the rest is one raw AAC-LC access unit. The AudioSpecificConfig comes on type 1 with field 2 set to 1.

The phone asks for AAC-LC audio with ConfigureAudioStreamRequest (type 3). Its field 2 holds an AudioConfig:

Field	Content
1	Mime, audio/mp4a-latm
2	Sample rate, 48000 Hz
3	Bitrate, 96000 bit/s
4	Channels, 2 (stereo)
5	AAC profile, 0 (AAC-LC)

The Live stream sets enabled (field 3) to 1. Live AI sets it to 0, because its speech goes over svc15 instead. See Assistant-protocol.

VideoConfig

ConfigureVideoStreamRequest carries a VideoConfig in field 2:

Field	Content
1	Mime, video/hevc
2	Width
3	Height
4	Bitrate
5	Frame rate
6	I-frame interval, in seconds
7	eisMode, 1
8	Capture frame rate

The Live stream sends an I-frame interval of 1. Live AI sends 0, which asks for every frame to be a keyframe.

Photo

A photo comes as chunks of type 34 closed by type 35. The HEIC arrives bare, so ftyp sits at byte 4. The phone asks for one with type 23 and field 2 set to 4.

Health events

SystemHealthEvent (type 21) has three fields. Field 1 is the status: 1 Critical, 2 PreCritical, 3 Warning, 4 Normal, 5 Ideal. Field 2 is the type: 1 connectivity, 2 battery, 3 thermal, 4 peak power. Field 3 is the level, a signed 64-bit value.

The glasses send these events on their own over Wi-Fi. Bluetooth device status also reports heat and battery.

Stop reasons from the glasses

The glasses send ControlStreamStopped (type 13) with a reason in field 2:

• 4: battery low
• 5: thermal threshold
• 8: doff
• 9: temples closed

Other values exist. The full list is unconfirmed.

Stopping from the phone

1. The phone sends ControlStreamRequest with action 2 (STOP), if the link is up. The glasses answer with type 8.
2. The phone sends LeaveAccessPoint over Wi-Fi, unless the link has dropped, and stops the TCP session.
3. The phone removes the group, then resumes the Bluetooth session. See Switch from Bluetooth to Wi-Fi.

Live AI

Live AI runs the glasses' MMAI_LIVE app over the Bluetooth session, not over Wi-Fi. Its speech runs on svc15. See Assistant-protocol.

The stream runs over the Wi-Fi Direct session, because Bluetooth is too slow for video. See Wi-Fi-Direct.

Reference code: StreamProto.kt, LiveStream.kt

Open questions

• Unconfirmed: whether 1080p at 30 fps holds on a long stream.
• The effect of application type 3 on the glasses is unknown.
• Unconfirmed: how often the glasses send health events.
• Unconfirmed: whether Bluetooth carries about 1 Mbit/s of video.
• Unconfirmed: the full list of ControlStreamStopped reasons.