Assistant protocol

This page covers svc15, the assistant service the glasses use for "Hey Meta". It shows who answers a turn, how the phone registers as the assistant, and how an answer reaches the glasses. Event ids are on Assistant-events. Voice, language, follow-up and photo details have their own pages: Assistant-voice-and-languages and Assistant-follow-ups-and-photos.

Roles

The glasses recognize the wake word and the first question on their own speech model. They send the result to the phone as ev57, with the transcript and their own understanding of the request.

The glasses expect a server behind svc15. The phone plays that server. It registers as the assistant, answers each ev57, and returns the reply as audio or as text for the glasses to speak. Meta's servers take no part in a turn.

The phone hears the microphone only when the glasses leave a question to the server. That happens for follow-ups after an answer, and for Live AI. See Assistant-follow-ups-and-photos.

Channels

Two channels on svc15 carry a registered link:

Channel	Registered with	Answered with	Carries
m0 (message)	type 1000	type 1001	NetEvents at type 1010, two events at tagged type 0, envelope acks at type 9001
s0 (stream)	type 5000	type 5001	Opus audio: header at type 5002, first message at type 5003, bare continuations, end at type 6000

The glasses use their own channel ids for the streams they open. They wrap what they send in type 9000 envelopes.

Bring-up

The phone runs bring-up once the link is ready. It runs again after the glasses close the assistant channels, and when the assistant moves onto a Wi-Fi session.

1. Open m0 and send the registration at type 1000, tagged with service 15. The body is field 1 (session string, "-"), field 2 (client uuid), field 3 ("-m0") and field 4 (varint 2). Wait for type 1001.
2. Open s0 and send the same registration at type 5000 with "-s0" in field 3. Wait for type 5001.
3. Declare seven companion resources on m0, in this order. Type 11 carries "-" in field 1. Types 1031 and 1038 carry a uuid in field 1. Type 1016 is empty. Type 1029 is the string "1". Types 1039 and 1032 carry a uuid in field 1.
4. Announce a session on m0 with four NetEvents: ev265 (open), ev263 with field 4 = 1, ev263 with field 4 = 2, ev262 (close).
5. The glasses now route "Hey Meta" to the phone.

Without the resources or with them out of order, the glasses decide that no assistant is behind the client. Without the session announcement they never hand over a transcript.

Field 4 of the registration is 2 in every registration. The meaning of that field is unknown.

Continuous responder

The phone reads svc15 messages on the link's reader at all times, whether or not a turn is in progress.

• Every envelope (type 9000, fields 1 seq, 2 inner type, 3 payload) gets an ack (type 9001, field 1 seq) when it has a seq. Unacked, the glasses resend the same envelope for minutes.
• An inner type 1010 carries a NetEvent. For ev289 the phone confirms the interaction with ev71, once per interaction id. The glasses wait for that confirmation before they hand over a transcript.
• ev339 (interaction state) is echoed as ev333 with the same integer fields, sorted by field number.
• ev57 starts an answer. See the next section.
• An inner type 1023 (START_INTERACTION) names a use case in field 5. Use case 3 is Live AI. Use case 7 is a session turn.
• The glasses' type 5002 names the stream id they allocated for an interaction (field 1), with the interaction id in field 2. The phone keeps one id per interaction and falls back to the latest one.
• Type 5003, or a type 0 message with no tag, on s0 is microphone audio from the glasses. Type 6000 on s0 ends it.
• Type 5005 reports how much of an answer the glasses decoded. See The 5005 diagnostic.

One "Hey Meta" turn

The diagram shows one question the glasses recognize and the phone answers with a spoken reply.

sequenceDiagram
    participant Glasses
    participant Phone
    Glasses->>Phone: type 5002 header and microphone audio on s0
    Glasses->>Phone: 9000 envelope with ev339 state
    Phone->>Glasses: 9001 ack and ev333 mirror
    Glasses->>Phone: 9000 envelope with ev57 question
    Phone->>Glasses: 9001 ack
    Phone->>Phone: make the answer
    loop while the answer is being made
        Phone->>Glasses: ev392 AwaitingResponse
    end
    Phone->>Glasses: ev71, ev52 and ev342 on m0
    Phone->>Glasses: ev388 on m0
    Phone->>Glasses: type 5002 header on s0
    Phone->>Glasses: type 5003 with two Opus packets
    Phone->>Glasses: ev53 VARM at type 1010
    Phone->>Glasses: bare Opus messages, one every 40 ms
    Glasses->>Phone: ev339 state, mirrored as ev333
    Phone->>Glasses: type 6000 on s0, stream end
    Phone->>Glasses: ev389 and ev392 on m0
    Glasses->>Phone: type 5005 decoded bytes and milliseconds

Loading

Each step in order:

1. The glasses open their microphone stream for the turn. The stream header is type 5002, with the glasses' interaction id in field 2.
2. The glasses send their state as ev339. The phone acks the envelope and mirrors the state as ev333 at once.
3. The glasses send ev57 in a 9000 envelope. The phone acks it. A request that arrives before both channels are registered is dropped. The glasses send it again once they are.
4. The phone reads the transcript, the intent and the slots, and decides what to answer. A turn that the glasses fulfill themselves, or a wake word with nothing after it, gets no answer. See the ev57 table below.
5. While the answer is being made, the phone sends ev392 with AwaitingResponse and a counter, so the glasses keep waiting.
6. The preamble comes first: ev71, then ev52 and ev342 at tagged type 0.
7. ev388 marks the start of the speech. The stream header goes to s0 with the glasses' stream id in field 1.
8. The first audio message carries two Opus packets. The VARM (ev53) follows on m0 at type 1010.
9. The remaining audio goes out as bare DataX messages at real-time pace. The glasses' state change lands here, and the phone mirrors it from the reader.
10. Type 6000 ends the stream. ev389 and ev392 close the delivery.
11. The glasses report what they decoded in type 5005.

The ev57 request

The glasses send ev57 as a fulfillment request with the fields below.

Field	Content
1	interaction id, string
2.1.1	transcription of the question
2.4	visual context: 2.4.3 image uuid, 2.4.4 inline image, 2.4.8 capture handle, 2.4.16 second inline image, 2.4.20 image needed
2.5	image capture confirmed
7, 8	stream id as an integer, and as a string. The string applies when no type 5002 gave one
23	intent, such as IN:REPLY_MESSAGE
26	slots: a map from slot name to a list of {1 name, 4 text, 5 start, 6 end}
55.1	on-device intent
55.8	multimodal invocation confirmed
24	domain
28	utterance

Routing on the intent:

• IN:START_LIVE_AI: the glasses start Live AI themselves. No answer.
• IN:WAKE_WORD: the glasses close the turn at once. No answer.
• IN:REPLY_MESSAGE and the cancel intents: a reply-message flow. This page does not cover it.
• Anything else: the phone answers.

The answer

The phone fills a fixed template VARM rather than encoding the struct from scratch. Re-encoding changes integer widths, and the glasses reject the result.

VARM field	Meaning	Phone
1.2.1	transcript echo	filled in, or the answer text when the transcript is empty
1.3[0].2.1	answer text	filled in
6	streamingBackTts	true in the template, false for the glasses' own voice
8	interaction id	filled in
13	speakRequestVoiceId	filled in for the glasses' own voice
14	isUsingPartialResponseStreaming	false for the glasses' own voice
7	speakRequestText	not set. It has no effect on the glasses' own voice
16, 21, 24, 26, 27	product domain, intent, expected template, fulfillment type, model name	kept from the template

The intent in ev342 is the fixed string IN:GET_WEATHER. The glasses accept it. Whether they use it for anything is unknown.

Streamed audio

The phone streams the answer as Opus audio on s0. The stream header asks for audio encoding 1, which the glasses map to audio/opus-demo.

The packets are 48 kHz mono Opus, 20 ms each. Each message holds two packets, and each packet is prefixed with its length as a little-endian u32. The phone sends one message every 40 ms, so the stream runs in real time.

Glasses' own voice

The phone can send a non-streaming answer and let the glasses speak the text. The phone sends the VARM with fields 6 and 14 false and no audio stream. Right after it the phone sends ev60 (MwaVoiceInteractionComplete) with the text:

• field 1: interaction id
• field 2: status 1000, which is success
• field 3: empty string
• field 4: fulfillment result, with field 1 the text, field 2 fulfillment type 1 (server), and fields 3, 4, 7 and 8 false. Field 8 is streamingBackTts.

The glasses then speak the text in the voice named by assistant_voice_id. The phone sends ev392 after it.

Session turns (use case 7) never take this path. On that path the glasses do not speak text the phone sends, so the phone streams audio there. The English case is on Assistant-voice-and-languages.

Silent faults

Each of these fails with no error and no status. The glasses take the turn and play nothing, or play only a chime.

1. A continuation audio message carries a bare DataX header, with no message type and no service tag. A tagged type 0 makes the glasses read a message of type zero, and only the first 40 ms ever decode. The phone sends continuations with no tags.
2. The ev339 mirror must go out during the audio burst. The state change lands about 90 ms after the stream header. Mirroring after the stream ends is too late. The phone mirrors from its reader, so the mirror runs during the burst.
3. The VARM goes after the first audio message, never before it. It is sent at type 1010.
4. Field 1 of the stream header must be the stream id the glasses allocated in their type 5002. Without it the glasses drop the stream before they log anything.
5. An answer goes out only when both channels are registered. An answer sent earlier streams to nothing and marks the request done, so the retry is dropped.
6. Every NetEvent writes field 4 (uncompressed size) as i64. An i32 makes the glasses discard the whole event.
7. Only one writer may use the socket. Each direction of the cipher keeps its own counter, and a second writer breaks every later frame without an error. Serialize sends with the link's lock.

The 5005 diagnostic

The glasses report what they decoded in type 5005. The payload is text:

<stream-id>_<interaction-id>_<pcm-bytes>_<milliseconds>

The glasses decode 96 bytes of PCM per millisecond (48 kHz, mono, 16 bit). A complete answer shows pcm-bytes close to milliseconds times 96.

• pcm-bytes well below milliseconds times 96 means the first audio message only was decoded. That is fault 1 in the list above.
• Zero over zero means the glasses dropped the stream before decoding it. The usual causes are a missing field 1 in the header, or a VARM at the wrong message type.

Timing

• The glasses wait 20 s after an ev57 for an answer.
• Audio pacing is one message of two 20 ms packets every 40 ms.

The canned error

When an ev57 gets no answer in time, the glasses speak a fixed line: "Open the Meta AI app on your phone to reconnect". Unconfirmed: the exact trigger for this line.

Attention states

The glasses report an attention state through the svc28 device state, field assistant_attention, type 2037. The values are UNKNOWN, IDLE, WAKE, LISTENING, PROCESSING, RESPONDING and LONG_RUNNING_SESSION_ACTIVATED. The diagram covers the five turn states.

stateDiagram-v2
    [*] --> IDLE
    IDLE --> WAKE: wake word heard
    WAKE --> LISTENING: microphone open
    LISTENING --> PROCESSING: ev57 sent with the question
    PROCESSING --> RESPONDING: answer audio starts
    RESPONDING --> IDLE: stream ends
    LISTENING --> IDLE: no question

Loading

Unconfirmed: the transitions. They are inferred from the order of events in a turn.

Reference code: Assistant.kt, Oxide.kt

Open questions

• The meaning of field 4 in the registration, which is 2 in every registration.
• Whether the glasses need a pause between the bring-up messages. The phone sends them back to back.
• The bodies of types 1036 and 1037 are unknown. Type 1026 may carry more than the locale.
• Whether ev71 must be sent once per interaction or once per stream. The phone sends it once per interaction id.
• The meaning of fields 7 and 19 in the stream header. The phone sends 1 for both.
• Field 13 (speakRequestVoiceId) on the glasses' voice path. The phone fills it in. Whether it has any effect is unconfirmed.
• Whether non-streaming with speakRequestText can speak. The phone does not use it.
• The meaning of the fixed intent IN:GET_WEATHER in ev342.
• The transitions of the attention states.
• Whether a second turn on the same link works in every case.