FunXMPP Protocol

mgp25 edited this page Jan 14, 2017 · 4 revisions

WhatsApp Protocol - FunXMPP

WhatsApp is a popular messenger for Android, iOS and other mobile devices. This document aims to provide an explanation for how the protocol of WhatsApp, FunXMPP, works.

WhatsApp uses a protocol which is a slimmed-down version of XMPP. I won't explain the entire XMPP protocol (see RFC 6120, 6121 and 7622 for that), but it is a messaging protocol using XML as its syntax. A simple example of an XMPP message would be:

<message to="34123456789@s.whatsapp.net" type="text" id="message-1417651059-2" t="1417651059">
   <body>Test</body>
</message>

But apparently the creators of WhatsApp thought this was too bloated and found a way to express XMPP messages using only a few bytes, which they called FunXMPP. Since WhatsApp is intended for mobile devices which often lack a good internet connection, it is logical they wanted as few overhead as possible. Using FunXMPP they achieved that, while still using a standard internet protocol.

So how does FunXMPP accomplish this?

First of all, all keywords are assigned a byte. In the above example there are a lot of keywords which are common in xmpp (eg message, from, type, text).

If you can replace those by just one byte, it would reduce a lot of overhead. FunXMPP uses a HashTable for this, containing most (if not all) keywords.

Given the syntax \xnn for one byte with the hexadecimal value nn, the above example could be reduced to:

<\x59 \xa5="01234567890@\x91" \xa7="\xa2" \x44="message-1417651059-2" \xa1="1417651059">
    <\x12>Test</\x12>
</\x58>

Keeping in mind that \xnn stands for just one byte, this is already a significant reduce in size. Note that all remaining ascii values (eg 1417651059, Test, message-1417651059-2) cannot be replaced by anything because they are variable (ie can be set by the user).

XML is a human readable format employing tags that must be opened and closed. Is this really necessary for a computer to read an XML structure? The creators of FunXMPP must have thought the same thing because the other method of decreasing the size of messages is the encoding of the XML structure as a few bytes.

The only thing that remains now is the XML structure. In FunXMPP this structure is expressed as a set of lists. A list is designated by a \xf8 byte. After this \xf8 byte comes a byte with the number of items the list contains. Things that count as one item here are: the tagname, keys, values and the body.

In general: a list followed directly by a list means there are several nodes at the same level and the first list is not a tag or anything visible in the XML.

Hopefully you will now be able to communicate with the WhatsApp service. While the protocol is more efficient than plain XML, it sadly is not documented at all. With this barrier removed, hopefully the XMPP requests itself will be as described in the RFC's.

TokenMap lookup

You can check our implementation: tokenmap.class.php

Primary

0x00 =>
0x01 =>
0x02 =>
0x03 => 'account',
0x04 => 'ack',
0x05 => 'action',
0x06 => 'active',
0x07 => 'add',
0x08 => 'after',
0x09 => 'all',
0x0a => 'allow',
0x0b => 'apple',
0x0c => 'auth',
0x0d => 'author',
0x0e => 'available',
0x0f => 'bad-protocol',
0x10 => 'bad-request',
0x11 => 'before',
0x12 => 'body',
0x13 => 'broadcast',
0x14 => 'cancel',
0x15 => 'category',
0x16 => 'challenge',
0x17 => 'chat',
0x18 => 'clean',
0x19 => 'code',
0x1a => 'composing',
0x1b => 'config',
0x1c => 'contacts',
0x1d => 'count',
0x1e => 'create',
0x1f => 'creation',
0x20 => 'debug',
0x21 => 'default',
0x22 => 'delete',
0x23 => 'delivery',
0x24 => 'delta',
0x25 => 'deny',
0x26 => 'digest',
0x27 => 'dirty',
0x28 => 'duplicate',
0x29 => 'elapsed',
0x2a => 'enable',
0x2b => 'encoding',
0x2c => 'error',
0x2d => 'event',
0x2e => 'expiration',
0x2f => 'expired',
0x30 => 'fail',
0x31 => 'failure',
0x32 => 'false',
0x33 => 'favorites',
0x34 => 'feature',
0x35 => 'features',
0x36 => 'feature-not-implemented',
0x37 => 'field',
0x38 => 'first',
0x39 => 'free',
0x3a => 'from',
0x3b => 'g.us',
0x3c => 'get',
0x3d => 'google',
0x3e => 'group',
0x3f => 'groups',
0x40 => 'groups_v2',
0x41 => 'http://etherx.jabber.org/streams',
0x42 => 'http://jabber.org/protocol/chatstates',
0x43 => 'ib',
0x44 => 'id',
0x45 => 'image',
0x46 => 'img',
0x47 => 'index',
0x48 => 'internal-server-error',
0x49 => 'ip',
0x4a => 'iq',
0x4b => 'item-not-found',
0x4c => 'item',
0x4d => 'jabber:iq:last',
0x4e => 'jabber:iq:privacy',
0x4f => 'jabber:x:event',
0x50 => 'jid',
0x51 => 'kind',
0x52 => 'last',
0x53 => 'leave',
0x54 => 'list',
0x55 => 'max',
0x56 => 'mechanism',
0x57 => 'media',
0x58 => 'message_acks',
0x59 => 'message',
0x5a => 'method',
0x5b => 'microsoft',
0x5c => 'missing',
0x5d => 'modify',
0x5e => 'mute',
0x5f => 'name',
0x60 => 'nokia',
0x61 => 'none',
0x62 => 'not-acceptable',
0x63 => 'not-allowed',
0x64 => 'not-authorized',
0x65 => 'notification',
0x66 => 'notify',
0x67 => 'off',
0x68 => 'offline',
0x69 => 'order',
0x6a => 'owner',
0x6b => 'owning',
0x6c => 'p_o',
0x6d => 'p_t',
0x6e => 'paid',
0x6f => 'participant',
0x70 => 'participants',
0x71 => 'participating',
0x72 => 'paused',
0x73 => 'picture',
0x74 => 'pin',
0x75 => 'ping',
0x76 => 'platform',
0x77 => 'port',
0x78 => 'presence',
0x79 => 'preview',
0x7a => 'probe',
0x7b => 'prop',
0x7c => 'props',
0x7d => 'query',
0x7e => 'raw',
0x7f => 'read',
0x80 => 'readreceipts',
0x81 => 'reason',
0x82 => 'receipt',
0x83 => 'relay',
0x84 => 'remote-server-timeout',
0x85 => 'remove',
0x86 => 'request',
0x87 => 'required',
0x88 => 'resource-constraint',
0x89 => 'resource',
0x8a => 'response',
0x8b => 'result',
0x8c => 'retry',
0x8d => 'rim',
0x8e => 's_o',
0x8f => 's_t',
0x90 => 's.us',
0x91 => 's.whatsapp.net',
0x92 => 'seconds',
0x93 => 'server-error',
0x94 => 'server',
0x95 => 'service-unavailable',
0x96 => 'set',
0x97 => 'show',
0x98 => 'silent',
0x99 => 'stat',
0x9a => 'status',
0x9b => 'stream:error',
0x9c => 'stream:features',
0x9d => 'subject',
0x9e => 'subscribe',
0x9f => 'success',
0xa0 => 'sync',
0xa1 => 't',
0xa2 => 'text',
0xa3 => 'timeout',
0xa4 => 'timestamp',
0xa5 => 'to',
0xa6 => 'true',
0xa7 => 'type',
0xa8 => 'unavailable',
0xa9 => 'unsubscribe',
0xaa => 'uri',
0xab => 'url',
0xac => 'urn:ietf:params:xml:ns:xmpp-sasl',
0xad => 'urn:ietf:params:xml:ns:xmpp-stanzas',
0xae => 'urn:ietf:params:xml:ns:xmpp-streams',
0xaf => 'urn:xmpp:ping',
0xb0 => 'urn:xmpp:whatsapp:account',
0xb1 => 'urn:xmpp:whatsapp:dirty',
0xb2 => 'urn:xmpp:whatsapp:mms',
0xb3 => 'urn:xmpp:whatsapp:push',
0xb4 => 'urn:xmpp:whatsapp',
0xb5 => 'user',
0xb6 => 'user-not-found',
0xb7 => 'value',
0xb8 => 'version',
0xb9 => 'w:g',
0xba => 'w:p:r',
0xbb => 'w:p',
0xbc => 'w:profile:picture',
0xbd => 'w',
0xbe => 'wait',
0xbf => 'WAUTH-2',
0xc0 => 'xmlns:stream',
0xc1 => 'xmlns',
0xc2 => '1',
0xc3 => 'chatstate',
0xc4 => 'crypto',
0xc5 => 'phash',
0xc6 => 'enc',
0xc7 => 'class',
0xc8 => 'off_cnt',
0xc9 => 'w:g2',
0xca => 'promote',
0xcb => 'demote',
0xcc => 'creator',
0xcd => 'Bell.caf',
0xce => 'Boing.caf',
0xcf => 'Glass.caf',
0xd0 => 'Harp.caf',
0xd1 => 'TimePassing.caf',
0xd2 => 'Tri-tone.caf',
0xd3 => 'Xylophone.caf',
0xd4 => 'background',
0xd5 => 'backoff',
0xd6 => 'chunked',
0xd7 => 'context',
0xd8 => 'full',
0xd9 => 'in',
0xda => 'interactive',
0xdb => 'out',
0xdc => 'registration',
0xdd => 'sid',
0xde => 'urn:xmpp:whatsapp:sync',
0xdf => 'flt',
0xe0 => 's16',
0xe1 => 'u8',
0xe2 => 'adpcm',
0xe3 => 'amrnb',
0xe4 => 'amrwb',
0xe5 => 'mp3',
0xe6 => 'pcm',
0xe7 => 'qcelp',
0xe8 => 'wma',
0xe9 => 'h263',
0xea => 'h264',
0xeb => 'jpeg',

Secondary

0x00 => 'mpeg4',
0x01 => 'wmv',
0x02 => 'audio/3gpp',
0x03 => 'audio/aac',
0x04 => 'audio/amr',
0x05 => 'audio/mp4',
0x06 => 'audio/mpeg',
0x07 => 'audio/ogg',
0x08 => 'audio/qcelp',
0x09 => 'audio/wav',
0x0a => 'audio/webm',
0x0b => 'audio/x-caf',
0x0c => 'audio/x-ms-wma',
0x0d => 'image/gif',
0x0e => 'image/jpeg',
0x0f => 'image/png',
0x10 => 'video/3gpp',
0x11 => 'video/avi',
0x12 => 'video/mp4',
0x13 => 'video/mpeg',
0x14 => 'video/quicktime',
0x15 => 'video/x-flv',
0x16 => 'video/x-ms-asf',
0x17 => '302',
0x18 => '400',
0x19 => '401',
0x1a => '402',
0x1b => '403',
0x1c => '404',
0x1d => '405',
0x1e => '406',
0x1f => '407',
0x20 => '409',
0x21 => '410',
0x22 => '500',
0x23 => '501',
0x24 => '503',
0x25 => '504',
0x26 => 'abitrate',
0x27 => 'acodec',
0x28 => 'app_uptime',
0x29 => 'asampfmt',
0x2a => 'asampfreq',
0x2b => 'audio',
0x2c => 'clear',
0x2d => 'conflict',
0x2e => 'conn_no_nna',
0x2f => 'cost',
0x30 => 'currency',
0x31 => 'duration',
0x32 => 'extend',
0x33 => 'file',
0x34 => 'fps',
0x35 => 'g_notify',
0x36 => 'g_sound',
0x37 => 'gcm',
0x38 => 'gone',
0x39 => 'google_play',
0x3a => 'hash',
0x3b => 'height',
0x3c => 'invalid',
0x3d => 'jid-malformed',
0x3e => 'latitude',
0x3f => 'lc',
0x40 => 'lg',
0x41 => 'live',
0x42 => 'location',
0x43 => 'log',
0x44 => 'longitude',
0x45 => 'max_groups',
0x46 => 'max_participants',
0x47 => 'max_subject',
0x48 => 'mimetype',
0x49 => 'mode',
0x4a => 'napi_version',
0x4b => 'normalize',
0x4c => 'orighash',
0x4d => 'origin',
0x4e => 'passive',
0x4f => 'password',
0x50 => 'played',
0x51 => 'policy-violation',
0x52 => 'pop_mean_time',
0x53 => 'pop_plus_minus',
0x54 => 'price',
0x55 => 'pricing',
0x56 => 'redeem',
0x57 => 'Replaced by new connection',
0x58 => 'resume',
0x59 => 'signature',
0x5a => 'size',
0x5b => 'sound',
0x5c => 'source',
0x5d => 'system-shutdown',
0x5e => 'username',
0x5f => 'vbitrate',
0x60 => 'vcard',
0x61 => 'vcodec',
0x62 => 'video',
0x63 => 'width',
0x64 => 'xml-not-well-formed',
0x65 => 'checkmarks',
0x66 => 'image_max_edge',
0x67 => 'image_max_kbytes',
0x68 => 'image_quality',
0x69 => 'ka',
0x6a => 'ka_grow',
0x6b => 'ka_shrink',
0x6c => 'newmedia',
0x6d => 'library',
0x6e => 'caption',
0x6f => 'forward',
0x70 => 'c0',
0x71 => 'c1',
0x72 => 'c2',
0x73 => 'c3',
0x74 => 'clock_skew',
0x75 => 'cts',
0x76 => 'k0',
0x77 => 'k1',
0x78 => 'login_rtt',
0x79 => 'm_id',
0x7a => 'nna_msg_rtt',
0x7b => 'nna_no_off_count',
0x7c => 'nna_offline_ratio',
0x7d => 'nna_push_rtt',
0x7e => 'no_nna_con_count',
0x7f => 'off_msg_rtt',
0x80 => 'on_msg_rtt',
0x81 => 'stat_name',
0x82 => 'sts',
0x83 => 'suspect_conn',
0x84 => 'lists',
0x85 => 'self',
0x86 => 'qr',
0x87 => 'web',
0x88 => 'w:b',
0x89 => 'recipient',
0x8a => 'w:stats',
0x8b => 'forbidden',
0x8c => 'aurora.m4r',
0x8d => 'bamboo.m4r',
0x8e => 'chord.m4r',
0x8f => 'circles.m4r',
0x90 => 'complete.m4r',
0x91 => 'hello.m4r',
0x92 => 'input.m4r',
0x93 => 'keys.m4r',
0x94 => 'note.m4r',
0x95 => 'popcorn.m4r',
0x96 => 'pulse.m4r',
0x97 => 'synth.m4r',
0x98 => 'filehash',
0x99 => 'max_list_recipients',
0x9a => 'en-AU',
0x9b => 'en-GB',
0x9c => 'es-MX',
0x9d => 'pt-PT',
0x9e => 'zh-Hans',
0x9f => 'zh-Hant',
0xa0 => 'relayelection',
0xa1 => 'relaylatency',
0xa2 => 'interruption',
0xa3 => 'Apex.m4r',
0xa4 => 'Beacon.m4r',
0xa5 => 'Bulletin.m4r',
0xa6 => 'By The Seaside.m4r',
0xa7 => 'Chimes.m4r',
0xa8 => 'Circuit.m4r',
0xa9 => 'Constellation.m4r',
0xaa => 'Cosmic.m4r',
0xab => 'Crystals.m4r',
0xac => 'Hillside.m4r',
0xad => 'Illuminate.m4r',
0xae => 'Night Owl.m4r',
0xaf => 'Opening.m4r',
0xb0 => 'Playtime.m4r',
0xb1 => 'Presto.m4r',
0xb2 => 'Radar.m4r',
0xb3 => 'Radiate.m4r',
0xb4 => 'Ripples.m4r',
0xb5 => 'Sencha.m4r',
0xb6 => 'Signal.m4r',
0xb7 => 'Silk.m4r',
0xb8 => 'Slow Rise.m4r',
0xb9 => 'Stargaze.m4r',
0xba => 'Summit.m4r',
0xbb => 'Twinkle.m4r',
0xbc => 'Uplift.m4r',
0xbd => 'Waves.m4r',
0xbe => 'voip',
0xbf => 'eligible',
0xc0 => 'upgrade',
0xc1 => 'planned',
0xc2 => 'current',
0xc3 => 'future',
0xc4 => 'disable',
0xc5 => 'expire',
0xc6 => 'start',
0xc7 => 'stop',
0xc8 => 'accuracy',
0xc9 => 'speed',
0xca => 'bearing',
0xcb => 'recording',
0xcc => 'encrypt',
0xcd => 'key',
0xce => 'identity',
0xcf => 'w:gp2',
0xd0 => 'admin',
0xd1 => 'locked',
0xd2 => 'unlocked',
0xd3 => 'new',
0xd4 => 'battery',
0xd5 => 'archive',
0xd6 => 'adm',
0xd7 => 'plaintext_size',
0xd8 => 'compressed_size',
0xd9 => 'delivered',
0xda => 'msg',
0xdb => 'pkmsg',
0xdc => 'everyone',
0xdd => 'v',
0xde => 'transport',
0xdf => 'call-id'