Skip to content
Brendan Powers edited this page Dec 24, 2021 · 17 revisions

Protocol Definition

Design Goals

  • Simple
  • Easily understood on-wire layout
  • Composable
  • Suitable for working with hardware
  • Low boilerplate

Design Non-Goals

  • Not intended for use as a file interchange format
  • Not useful in situations where one side doesn't know the schema

Primitive Types

Numeric Types

Name Size (Bytes) Description
int8, int16, 1nt32, int64 8, 16, 32, 64 Signed integer
uint8, uint16, uint32, uint64 8, 16, 32, 64 Unsigned integer
float32, float64 32, 64 Floating point number
boolean 1 true/false value

Variable Length

Name Size (Bytes) Description
bytes Fixes, N+1 A list of uint8s
string Fixed, N+1 A null-terminated list of uint8s, with optional encoding (ascii or utf8)

Bytes (Fixed Length) bytes[n]

Fixed length byte types contain a defined number of bytes. Each byte can have any value, including NULL.

For Example:

Type bytes[4]
Byte 1 2 3 4
Value 0x05 0x00 0xAF 0xDE

Bytes (Variable Length) bytes[]

Variable length byte arrays may be any size up to 255 bytes. When one of these fields is serialized, the first byte indicates the size, while the remaining bytes contain the data.

For example:

Type bytes[]
Byte 1 2 3 4
Value 0x03 0xAA 0x00 0xDE
Purpose Size Data

String (Fixed) string[n]

Strings are always null terminated. If a string has a fixed length, any un-used characters will be null.

For example:

Type string[5]
Byte 1 2 3 4 5
Value H e y 0x00 0x00

Strings must always contain a null byte. The below example is invalid, because there is no room to store the null byte.

Type string[5]
Byte 1 2 3 4 5
Value H e l l o

String (Variable) string[]

Unlike bytes[] variable lengths strings do not contain a size byte. Instead, the serializer will read untill it encounters the first null byte.

For example:

Type string[]
Byte 1 2 3 4
Value H e y 0x00

Bitfield Types

Not for V1

Name Size (Bits) Description
uint{N} N bits Unsigned integer
int{N} N bits Signed integer
flag 1 bit True/False value
unused{N} N bit reserved or padding bits

All types have a maximum length of 64 bits.

Enums

Enums are a set of named constants that help make a protocol defenition easier to understand. For example:

enum PinDirection: uint8 {
  Input = 0
  Output = 1
  Floating = 2
}

Enums are always numerical, and you must specify the underlying type when declaring the enum. The enum above is stored using a uint8.

Struct

A struct is a composite data type made of named members of other data types. For example, this struct contains three members, each with a different type.

struct MyTestStruct {
  a: int32,
  b: boolean,
  c: uint16
}

Structs can contain any primitive type, bytes, string, arrays, enums, bitfield structs and other structs. They cannot contain bitfield types (unless using an embedded bitfield). The members of a struct are encoded in-order. There is no type or size information encoded with the struct.

For example:

Type MyTestStruct
Byte 1 2 3 4 5 6 7
Member a b c

Bitfield Structs

Not for V1

A bitfield struct lets you store data which is not byte aligned. These can be used in situations where memory or network bandwidth is very tightly constraoned, or, when communicating with register based hardware. I2C devices, for example.

Below is an example of a bitfield defenition for the control register of a TMP1075 tempurature sensor.

enum ConversionRate: uint{2} {
  Rate_27_5  = 0  # 27.5ms
  Rate_55    = 1  # 55ms
  Rate_110   = 2  # 110ms
  Rate_220   = 3  # 220ms
}

bitfield struct ControlRegister {
  oneShot: flag
  conversionRate: ConversionRate
  conversionFaults: uint{2}
  outputPriority: flag
  alertFunction: flag
  shutdownMode: flag
  reserved: unused{8}
}

Embedded bitfields

Bitfield types are not allowed in structs. You can include bitfield structs as members of a standard struct (see limitations below). Sometimes though, this can become verbose. In cases where you only need a few non-byte alligned fields in an otherwise normal struct, embedded bitfields may be useful.

For example this struct uses an embedded bitfield to represent the states of 8 pins in a port.

struct WritePort {
  portNum: uint8
  bitfield {
    pin1: flag
    pin2: flag
    pin3: flag
    pin4: flag
    pin5: flag
    pin6: flag
    pin7: flag
    pin8: flag
  }
}

Limitations

Due to their flexibility, bitfield structs may have a size that's not an even multiple of 8bit. This causes many non-byte aligned reads. This is size efficient, but makes them expensive to serialize. To limit this impact, in order to use a bitfield in a normal struct (embedded or otherwise) it must be a multiple of 8 bits. This limitation also applies when directly serializing a bitfield struct. If your bitfield struct is not naturaly a multiple of 8 bits, the unused{n} data type can be used to pad the data to the right size.

Arrays

Arrays are created by appending square brackets to the end of a type. [n] for fixed leng arrays, [] for variable length arrays. There is some ambiguity when using arrays of strings or bytes. In such cases, the array indicator is placed after the type size. For example. A list of 5 variable length strings would look like this string[][5].

Fixed Length

Fixed length arrays always contain all their elements. For example, an array of type uint16[3] will be encoded like this:

Type uint16[3]
Byte 1 2 3 4 5 6
Element 1 2 3

Variable Length

Variable length arrays are encided with a length byte, followd by N elements. For example:

Type uint16[]
Byte 1 2 3 4 5 6 7
Element Size 1 2 ...

Choice Types

Not for V1

Union Types

Not for V1

Endianness

Not for V1

This will wait for a future version, but a default endianness will be assumed. It can then be changed on a protocol, struct, or field level.

Framing

A few different framing types will be supported. For the first verionsion, only COBS will be implemented.

Fixed

Not for V1

Frames messages into fixed length packets. Use this when your underlying hardware can reliably deliver corruption free fixed length messages.

This is not suitable for serial or TCP links.

Length Based

Not for V1

A simple framing methid where a 4 byte length prefix is added to the data before it is transmitted. It's a lightweight protocol suitable for situations where the underlying stack provides error correction.

This would be suitable for use with TCP, or with serial protocols that have well defined frame boundaries. It is not suitable for Serial, since you may start reading in the middle of a packet.

COBS

COBS is a robust framing alorithm with a low, fixed overhead. It it is also computationally efficient. Combined with CRC error checking, it is robust against corruption and synchronization loss. See the Wikipedia article for more details.

COBS is suitable for serial protocols.

Error Checking

By default, CRC32 error checking is used to ensure the frame has been decoded without errors. This can be disabled if the link is reliable. Future versions may implement other error detection/correction schemes.

Protocol