Skip to content

Technical Wire Format

Jump to bottom
Rylan Meilutis edited this page Jun 29, 2026 · 7 revisions

Wire Format (Technical)

This page documents the compact v2 wire format implemented in src/wire_format.rs (source).

Public API Names

The wire-format module is named wire_format; the former generic encoding module path is not part of the API.

Rust callers use:

  • sedsnet::wire_format::pack_packet(...)
  • sedsnet::wire_format::unpack_packet(...)
  • sedsnet::wire_format::pack_packet_with_reliable(...)
  • sedsnet::wire_format::pack_reliable_ack(...)
  • sedsnet::wire_format::peek_envelope(...)
  • sedsnet::wire_format::peek_frame_info(...)
  • sedsnet::wire_format::packet_wire_size(...)

Python callers use:

  • Packet.pack()
  • unpack_packet_py(...)
  • peek_header_py(...)

C/C++ callers use the packed-wire names:

  • seds_pkt_pack_len(...)
  • seds_pkt_pack(...)
  • seds_pkt_unpack_owned(...)
  • seds_pkt_unpack_header_owned(...)
  • seds_pkt_validate_packed(...)
  • packed side APIs such as seds_router_add_side_packed(...), seds_router_rx_packed_packet_to_queue(...), and seds_relay_rx_packed_from_side(...)

These functions pack and unpack SEDSnet protocol frames.

Frame layout

[FLAGS: u8]
    bit0: payload compressed
    bit1: reserved
    bit2: wire contract present
    bit3: packet nonce present
    bit4: E2E encrypted payload wrapper present
    bit5: endpoint bitmap present
    bit6: reliable header uses compact varint form
    bit7: reserved
[NEP: u8]                         // number of selected endpoints
VARINT(ty: u32 as u64)            // ULEB128
VARINT(data_size: u64)            // logical payload size after decompression
VARINT(timestamp_ms: u64)
[VARINT(nonce: u16 as u64)]       // only when bit3 is set
VARINT(src_addr: u32 as u64)      // assigned/discovered source address
[ENDPOINT_BITMAP]                 // only when bit5 is set
[VARINT(contract_len: u64)]       // only when bit2 is set
[WIRE_CONTRACT_BYTES]
[RELIABLE_HEADER]                 // present for reliable schema types or when contract says so
PAYLOAD_BYTES                     // raw/compressed, or E2E wrapper around those bytes
[CRC32: u32 LE]                   // checksum of every prior byte in the frame

Flags

Top-level frame flags:

  • 0x01: payload compressed
  • 0x04: wire contract present
  • 0x08: packet nonce present
  • 0x10: payload bytes are wrapped by the feature-gated E2E cryptography provider
  • 0x20: endpoint bitmap present
  • 0x40: reliable header uses compact varint form

When 0x10 is set, routing metadata remains visible, but the payload region is:

VARINT(key_id)
VARINT(plaintext_wire_payload_len)
VARINT(nonce_len)
NONCE_BYTES
VARINT(tag_len)
TAG_BYTES
CIPHERTEXT_BYTES

The authenticated data passed to the cryptography provider is the packed frame prefix through the optional reliable header, excluding the payload wrapper and CRC. Unpackers built without cryptography reject frames with 0x10 rather than exposing ciphertext as application data.

For multi-board endpoints, a sender can use an application-managed endpoint/group traffic key so each listed board can decrypt the same payload. Any board that changes visible routing metadata or ciphertext invalidates the AEAD tag for the other boards.

Reliable-header flags:

  • 0x01: ACK-only reliable control frame
  • 0x02: reliable but unordered
  • 0x80: unsequenced best-effort reliable wrapper

ACK-only reliable control frames are emitted by the router or relay reliable layer. They are not valid application Packet values and are consumed before normal packet unpacking.

Endpoint bitmap

NEP is the number of selected endpoints for the frame.

When flag 0x20 is clear, no endpoint bitmap bytes are present. The endpoint set is the default endpoint set from the local data type metadata for ty, expanded in ascending endpoint-ID order, and NEP must match that set size.

When flag 0x20 is set, a fixed-width endpoint bitmap follows src_addr. This form is used for custom endpoint sets, subsets, ACK-only reliable control frames, wire-contract frames, and frames whose endpoints cannot be inferred from the data type metadata.

  • EP_BITMAP_BITS = MAX_VALUE_DATA_ENDPOINT + 1
  • EP_BITMAP_BYTES = ceil(EP_BITMAP_BITS / 8)

Bitmap packing is LSB-first within each byte. NEP is the popcount of the bitmap and is used as a sanity check during decode.

The bitmap width is stable for a given build. Adding or removing runtime schema entries does not change the bitmap width.

Wire contract

When FLAG_WIRE_CONTRACT is set, the frame carries a compact contract immediately after the source address bytes.

VARINT(contract_len)
[contract flags: u8]
[wire shape bytes]                // if contract flag 0x02 set
[target count: ULEB128]           // if contract flag 0x01 set
[target sender hash 0: u64 LE]
[target sender hash 1: u64 LE]
...

Contract flags:

  • 0x01: explicit frozen destination sender hashes are present
  • 0x02: inline payload shape is present
  • 0x04: a reliable header is present even if current schema lookup would no longer imply that

Inline payload shape

The shape is packed compactly:

[packed: u8]
    bits 0..3: MessageDataType code
    bits 4..5: MessageClass code
    bit 6: static-layout flag
[static_count: ULEB128]           // only when bit 6 is set

When present, the inline shape is used to construct the Packet instead of the current registry definition.

Frozen destination sender hashes

The target list contains u64 sender hashes for destination holders. Routers and relays use this list as an explicit destination set for the packed frame.

Reliable header

The reliable header appears after the optional wire contract for reliable schema types or when the wire contract says a reliable header is present.

When top-level flag 0x40 is clear, the reliable header is the fixed 9-byte form:

[REL_FLAGS: u8]
[SEQ: u32 LE]
[ACK: u32 LE]

When top-level flag 0x40 is set, the reliable header is the compact form:

[REL_WIRE_FLAGS: u8]
    bit0: ACK-only reliable control frame
    bit1: reliable but unordered
    bit2: SEQ varint present
    bit3: ACK varint present
    bit7: unsequenced best-effort reliable wrapper
[VARINT(seq)]                    // only when REL_WIRE_FLAGS bit2 is set
[VARINT(ack)]                    // only when REL_WIRE_FLAGS bit3 is set

The compact form is used only when it is smaller than the fixed form. Normal reliable data frames with small sequence numbers and no piggyback ACK encode as REL_WIRE_FLAGS + VARINT(seq). ACK-only reliable frames with small ACK values encode as REL_WIRE_FLAGS + VARINT(ack).

Reliable control traffic now primarily uses built-in internal packet types such as:

  • ReliableAck
  • ReliablePartialAck
  • ReliablePacketRequest
  • DiscoveryAddress
  • P2pMessage

Those are router/relay-owned control packets, not user endpoint traffic.

Their payload layouts are documented in Technical-Discovery-and-Internal-Formats.

Side Transport Wrappers

Routers and relays can add a side-local wrapper around packed frames for constrained links. This wrapper is not part of the application Packet; it is consumed by rx_packed_from_side(...) before normal unpacking.

[magic: "SDT"]
[kind: u8]
[body bytes]
[CRC32: u32 LE]                   // checksum of magic + kind + body

Kinds:

  • 0x01: full packed frame plus a side-local ULEB template id
  • 0x02: compact frame using a previously learned side-local template id
  • 0x03: ordered chunk of a full or compact side-transport frame
  • 0x04: compact frame using a template id plus timestamp delta from the previous frame for that template
  • 0x05: compact frame using a template id and the unchanged previous timestamp for that template

Router packed sides support header-template reuse with Router::add_side_packed_small_packets(...) or RouterSideOptions::with_small_packet_transport(...). The first stable header shape is sent as a full SDT frame and assigns a compact side-local ULEB template id. Later packets with the same static header shape can use kind 0x02, replacing repeated type/endpoint/sender/contract bytes with that template id plus the fields that still vary per packet. When the previous timestamp for that template is known and the nonnegative delta is smaller than the full timestamp varint, the sender uses kind 0x04 and carries the timestamp delta instead. When unchanged-timestamp omission is enabled and the timestamp is identical to the previous frame for that template, the sender uses kind 0x05 and omits the timestamp field entirely. Omission can be enabled side-wide, by the IPv4-like profile, or for selected data types on a mixed link.

Python and C bindings expose the same profiles through add_side_packed_profile(...) and seds_*_add_side_packed_profile(...). ipv6_like uses a 40-byte compact-header profiling target; ipv4_like uses a 20-byte target and enables unchanged-timestamp omission.

Router and relay packed sides both support bounded frame sizes. Relay small-packet sides use the same side-local template id compaction when max_frame_bytes is non-zero. When the side-transport frame is too large, the sender emits kind 0x03 chunks whose individual callback payloads do not exceed the configured maximum. The receiver reassembles those chunks into the original side-transport frame and then resumes normal packet processing.

Compact Side Profiles

Routers and relays expose per-side compact-header profiles. with_small_packet_transport sets the compact-header profile to ipv6_like; with_ipv4_like_compact_header_target sets it to ipv4_like.

Each side also exports an effective side-transport profile:

  • canonical: no side-transport wrapping is needed
  • template: template reuse is enabled without an IPv4/IPv6-size target
  • ipv6_like: compact follow-up frames are profiled against a 40-byte target
  • ipv4_like: compact follow-up frames are profiled against a 20-byte target and omit unchanged compact timestamps

Runtime stats report:

  • whether header templates are enabled on each side
  • max fixed-frame size
  • compact-header target bytes
  • effective side-transport profile
  • maximum retained side-local templates
  • full, compact, timestamp-delta compact, unchanged-timestamp compact, and chunk side-transport frame counts
  • raw canonical bytes, emitted side-transport bytes, and bytes saved
  • minimum and maximum compact follow-up overhead bytes
  • compact-header target misses
  • active TX/RX template counts and template evictions

Side-local template dictionaries are bounded by max_side_transport_templates, which defaults to 64 entries per side. When the dictionary is full, the sender or receiver evicts a deterministic entry and later refreshes that shape with a full template frame.

Discovery and Internal Payloads

Discovery, P2P, managed-variable, reliable-control, and time-sync packet payloads are documented in Technical-Discovery-and-Internal-Formats.

Varints

All integer metadata fields use unsigned LEB128.

This includes:

  • type ID
  • payload size
  • timestamp
  • sender lengths
  • contract length
  • target count
  • static wire-shape count

read_uleb128 rejects values that require more than 10 bytes.

Source Address

The canonical packet header does not carry the sender hostname. It carries src_addr, a compact source address assigned or learned through network configuration and discovery. Hostnames, board names, endpoint holders, and topology are discovery/config state rather than repeated packet-header data.

Standalone pack/unpack helpers maintain a host-side address cache so decoded Packet values can still expose sender() when the process has seen the corresponding sender name. If no mapping is known, the decoded sender is represented as @addr:<number>.

Compression

Payload compression is used only when it is actually smaller on the wire. The logical uncompressed payload length is still transmitted in the frame metadata, and decode validates the decompressed size against that logical length.

Decode flow

High-level decode order:

  1. Verify CRC32.
  2. Parse the fixed prelude and varints.
  3. Resolve endpoints from the fixed-width endpoint bitmap when flag 0x20 is set, otherwise from the default endpoint list in local data type metadata.
  4. Resolve the source address to a sender name from discovery/config state.
  5. Decode the optional wire contract.
  6. Decode the reliable header if current schema or contract says it is present.
  7. Decode the payload bytes.
  8. Construct Packet::new_with_wire_contract(...).

The optional wire contract provides the inline shape and target sender hashes used during unpacking and routing.

Envelope peek

peek_envelope(...) parses only the envelope and returns:

  • ty
  • endpoints
  • sender
  • source_address
  • timestamp_ms
  • wire_shape
  • target_senders

peek_frame_info(...) extends that with the reliable header when present.

Routers and relays use these helpers for routing and reliable-layer decisions without fully decoding payload data.

Packet ID from wire

packet_id_from_wire(...) computes the same ID as Packet::packet_id() from a packed frame. It hashes:

  • compact source address
  • message name
  • endpoint names in ascending bitmap order
  • timestamp
  • logical payload size
  • payload bytes after decompression

The wire contract is not part of the packet ID.

CRC32 trailer

Every frame ends with a 4-byte little-endian CRC32 computed over all preceding bytes.

On CRC failure the frame is rejected before normal decode. Recovery behavior after that depends on whether the surrounding router/relay reliable layer is active for that hop.

Common decode failures

  • short prelude or short read
  • CRC32 mismatch
  • invalid endpoint bit set
  • malformed or overlong ULEB128
  • bad wire-shape type/class/count
  • malformed wire-contract target list
  • reliable control frame passed into full packet decode
  • decompression failure or decompressed-size mismatch
  • sender not valid UTF-8

Clone this wiki locally