Codec

How the codec works as built: one declaration per message, a source generator that writes each declaration's reader and writer, and a table of the client's own layouts that every declaration is checked against. The facts to look up are on Codec declarations, Codec layout tree and Codec facts check; every declared message is in the generated message reference, how each is checked for what it means is on Message audit design, and how the codec was built is on Codec design.

Note

This page and its references replace Codec design for what's built. The design page keeps the history of how the codec was built.

How it fits together

The shape

flowchart TD
    D["Declarations"] -->|"compiled by"| G["Source generator"]
    FT["Client facts table"] --> FC["Facts check"]
    G -->|"reader, writer, tree"| R["WireMessages registry"]
    R -->|"read and write"| GW["Gateway"]
    R -->|"layout trees"| FC
    R -->|"generated pages"| REF["Message reference"]

Loading

A message is declared once. Each message is a C# class whose fields carry their exact widths and encodings, and that declaration is the only description of the message in the tree. Wire order is declaration order, and no width or encoding is ever inferred from a C# type.

The generator writes everything else. From each declaration it writes a reader, a writer, a size, and the message's full layout as a tree. One registry per assembly, WireMessages, holds every message by opcode and direction. The gateway reads and writes messages only through those registries, and the message reference is generated from them. A declaration that breaks a rule gets no code at all, so the build fails at the declaration, never in generated code.

The client's own layouts check every declaration. The client facts table holds the full layout of every message the client registers, written as our own numbers. The facts check compares each declaration's tree with the client's, node by node, in public CI with no client data. A disagreement is pinned with its reason, and the list of pins can only shrink.

How far a message is verified is a separate check. A declaration records its rung, and the verification check, beside the facts check, holds every message to it. That's the message audit's half: the facts check proves a layout, the audit proves what it means.

Why it's built this way

The message models the project started from were written by hand, a read method and a write method per type, and the width of most fields was a default rather than a decision.

• Defaults hid the width. Every reader and writer primitive took an optional width defaulting to the C# type's size, and enums defaulted to 64 bits. Of about 3,600 primitive calls in that code, 2,175 left the width out.
• The client disagreed. A first check against the client's layouts found 70 models wrong, mostly enums at the default 64-bit width.
• Read and write drifted apart. 55 types both read and wrote, and 5 of them disagreed with themselves.

A field's width is a fact about the wire. Writing it down once, where the compiler can see it, removes that whole class of bug, and checking it against the client in CI keeps it removed.

What we chose not to do

Idea	What killed it
Keeping default widths on the primitives	Defaults are how 64-bit enums went unnoticed. The primitives have none
Generating declarations from the facts table	It holds layouts only: no names, no types, no meanings. It's the check, never the source
Inferring signedness or float encoding from the C# type	A float already travelled as an integer in one of the old models; the declaration says the encoding
A simple "columnar" flag for arrays written column by column	One message writes a scalar between its two columns, so arrays bind to a named count field instead
Fixing mismatches during the conversion	A conversion that also fixes can't be proven equivalent. Equivalence first, then one fix per cause

Vocabulary

Term	Means here
Declaration	The C# class that describes one message or nested layout, field by field
Layout tree	A layout written as one line of text, the same notation for a declaration and for the client
Empty path	The widths a layout puts on the wire with every array empty and every union at its first arm
Selector	The earlier field whose value picks a union's arm
Facts row	The client's layout for one opcode, in the client facts table
Pin	A disagreement with the client, recorded with its reason so the facts check passes until it's fixed
Rung	How far a message has been verified against the client, from its identity to its conversation

Knobs

The codec has no runtime settings. Two things are set at build time: <HotPath>true</HotPath> puts a project under the hot-path rules, as the codec's own project is (Code standard), and the client facts table is one file per client build.

Known edges

• 43 disagreements with the client's layouts are pinned, each settled as the message audit reads its area. Codec facts check has the count from the latest run.
• The facts check can't see a swap of two fields of equal width, signedness, or meaning; generated round trips and the audit cover those.
• Which client fields are signed is settled message by message, as the audit reads each one's fields.

Where the code lives

Piece	Path
The bit reader and writer, the attributes, the layout tree	Source/NexusTomorrow.Network.Codec/
The message and rung attributes	Source/NexusTomorrow.Network/Message/
The source generator	Source/NexusTomorrow.Network.Codec.Generator/
The client facts table	Source/NexusTomorrow.Network.Codec/Facts/
The message declarations	Source/NexusTomorrow.Network/, Source/NexusTomorrow.Network.Auth/, Source/NexusTomorrow.Network.World/
The facts check and the verification check	Tests/NexusTomorrow.Network.Codec.Tests/
The generator's tests	Tests/NexusTomorrow.Network.Codec.Generator.Tests/