serialize.cs

Status: DRAFT — not yet released.

C# port of the C++ serialize bitpacking library. Produces bit-for-bit identical output to the C++ library (and to the Go and Rust ports), so streams written in one language can be read by any other. Wire compatibility is proven, not asserted: a golden wire test pins 72 bytes copied verbatim from the C++ test suite, and a live interop harness cross-checks against the real serialize.h compiled with clang++.

Family values: zero third-party dependencies (including test frameworks); malicious packet data never throws — reads fail cleanly with a sticky latched error (exceptions are reserved for API misuse); no unsafe code; zero allocation on serialization paths (strings on the read path are the documented exception).

Layout

• src/Serialize.cs — the whole library, one file (mirrors the C++ single header): BitWriter, BitReader, IBitStream, WriteStream, ReadStream, MeasureStream, SerializeUtil, ISerializer, plus the C#-only batch layer WriteBatch / ReadBatch (see below).
• tests/ — console test runner, no test framework: prints each test name, exit code is the verdict. Includes the golden wire test, an extended wire test pinning the 64-bit paths, and deterministic differential/hostile seeded tests.
• compat/ — the cross-language interop harness: Compat.csproj (C# half) and cpp/compat.cpp (C++ half, built against the real serialize.h).
• scripts/interop.sh — the interop gate as one runnable command.
• STANDARD.md — the wire format spec, vendored verbatim from the C++ repo (family precedent: CI should diff it against upstream and fail on drift).

Build and test

dotnet build src/Serialize.csproj                          # builds net8.0 + net10.0
dotnet run --project tests/Tests.csproj -f net10.0         # add "short" to skip the 320 MB test
dotnet run --project tests/Tests.csproj -f net8.0          # the LTS leg (needs the .NET 8
                                                           # runtime, or DOTNET_ROLL_FORWARD=LatestMajor)
dotnet run --project tests/Tests.csproj -f net10.0 -- golden   # run only tests matching a substring

The library targets net8.0 (LTS game servers) and net10.0. A netstandard2.1 target for Unity-class runtimes is an open deliverable: it needs shims for BitOperations, the unsigned BitConverter bit casts, Rune enumeration and Utf8.IsValid, plus an emulated 128 bit pair standing in for Int128/UInt128 (mirroring the C++ emulated types' two's complement semantics), each proven wire-neutral by the golden test per TFM before it ships.

Batches: the hot path for tiny messages

The streams are heap objects, so even with the serialize methods inlined the JIT reloads and stores the packer state (scratch, scratch bits, bits written) around every call — heap fields cannot live in registers across calls, and on tiny messages that traffic dominates. A batch lifts the state into the fields of a ref struct at BeginBatch, serializes against locals with the same wire logic, the same validation and the same latched error model — byte-for-byte identical output, proven by a batch golden-wire test and randomized differential tests — and stores the state back once at End:

WriteBatch batch = stream.BeginBatch();   // ReadStream: ReadBatch
batch.SerializeBits(ref value, 8);
// ... the same serialize surface as the stream ...
batch.End();                              // always, on every path out

The contract is small: the batch owns the stream between BeginBatch and End (stream calls or Reset while a batch is open are API misuse); always call End on every path out — it is idempotent, and it is what publishes the batch's work back to the stream. Fixed-size scalar operations up to 64 bits run register-resident; everything else — bulk and variable-size operations (SerializeBytes, strings, objects, SerializeIntRelative) and the 128 bit and fixed point operations (SerializeInt128, SerializeUInt128, SerializeFixed) — delegates to the class path and recaptures, byte identical. Batches are additive: code that never begins one behaves exactly as before. IBitStream-based unified serialize functions are unchanged — batches are for per-direction hot paths (e.g. generated code) where tiny-message throughput matters.

Two measured rules (Apple M2, schema harness): pass a batch by ref only to helpers marked AggressiveInlining — a real call taking ref WriteBatch address-exposes the struct and kills enregistration for the whole scope, measured slower than no batch at all; and batch scalar-dense bodies only — a body dominated by one bulk op (length int + SerializeBytes) pays the batch capture/restore without winning it back.

Interop gate (head-to-head vs C++)

scripts/interop.sh path/to/serialize    # or run the steps below by hand

clang++ -O2 -std=c++17 -ffp-contract=off -Wall -I path/to/serialize -o compat-cpp compat/cpp/compat.cpp
dotnet run --project compat/Compat.csproj -- write cs.bin
./compat-cpp write cpp.bin
cmp cs.bin cpp.bin                                  # must be byte identical
dotnet run --project compat/Compat.csproj -- read cpp.bin
./compat-cpp read cs.bin

-ffp-contract=off on the C++ build is required, not optional: strict IEEE evaluation is the normative wire for compressed floats. Default clang/gcc on ARM64 contract the quantization normalized * maxInteger + 0.5f into a fused multiply-add, which rounds differently within 1 ULP of a .5 boundary and shifts the written integer by one wire quantum (~1 in 10^7 values). C# and Rust always evaluate strictly; the compat sequence carries a value pinned on such a boundary (0.005f in [0,10] res 0.01), so a contracted C++ build fails the cmp instead of passing silently. (The Go port on ARM64 currently fuses and should be flagged upstream.)

When CI is created, pin the C++ clone to one release tag — the Go and Rust ports pin v1.4.3; the sibling clone here is currently at a later head — and pick one tag for both the interop job and the spec-sync job.

Fixed point and 128 bit integers

The fixed point + 128 bit additions to the C++ library (its fixed-point branch) are ported in full, on the native Int128/UInt128 — both TFMs are net7.0+, so the C++ native/emulated distinction does not exist here; there is exactly one representation:

• SerializeUInt128 — raw, always 128 bits on the wire: the low 64 bit half first, then the high half. When the stream is byte aligned the result is the 16 bytes of the value in little endian order.
• SerializeInt128 — the ranged counterpart of SerializeInt64: SerializeUtil.BitsRequired128(min,max) bits, computed and offset encoded in the unsigned domain (ranges wider than 2^127 are exact), written in 32 bit groups from least significant upward. Where the range fits 64 bits or fewer the bytes are identical to SerializeInt64 over the same bounds, so a field can be widened from 64 to 128 bits without a wire change.
• SerializeFixed — Q format fixed point, one overload per integer storage type from 16 to 128 bits, signed and unsigned. The whole unit bounds are shifted to raw integer-exact bounds and the raw value is offset encoded in the minimal bits for the range; the codec touches no floats, so unlike SerializeCompressedFloat the round trip is exact and identical on every platform. For storage of 64 bits or fewer the wire is byte identical to SerializeInt64 of the raw value over the raw bounds, and fractionBits = 0 is a ranged integer.

Offsets smuggled into the bit headroom past the top of a range are rejected on read, never clamped. The Q format and bounds are trusted parameters validated as API misuse (see below), like every other range in the library.

The wire pins for all three operations were derived from STANDARD.md's text by an independent oracle and cross-checked byte for byte against the pins in the C++ fixed-point test suite; test_fixed_wire_format pins the byte aligned fixed point tail of the C++ 112 byte golden message, leaving the original 72 byte golden pin untouched.

Reading untrusted data

Errors are sticky: the first failure latches on the stream and later serialize calls are no-ops that leave values unmodified. One rule follows: a value that controls a loop must have its result checked before the loop uses it, otherwise a truncated or malicious packet spins the loop forever. Use stream.Continue(ref more) / stream.Until(ref done) for sentinel-driven loops, and check the result of any serialized loop count before looping — on a reused stream a failed read leaves the previous packet's count in place.

Two further rules:

• Ranges are trusted inputs. min, max, resolution, bufferSize and fixed point Q format (integerBits, fractionBits, whole unit bounds) parameters are validated as API misuse and throw ArgumentException — even on a stream with a latched error. If you compute a range from previously decoded packet data, validate it before passing it in, or one malicious packet becomes an unhandled exception.
• Compressed float ranges must have a finite difference. When max - min overflows to infinity (e.g. [-3.4e38, +3.4e38]), the read can decode NaN or infinity and still report success — behavior inherited from the C++ library for wire fidelity. Choose ranges whose difference is finite.

License

AGPL-3.0. See LICENSE.