Skip to content

Releases: mas-bandwidth/schema

v2.5.0 — every protocol id moves; tables in nine languages; write checks are debug-only

Choose a tag to compare

@gafferongames gafferongames released this 07 Sep 19:49

What you get

  • Tables in all nine languages. C, C#, Go, Rust, Java, JavaScript, Dart and Elixir join C++: every language reads and writes fixed-class tables (with ?T optional fields and [E]T enum-keyed arrays) on the tolerant wire, the JSON text form and the reflection descriptors. Block and cook readers are in every language; the block builder is in C++ and C, and the cook's write side is in C++.
  • The tables language grew, in the C++ reference. Arrays of pointers and of unions, optional arrays, unbounded []T, map[K]V, *bytes and *string blobs, string/bytes/flags defaults, was on tables, enum variants and union arms, and a union arm that takes any field type. ROADMAP.md says what each other language carries today.
  • A leaner table wire, in C++. Files use the id-table form: each 64-bit name hash appears once, in a table at the end, so a file naming forty ids across ten thousand fields carries forty ids. A message form for tables over a connection, with batching and a bitpacked body. Unknown fields can be retained through a read, values widen on read, and every refusal has a name. The other eight languages write the previous form today and move to this one in a later release.
  • The cook and the block form. schema cook, cook-check and uncook are complete to the byte; C++ cooks from code as well as from the tool; every fixed table has a block form on the side at zero cost to the ordinary headers.
  • Packet wire, in all nine languages. wstring(N) wide strings. Malformed UTF-8 is refused on read in every build mode. string, bytes and flags fields take declared defaults. A union arm with no payload is encoded as its tag alone. Every enum exports Count beside Max.
  • Write-side checks are debug-only. In C, C++, C#, Rust, Java, Dart and JavaScript's flat tier, a release build no longer checks what you write; a debug build asserts it at the site. Go and Elixir keep refusing in every build, because those languages have no debug-only idiom, and JavaScript's stream tier still refuses in every build for now. Read-side checks run in every build, everywhere.
  • Faster in release. With the write-side checks compiled out, Rust's packet write is 11.6% faster and C#'s 9.2%, and C's read path closed to a statistical tie with C++. The README's table was measured on a new machine, so its percentages do not compare with 2.4.0's; PERFORMANCE.md says what moved.
  • Docs. TUTORIAL.md takes you from an empty directory to every feature. VERSIONING.md states the commitment for both wires. PORTING.md, COMPETITION.md, COMPARISON-TABLES.md and ROADMAP.md are new. Every page but README.md and ROADMAP.md now lives under docs/.
  • Tooling. Only schema fmt writes a schema file; every other command leaves your files alone. schema pack and schema unpack turn a directory tree into a table file and back. schema tables-baseline commits a projection that schema check diffs against, so an edit that changes what already-written data means cannot slip through unannounced.

Compatibility

Every protocol id moves. Rebuild and redeploy both peers together: a 2.4.0 peer refuses a 2.5.0 one rather than misreading it. Table files written under 2.4.0 do not open under 2.5.0; the table wire is pre-release until 3.0.0.

A schema stops compiling if a field's range excludes zero and it declares no default (an array cannot take a default, so widen its range), or if an enum has a variant named Count. [E.Max]T in a table body is refused; spell it [E]T. wstring(N) inside a table generates for C++ only.

v2.4.0 — faster generated Dart and JavaScript

Choose a tag to compare

@gafferongames gafferongames released this 01 Sep 09:44
b63c22e

What you get

  • Generated Dart is 1.6x faster. Chunked write merges, chunked read windows, grouped element loops, and a bulk path for byte content. Regenerate and it's yours — no API change, no wire change.
  • Generated JavaScript is 1.5x faster (writes 1.9x). Wide values ride an allocation-free BigInt lane through scratch stores, small fields pack into single-word merges, and string/bytes bodies move four bytes at a time.
  • The performance table (README) reflects both, from one committed sweep: Dart at 225% and JavaScript at 318% of generated C++, alongside Rust 153%, Java 156%, Go 209%, C# 224%.
  • The batch planner counts honestly in C#: fixed-point, 128-bit, and mixed-arm union bodies no longer receive batch entries their delegation costs don't earn.
  • An absolute performance ledger guards the reference: every committed sweep joins a per-language time series (./bench/tools ledger), and CI fails if the C++ curve ever regresses beyond its noise gate — relative numbers can shuffle; the absolute curves cannot lie.

Wire bit-identical across all nine languages throughout, held by the golden corpus and differential review.

v2.3.0 — tables: declare your own file formats

Choose a tag to compare

@gafferongames gafferongames released this 01 Sep 02:29
1dba683

What you get

  • Tables — declare your own file formats. A table declaration is a data record for content that outlives builds: config files, asset archives, tool output. Declare a root table, nest tables by value, and the generated C++ saves and loads it — no envelope imposed, no document concept, no prescribed structure. Your root table IS your format.
  • Version tolerance built into the wire. Add, remove, and reorder fields freely; old readers skip what they cannot name, new readers default what old writers never wrote, changed kinds are skipped and counted, out-of-range values clamp and count — nothing crashes on data from a different build, in either direction, and every event lands in a read report. Renames keep identity with was = "old_name", and id collisions are refused at compile time.
  • Reflection for tools. Generated descriptors carry every field's name, kind, offset, bounds and nesting — walk properties by name, print, diff, or bind an editor with no RTTI and no schema files at runtime.
  • Relocatable, and built wide. Generated table structs are trivially copyable (statically asserted); encode splits into measure and save over caller-provided buffers, and nested tables are length-prefixed — so builds can measure in parallel, prefix-sum, and scatter-write from many threads, and saves into exactly-measured buffers succeed byte for byte.
  • Self-contained headers. Generated <Unit>Table.h has no dependency on the serialize runtime — includable from any TU. C++ today; every other backend refuses table units by name.
  • Dart fixes. Fixed-point field defaults are no longer double-scaled, and declarations named after dart:core types (List, String, …) are refused at generation instead of emitting shadowing classes.

SPEC-TABLES.md specifies the full wire — neutral little-endian TLV a third party can implement from the document alone. The packet wire (type, the protocol id) is untouched by all of this: tables and packets version independently by construction.

v2.2.1 — generated C++ string helpers are guarded per package

Choose a tag to compare

@gafferongames gafferongames released this 01 Sep 01:08
474cbff

What you get

  • Two generated C++ units in one translation unit now compile. The emitted string helpers are guarded per package, so a TU that includes wire headers from more than one schema package keeps every package's helpers. Regenerate your C++ and the fix is yours; the wire is unchanged. A two-package regression test now holds this in CI.

Generated C is unaffected — its file-scope helpers were always correct.

v2.2.0 — faster generated code in Rust, Go, C# and Elixir

Choose a tag to compare

@gafferongames gafferongames released this 01 Sep 00:53
e88bcd7

v2.2.0 — faster generated code in Rust, Go, C# and Elixir

What you get

  • Faster serialization in four languages. Generated Rust, Go and C# use a flat word codec: field values pack into 64-bit words at compile-time-known shifts, one fused bounds check per chunk. Generated C# composes unions through batch cores with scoped batches. Generated Elixir builds one binary per barrier, packing group bits into fixnum-safe words. Regenerate your types and the speedup is yours — no API change, no wire change.
  • Go pairs with serialize.go v1.14.0, which halves the work in the per-chunk bit path. CI certifies the pair.
  • A benchmark you can run. ./bench/run.sh measures the real generated code in every language over the same wire corpus — a representative 438-byte game packet with integers carrying 92% of the bytes — and ./bench/run.sh --render <csv> prints the table from any sweep. Raw sweeps are committed under bench/results/.
  • make builds the compiler and nothing else. A fresh clone needs only Go; the nine-language conformance chain is make test.
  • The same bits everywhere. The wire is bit-identical across all nine languages, held by the cross-language golden corpus — including a degenerate-construct suite (zero-width fields, empty unions, boundary widths).

Per-language performance

Cost to serialize the benchmark packet, relative to generated C++ at 100% — lower is faster:

Language %
C 100%
C++ 100%
Java 148%
Rust 153%
Go 208%
C# 221%
Dart 363%
JavaScript 461%
Elixir 1348%

C and C++ report as a statistical tie: the pair sits inside the sweep's noise band, so neither is called a winner.

Nine Languages

Choose a tag to compare

@rowan-claude rowan-claude released this 30 Aug 22:17
d9fd170

Write your data types once and generate the code that reads and writes them — bit-identical — in C, C++, C#, Go, Rust, JavaScript, Dart, Java and Elixir.

  • Three new backends: Dart, Java and Elixir. Each emits the same straight-line serializer the other targets do — monomorphic, constant-width, no reflection and no schema walking at runtime.
  • Nine-language conformance runs in CI on shared corpora: every target reads and writes the same bits, proven on every commit.
  • One benchmark contract covers all nine languages, and bench/run.sh --quick runs the mixed shape under a minute per language, reporting nanoseconds per message and percent of fastest. The full interleaved standard remains the certification form.
  • Dynamically sized unions read correctly in the JavaScript flat tier.
  • String reads carry no validation cost: well-formed UTF-8 is the writer's obligation under SPEC 4.7, and the reader trusts it.

The wire did not move. No protocol id changed in this release: a schema built with v2.0.0 and with v2.1.0 produces the same bytes and the same id, so deployed peers interoperate across the upgrade.

Production Ready

Choose a tag to compare

@rowan-claude rowan-claude released this 25 Aug 23:58

Write your data types once and generate the code that reads and writes them — bit-identical — in C, C++, C#, Go, Rust and JavaScript.

  • Types, enums, flags, unions, constants and bitpacking: the complete language.
  • One declaration produces the reader and the writer in every language, so they cannot disagree.
  • The generated code is the straight-line serializer you would have hand-written — no reflection, no schema walking at runtime.
  • Bounded arrays and integer ranges are validated wire facts: out-of-range input is rejected at read time, and every value costs only the bits its declared range needs.
  • Six-language conformance runs in CI on shared corpora: every target reads and writes the same bits, proven on every commit.
  • On a representative gameplay packet: 28 bytes, against Cap'n Proto's 52, Protobuf's 56 and FlatBuffers' 72 — measured by running the real encoders, reproducible from COMPARISON.md.

The Go module path is github.com/mas-bandwidth/schema/v2.

v1.16.0 — success is silent, --verbose lists the files

Choose a tag to compare

@gafferongames gafferongames released this 23 Aug 22:12
77f3247

The CLI is quiet by default: a successful generate, pack, check or fmt run prints nothing, so a build that wraps the compiler stays clean without any wrapper-side filtering. The new --verbose flag on those four commands restores the per-file report — the files generate and pack wrote, the files the formatter rewrote, check's ok line with the package and protocol id.

Commands whose printed output is their answer — id, projection, version — print it exactly as before. Errors and diagnostics always reach stderr, and exit codes do not depend on verbosity.

The compiler library API is unchanged: verbosity is CLI policy, held in cmd/schema where CLI policy lives. The publicapi gate now pins the shape from both sides of the module boundary — silent defaults, and an identical --verbose report from the in-repo and external builds.

v1.15.0 — every emitter generates the precomputed compressed float path

Choose a tag to compare

@rowan-claude rowan-claude released this 23 Aug 14:21

Generated code now uses the runtime's precomputed compressed float entry point in every emitter — C++, C, C#, Rust and Go — so a declaration's wire parameters are computed once rather than re-derived per call, and the emitters share one arithmetic home with the runtimes they target (issue #82).

The C interop and Rust interop pins move to the releases carrying those entry points and the normative integer clamp: SERIALIZE_C_TAG to v1.5.0 and SERIALIZE_RS_TAG to v2.2.0.

Also in this release:

  • A bench row names its generated unit, so real_packet gets a verdict in all five compiled languages (issue #104).
  • The bench standard gains string and wstring row definitions, with the batch mix rebalanced to at most 15% bulk (issue #64).
  • The four non-C++ Objects units the advisory-round note missed are regenerated, and that note now says it is advisory, because the generated code does not honor it.
  • The TU-boundary attribution is stated as history rather than in the present tense across four documents.
  • Documentation describes what this library is, not speculative future work.

schema v1.14.0 — the extremes close out, and real_packet gets its instrument

Choose a tag to compare

@rowan-claude rowan-claude released this 20 Aug 14:31
8b29435

No wire change in this release: for any user schema the protocol id and every byte on the wire are identical to v1.13.0. Three merges: the integer extremes close out across the symbolic targets, and the bench suite's largest blind spot gets its instrument.

The extremes corpus rides in every target

A doubled unary minus over an INT64_MIN constant no longer emits rust that rustc rejects as compile-time overflow — the rust emitter gains the house arbitrary-precision carrier gate and folds the unprovable intermediate, matching the C and C# pattern (#101, closes #99). C++ member initializers for uint64 defaults above INT64_MAX now carry their ull suffix through the same guard emitConst always had (#102, closes #100). Both shapes live in the shared corpus with all six targets green on them.

real_packet rows for go, rust and cs

The compressed-float-dense workload is now measured on the three backends where compressed-float cost matters most, driving the same pinned corpus as the C++ reference with a byte-identity check as each row's setup — a corrupted corpus emits zero rows rather than wrong ones. All six runners now stamp the same corpus id, so cross-language ratios on these legs stop being refused. Found and fixed en route: the generated C# bench unit had never compiled (ufixed fields below 16 bits met a runtime whose narrowest overloads are 16-bit); the C# emitter now widens through a temp per the established golang pattern, wire bits unchanged, and the unit gained a compile gate so silence of that kind cannot recur. (#103, closes #80 and #65)