feat(ipc-codegen): add standalone codegen package + wire-compat examples#23314
Open
charlielye wants to merge 11 commits into
Open
feat(ipc-codegen): add standalone codegen package + wire-compat examples#23314charlielye wants to merge 11 commits into
charlielye wants to merge 11 commits into
Conversation
Add top-level ipc-codegen/ containing: - A schema-driven multi-language code generator (TS/C++/Rust/Zig) - Committed JSON schemas for BB API, WSDB, CDB, AVM - A self-contained 4-language echo example harness under ipc-codegen/examples/ with cross-language wire-compatibility tests The codegen ships ungated by any existing consumer. Only the echo examples invoke it via ipc-codegen/bootstrap.sh. Existing bb.js codegen (barretenberg/ts/src/cbind/) is untouched in this PR. Generated outputs are .gitignored under barretenberg/.gitignore so producers can regenerate without committing artifacts. Echo example generated files remain tracked as a frozen reference for the codegen. Verify: - cd ipc-codegen && ./bootstrap.sh generate # regenerate echo bindings - cd ipc-codegen && ./bootstrap.sh test # 4x4 wire-compat matrix
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
4 tasks
…generated files
Restructure ipc-codegen/ to follow the same bootstrap.sh shape as other components:
- bootstrap.sh: rename generate -> build (matches barretenberg/ts pattern).
Add test_cmds emitting the cross-language wire-compat command. Add test
wrapping test_cmds | filter_test_cmds | parallelize.
- Makefile: add ipc-codegen and ipc-codegen-tests targets, include in fast
bootstrap (alongside claude-tests).
- Drop committed pure-generated files (echo_types.*, echo_*_ipc_*.*,
api_types.ts, async.ts, sync.ts, server.ts, ipc_*.{ts,rs,zig,hpp}); they
regenerate on every build.
- Keep committed scaffolding files (backend.{rs,zig}, error.rs,
uds_backend.{rs,zig}); they're one-time-copied templates hand-customized
per example (echo's EchoError vs the template's BarretenbergError).
- Refine ipc-codegen/.gitignore to distinguish: ignore pure generated,
exempt scaffolding.
Net: 23 files changed, +49 / -2492 LOC.
Verify:
- cd ipc-codegen && ./bootstrap.sh build # regenerate echo bindings
- cd ipc-codegen && ./bootstrap.sh test # 18/18 wire-compat pass
- make ipc-codegen-tests # via root Makefile
Move build-once steps (cargo build, zig build, npm install, C++ compile, golden generation) from the test runner into bootstrap.sh build. Rename run_cross_language_tests.sh -> run_cross_language_test.sh (singular) and have it accept golden <lang> or matrix <server-lang> <client-lang>. Emit one command per pair in test_cmds so each test reports PASSED / FAILED independently in the parallelize framework. Verify: - ./bootstrap.sh build # codegen + per-language compilation + goldens - ./bootstrap.sh test # 18 tests, each reported individually
…y echo example
The generated <svc>_ipc_client.cpp and <svc>_ipc_server.hpp used to #include
"barretenberg/serialize/msgpack.hpp" and msgpack_impl.hpp, forcing any
consumer to be inside the barretenberg include path. The echo example
worked around this by hand-rolling its own msgpack pack/unpack loop
instead of using the generated EchoIpcClient.
Bundle the msgpack adaptor (struct_map_impl + concepts + drop_keys, all
inlined into one ~80-line header) under ipc-codegen/templates/cpp/. The
C++ codegen now emits #include "msgpack_struct_map_impl.hpp" and the
codegen copies the template alongside the generated client/server.
With that done, rewrite examples/cpp/echo/echo_client.cpp to actually use
the generated client:
echo::EchoIpcClient client(socket_path);
auto resp = client.fields({.a = 42, .b = 999, .name = "hello"});
instead of the previous 90 lines of manual pack/unpack. echo_server.cpp
similarly drops its barretenberg struct_map_impl.hpp include — the
generated server header now brings the adaptor in itself.
Also fixes a generatedInclude() bug where an empty --cpp-include-dir
produced "/foo.hpp" instead of "foo.hpp".
Goldens now cover the msgpack encoding boundaries that codegen tweaks are
most likely to silently break:
- Variable-width integer encodings (fixint / uint8 / uint16 / uint32 /
uint64) including u32::MAX and u64::MAX
- String encodings (fixstr / str8 / str16) plus multi-byte UTF-8
- Bin encodings (bin8 / bin16) including bin-len-0
- Optional<T> = Some(true), Some(false), and None
- Empty containers (Vec<u8>::new(), vec![Vec<u8>::new()], etc.)
Each language's golden test now both (a) decodes the bytes into the
expected typed value AND (b) re-encodes and asserts byte-identical
output, which pins down canonical msgpack output.
Goldens are committed and frozen. Run ./bootstrap.sh update_goldens
when intentionally changing the wire format, and review the binary diff
as a breaking-change signal for external implementations.
While here:
- Set msgpackr Encoder to variableMapSize: true everywhere it's used
(test, ipc_client.ts/ipc_server.ts templates, typescript_codegen.ts)
so it emits canonical fixmap for small maps instead of always reaching
for map16.
- One narrow exemption: msgpackr encodes positive BigInt as int64 (`d3`)
while rmp-serde uses uint64 (`cf`). Both encodings decode to the same
value across every msgpack lib, so wire interop is preserved.
echo_fields_uint_boundary opts out of strict roundtrip with a comment
explaining why.
…e BarretenbergError -> IpcError Two cleanups in one pass: 1) The "scaffolding" template files (backend.rs/error.rs/uds_backend.rs and the zig equivalents) were committed inside examples/*/echo/generated/ alongside the rest of the codegen output. That was a mistake — they are copied once from ipc-codegen/templates/ on a fresh build, so they belong in templates only and the generated/ dir should be 100% build- produced. Drop the committed copies and make the gitignore a blanket **/generated/ rule. 2) The templates' error type was named BarretenbergError, which leaked the codegen package's barretenberg origin into every generated consumer (Rust, Zig). Rename to IpcError so the templates are genuinely framework-agnostic; update the rust echo example consumer code (echo_server.rs) accordingly. After this, examples/*/echo/generated/ is reproducible from schema + templates with zero committed files, which is what one would expect of a directory named "generated".
…README Move ipc-codegen/src/README.md and ipc-codegen/src/SCHEMA_SPEC.md up to ipc-codegen/ root where they belong. Rewrite the README from scratch: - Quick start (bootstrap.sh build / test). - Package layout (schemas/, src/, templates/, examples/, scripts/). - Full src/generate.ts CLI reference: every flag, what it does, which language it applies to. - Worked examples for each of the four targets (TS bb client+server with curve constants; C++ wsdb under a barretenberg include path; Rust wsdb with UDS+FFI and skeleton scaffolding; Zig avm). - "Adding a new service" workflow. - Schema-of-truth + scripts/update_schemas.sh + validate_schemas.sh explanation. - Pointer to the frozen-goldens wire-format contract. Fix the SCHEMA_SPEC.md path references that still pointed at the old barretenberg/ts/src/cbind/ location.
Two issues in one cleanup:
1) examples/{rust,zig}/wsdb/ were skeleton scaffolds where every handler
returned "not implemented". They weren't part of the cross-language
test matrix and they duplicated what the README's "Worked examples"
section already documents. The echo example fully exercises codegen +
roundtrip across all four languages and is what reviewers should look
at. Delete the scaffolds.
2) examples/zig/echo/build.zig.zon pinned the zig-msgpack tarball URL to
heads/main, which drifts as the upstream branch moves. CI fetched a
newer snapshot than the committed hash and zig-fetch refused with a
hash-mismatch error, breaking the whole `make ipc-codegen` target
(and every dependent test job).
Switch the URL to `tags/0.0.17` (the latest tagged release, which is
also what CI was implicitly fetching from main) and use that release's
canonical hash. The tagged URL produces a stable hash; this won't
drift again.
Verify: cd ipc-codegen && ./bootstrap.sh test → 18/18 pass.
…solate The previous flow had `npx tsx ts/echo/echo_server.ts` for TS-side test binaries and waited only 2 seconds for the server's UDS socket to appear. That worked locally but routinely failed under CI's docker_isolate with CPUS=1, where `npx tsx` cold start took longer than 2s — every ts-as-server matrix pair came back as "server did not create socket within 2s". Two narrow fixes: - Add tsx as a real dependency in examples/ts/echo/package.json so it lands in node_modules/.bin after `npm install` (no more npx-resolve hop). - Replace `npx tsx ...` in run_cross_language_test.sh with `ts/echo/node_modules/.bin/tsx ...` and raise the wait-for-socket cap from 2s to 30s. Native binaries (rust/c++/zig) still bind in <100ms, so the longer ceiling only stretches worst-case TS startup. Verified: 18/18 still pass locally, now in 0s per test rather than 1-2s.
Make ipc-codegen a self-contained codegen package with zero knowledge of
its consumers:
- Delete schemas/{bb,wsdb,cdb,avm}_schema.json + bb_curve_constants.json.
Each consumer commits its own schema next to the C++ server that defines
the wire format, and passes the path on the command line.
- Delete scripts/{update,validate}_schemas.sh. Schema refresh + drift
guarding is each consumer's responsibility now; the snapshot tooling
belongs near the producing binary, not in the generator.
- README + SCHEMA_SPEC: rewrite consumer-facing sections with neutral
paths; drop bb/wsdb-specific layout and "yarn generate" instructions.
- .rebuild_patterns: drop schemas/ pattern; pick up examples/ (echo
schema + goldens live there).
- generated TS error message: "Unknown error from barretenberg" ->
"Unknown error from server".
- Bundled cpp msgpack adaptor + generator banners: drop barretenberg
attribution / "no barretenberg deps" wording from emitted output.
Also fold in the clang-format-templates-at-destination fix (originally
on PR2's branch) so PR1 ships a complete ipc-codegen and PR2 stays
focused on consumer migrations.
The echo example matrix (18 cross-language tests) still passes — the
package's own self-test is unaffected.
Previously the curve constants JSON was looked up relative to the generator's source directory (../schemas/bb_curve_constants.json), which baked a path to a committed-in-ipc-codegen file into the tool. Move ownership of the file to the consumer (alongside its other committed schema), and pass the path on the command line: generate.ts ... --curve-constants /path/to/curve_constants.json The flag now requires a path argument instead of being a bare boolean. No callers within this PR use it; the bb.js consumer that does will update in PR2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new top-level
ipc-codegen/package: a schema-driven multi-language IPC code generator producing matching C++, TypeScript, Rust, and Zig clients and servers from a single committed JSON schema. Ships with a self-contained 4-language echo example whose wire-compat test matrix runs inmake ipc-codegen-tests. Existing consumers are untouched — this PR lands the package alongside the legacy bb.jscbind/codegen; follow-up PRs migrate the production services (bb / WSDB / CDB / AVM) onto it.How to use it
ipc-codegen/src/generate.tsis the CLI. One invocation per (schema, language) pair.Required flags:
--schema,--lang,--out.Role flags (combine as needed):
--server,--client,--uds,--ffi.Naming:
--prefix Foo(type prefix; auto-detected if omitted);--strip-method-prefix(TS-only, strips prefix from client method names).C++-specific:
--cpp-namespace ns::sub,--cpp-wire-namespace wire,--cpp-include-dir path/to/generated.Other:
--curve-constants(TS, bb-only);--skeleton <dir>(one-time handler stubs +main+ build file for new services).Example invocations (full reference in
ipc-codegen/README.md)What ships
ipc-codegen/src/— generator (TS, runs under Node 22+, zero npm deps)ipc-codegen/templates/{cpp,ts,rust,zig}/— static transport + msgpack-adaptor templates copied alongside generated code, so generated artifacts are framework-agnostic (nobarretenberg/includes leak through)ipc-codegen/schemas/{avm,bb,cdb,wsdb}_schema.json— committed schemas of recordipc-codegen/scripts/{update,validate}_schemas.sh— refresh-from-binary + CI guardipc-codegen/examples/— 4-language echo example (cpp/ts/rust/zig clients & servers) used by the test harnessipc-codegen/examples/echo-schema/golden/— frozen msgpack fixtures covering every relevant encoding boundary (variable-width ints, fixstr/str8/str16, bin8/bin16, optional Some/None, empty containers, multi-byte UTF-8). The per-language golden tests both decode and re-encode to assert canonical-byte stability across implementations.ipc-codegen-teststarget wired intomake fast**/generated/) are gitignored — regenerated each build from schema + templatesLocal verification
All 18 pass.
Follow-up
This PR lands the package; nothing in production consumes it yet. Subsequent PRs in this stack migrate the existing services off the legacy
barretenberg/ts/src/cbind/codegen and ontoipc-codegen:barretenberg/ts/src/cbind/once every consumer is migratedAfter that point external contributors can implement a replacement for any service in their language of choice —
schemas/+examples/echo-schema/golden/*.msgpackdefine a complete wire-format contract, and the cross-language matrix test rig is reusable for verifying interop.