Skip to content

[rust] Check in generated protobuf code and drop protoc requirement - #3874

Open
seokjin0414 wants to merge 1 commit into
apache:mainfrom
seokjin0414:commit-generated-proto
Open

[rust] Check in generated protobuf code and drop protoc requirement#3874
seokjin0414 wants to merge 1 commit into
apache:mainfrom
seokjin0414:commit-generated-proto

Conversation

@seokjin0414

Copy link
Copy Markdown

Re-targeted from apache/fluss-rust#662 after the Rust client moved into this repo; the old PR and apache/fluss-rust#661 are closed with pointers here. Downstream context: apache/iggy#3688.

FlussApi.proto only changes when the wire protocol does, but fluss-rust/crates/fluss/build.rs reran prost-build on every build, which makes a system protoc a requirement for every consumer of the fluss-rs crate and for the Rust CI here. Vendoring the proto into published crates (scripts/vendor-proto.sh) ships the schema, but consumers still run the codegen, so they still need protoc.

This checks the generated code in instead, following the arrow-flight layout (committed prost output + a small gen crate + regen.sh):

  • crates/fluss/src/proto/fluss.rs: checked-in prost output, byte-identical to what build.rs produced (same prost-build version, same bytes config), with the ASF header and a do-not-edit notice prepended
  • crates/fluss/gen: a tiny bin crate that regenerates it. prost-build is a normal dependency there, so the workspace compiles without protoc; protoc is only needed when actually running crates/fluss/regen.sh. The proto is resolved the same way build.rs did it: the vendored proto/FlussApi.proto when present, the canonical fluss-rpc/src/main/proto/FlussApi.proto otherwise
  • build.rs and the [build-dependencies] on prost-build are removed; cargo package -p fluss-rs --list confirms fluss.rs ships in the crate and the nested gen crate is excluded automatically
  • a new proto-up-to-date job in rust-build-and-test.yml reruns the regeneration and fails on any diff, so a change to the canonical proto cannot land without regenerating the Rust side (the workflow already triggers on fluss-rpc/src/main/proto/**). The output is deterministic: prost-build is version-locked through Cargo.lock, and generation is byte-identical across protoc 27.1, 29.3 and 35.1
  • the Rust build/test and license/format workflows stop installing protoc otherwise, so CI itself now proves the protoc-free build
  • DEVELOPMENT.md: protobuf is now only a prerequisite for regenerating

Verified locally with no protoc installed: cargo build, cargo clippy --all-targets --workspace -- -D warnings, cargo test with the CI excludes, cargo fmt --all -- --check, and cargo doc --workspace --no-deps --exclude fluss_python, all from fluss-rust/. Rerunning regen.sh reproduces the committed file byte-for-byte.

Once this is in a release, the protoc requirement disappears for every fluss-rs consumer; the two Fluss connectors in flight for Apache Iggy (apache/iggy#3799, apache/iggy#3782) are currently blocked on exactly that.

FlussApi.proto only changes when the wire protocol does, but build.rs
reran prost-build on every build, forcing a system protoc onto every
consumer of the fluss-rs crate and onto the Rust CI. Check the generated
fluss.rs in and replace build.rs with a small gen crate plus regen.sh,
following the arrow-flight layout: prost-build is a normal dependency of
the gen tool, so building never invokes protoc, only regenerating does.
A proto-up-to-date CI job regenerates and fails on any diff, so canonical
proto changes cannot land with stale Rust code. The checked-in file is
byte-identical to the previous build.rs output.

Re-targeted from apache/fluss-rust#662 after the Rust client moved into
this repository.

Signed-off-by: seokjin0414 <sars21@hanmail.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant