[rust] Check in generated protobuf code and drop protoc requirement - #3874
Open
seokjin0414 wants to merge 1 commit into
Open
[rust] Check in generated protobuf code and drop protoc requirement#3874seokjin0414 wants to merge 1 commit into
seokjin0414 wants to merge 1 commit into
Conversation
FlussApi.proto only changes when the wire protocol does, but build.rs reran prost-build on every build, forcing a system protoc onto every consumer of the fluss-rs crate and onto the Rust CI. Check the generated fluss.rs in and replace build.rs with a small gen crate plus regen.sh, following the arrow-flight layout: prost-build is a normal dependency of the gen tool, so building never invokes protoc, only regenerating does. A proto-up-to-date CI job regenerates and fails on any diff, so canonical proto changes cannot land with stale Rust code. The checked-in file is byte-identical to the previous build.rs output. Re-targeted from apache/fluss-rust#662 after the Rust client moved into this repository. Signed-off-by: seokjin0414 <sars21@hanmail.net>
This was referenced Aug 5, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-targeted from apache/fluss-rust#662 after the Rust client moved into this repo; the old PR and apache/fluss-rust#661 are closed with pointers here. Downstream context: apache/iggy#3688.
FlussApi.protoonly changes when the wire protocol does, butfluss-rust/crates/fluss/build.rsreran prost-build on every build, which makes a system protoc a requirement for every consumer of thefluss-rscrate and for the Rust CI here. Vendoring the proto into published crates (scripts/vendor-proto.sh) ships the schema, but consumers still run the codegen, so they still need protoc.This checks the generated code in instead, following the arrow-flight layout (committed prost output + a small
gencrate +regen.sh):crates/fluss/src/proto/fluss.rs: checked-in prost output, byte-identical to whatbuild.rsproduced (same prost-build version, samebytesconfig), with the ASF header and a do-not-edit notice prependedcrates/fluss/gen: a tiny bin crate that regenerates it. prost-build is a normal dependency there, so the workspace compiles without protoc; protoc is only needed when actually runningcrates/fluss/regen.sh. The proto is resolved the same waybuild.rsdid it: the vendoredproto/FlussApi.protowhen present, the canonicalfluss-rpc/src/main/proto/FlussApi.protootherwisebuild.rsand the[build-dependencies]on prost-build are removed;cargo package -p fluss-rs --listconfirmsfluss.rsships in the crate and the nestedgencrate is excluded automaticallyproto-up-to-datejob inrust-build-and-test.ymlreruns the regeneration and fails on any diff, so a change to the canonical proto cannot land without regenerating the Rust side (the workflow already triggers onfluss-rpc/src/main/proto/**). The output is deterministic: prost-build is version-locked through Cargo.lock, and generation is byte-identical across protoc 27.1, 29.3 and 35.1Verified locally with no protoc installed:
cargo build,cargo clippy --all-targets --workspace -- -D warnings,cargo testwith the CI excludes,cargo fmt --all -- --check, andcargo doc --workspace --no-deps --exclude fluss_python, all fromfluss-rust/. Rerunningregen.shreproduces the committed file byte-for-byte.Once this is in a release, the protoc requirement disappears for every
fluss-rsconsumer; the two Fluss connectors in flight for Apache Iggy (apache/iggy#3799, apache/iggy#3782) are currently blocked on exactly that.