Skip to content

HDDS-5410. Improve protobuf compilation#9081

Merged
adoroszlai merged 7 commits intoapache:masterfrom
adoroszlai:HDDS-5410
Oct 6, 2025
Merged

HDDS-5410. Improve protobuf compilation#9081
adoroszlai merged 7 commits intoapache:masterfrom
adoroszlai:HDDS-5410

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented Sep 29, 2025

What changes were proposed in this pull request?

Several improvements in the build steps related to protobuf compilation.

  1. Protobuf versioning:
    • Add new properties for the version of shaded dependencies coming from ratis-thirdparty.
    • Rename proto3.hadooprpc.protobuf.version to hadoop-thirdparty.protobuf.version. Add a corresponding property for the artifact name, because it includes the version number.
    • Apply these consistently when compiling proto files for Ratis and Hadoop, respectively (do not mix with grpc.protobuf-compile.version).
    • Add notes about having to sync these versions with the thirdparty components.
    • Rename grpc.protobuf-compile.version to protobuf3.version, only used for CSI after this change, for now. Bump it to 3.25.8.
    • Rename proto2.hadooprpc.protobuf.version to protobuf2.version, since it is not specific to Hadoop RPC.
    • Combine all protobuf version items in the *-version-info.properties files into a single property. Previously each of these properties required an accessor in VersionInfo, only to be concatenated in HddsVersionInfo and OzoneVersionInfo.
    • For modules with single execution of protobuf-maven-plugin: define protobuf version being used in a property.
  2. Simplify and document steps for using shaded protobuf in the generated code, including:
    • Output directory for each execution of protobuf-maven-plugin is automatically added as source root (for compilation), so moving to the same dir at the end is unnecessary.
    • Separate output directories at the same level under generated-sources, instead of using a dir and its subdir (java and java/proto3). This allows using fewer <replace/> tasks.
  3. Misc:
    • Improve protobuf-maven-plugin execution <id>:
      • unshaded: include protobuf version
      • shaded: add for-hadoop and for-ratis, respectively
    • Add protobuf-maven-plugin to pluginManagement.
    • Remove plugin parameters that had default values.
    • In ozone-csi: merge global plugin config into that of the execution.

https://issues.apache.org/jira/browse/HDDS-5410

How was this patch tested?

CI:
https://github.com/adoroszlai/ozone/actions/runs/18108412916
https://github.com/adoroszlai/ozone/actions/runs/18152046866

Maven output excerpt for execution <id> changes:

[INFO] --- protobuf:0.6.1:compile (compile-proto-for-ratis) @ hdds-interface-client ---
[INFO] Compiling 1 proto file(s) to hadoop-hdds/interface-client/target/generated-sources/proto-java-for-ratis
[INFO] 
[INFO] --- protobuf:0.6.1:compile-custom (compile-proto-for-ratis) @ hdds-interface-client ---
[INFO] Compiling 1 proto file(s) to hadoop-hdds/interface-client/target/generated-sources/proto-java-for-ratis
[INFO] 
[INFO] --- protobuf:0.6.1:compile (compile-proto-2.5.0) @ hdds-interface-client ---
[INFO] Compiling 2 proto file(s) to hadoop-hdds/interface-client/target/generated-sources/proto-java-for-protobuf-2.5.0
[INFO] 
[INFO] --- protobuf:0.6.1:compile (compile-proto-for-hadoop) @ hdds-interface-client ---
[INFO] Compiling 2 proto file(s) to hadoop-hdds/interface-client/target/generated-sources/proto-java-for-hadoop
$ ozone version
...
Compiled with protoc 2.5.0, 3.25.8, 3.25.5 (Hadoop), 3.25.5 (Ratis)
...

@adoroszlai adoroszlai self-assigned this Sep 29, 2025
@adoroszlai adoroszlai added the build Pull request that modifies the build process label Sep 29, 2025
- Add `protobuf-maven-plugin` to `pluginManagement`.
- Include protobuf version in execution ID where using unshaded protobuf.
- In `ozone-csi`: move global plugin config to the execution's.

For modules with multiple executions:
- Better separate output directory.
- Simplify and document steps for using shaded protobuf in the generated code.

For modules with single execution:
- Define protobuf version in property
- Remove parameters with default value
@adoroszlai adoroszlai marked this pull request as draft October 1, 2025 05:06
@adoroszlai adoroszlai changed the title HDDS-5410. Harmonize protobuf3 version HDDS-5410. Improve protobuf compilation Oct 1, 2025
@adoroszlai adoroszlai marked this pull request as ready for review October 1, 2025 07:10
@adoroszlai adoroszlai requested a review from sodonnel October 6, 2025 13:10
Copy link
Contributor

@sodonnel sodonnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adoroszlai adoroszlai merged commit fbbfa79 into apache:master Oct 6, 2025
163 of 166 checks passed
@adoroszlai adoroszlai deleted the HDDS-5410 branch October 6, 2025 15:35
@adoroszlai
Copy link
Contributor Author

Thanks @sodonnel for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Pull request that modifies the build process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments