Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Oct 24, 2024

What changes were proposed in this pull request?

This PR aims to upgrade protobuf Python package from 4.25.1 to 5.28.3 for Apache Spark 4.0.0 on February 2025.

Why are the changes needed?

To bring the latest bug fixes and support Python 3.13 which requires grpcio*>=1.66.2 (grpcio-status>=1.66.2 requires protobuf>=5.26.1).

Does this PR introduce any user-facing change?

This is a dependency change with no behavior change.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-50106][PYTHON] Upgrade protobuf Python package to 5.28.3 [SPARK-50106][BUILD] Upgrade protobuf Python package to 5.28.3 and protobuf-java to 4.28.3 Oct 24, 2024
@dongjoon-hyun dongjoon-hyun marked this pull request as draft October 24, 2024 20:58
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-50106][BUILD] Upgrade protobuf Python package to 5.28.3 and protobuf-java to 4.28.3 [SPARK-50106][BUILD] Upgrade protobuf Python package to 5.28.3 Oct 24, 2024
@dongjoon-hyun dongjoon-hyun marked this pull request as ready for review October 24, 2024 21:03
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-50106][BUILD] Upgrade protobuf Python package to 5.28.3 [SPARK-50106][PYTHON] Upgrade protobuf Python package to 5.28.3 Oct 24, 2024
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Oct 24, 2024

cc @HyukjinKwon
This is intentionally focusing on Python side only. For Java-side, we have a separate PR for further discussion.

@dongjoon-hyun
Copy link
Member Author

All tests passed.

@dongjoon-hyun
Copy link
Member Author

Could you review this PR too, @zhengruifeng and @xinrong-meng ?

@dongjoon-hyun
Copy link
Member Author

Could you review this PR, @viirya ?

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon . Merged to master for Apache Spark 4.0.0 on February 2025.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-50106 branch October 25, 2024 13:59
Copy link
Member

@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@dongjoon-hyun
Copy link
Member Author

Thank you, @viirya .

HyukjinKwon added a commit that referenced this pull request Nov 5, 2024
…ents.txt

### What changes were proposed in this pull request?

This PR proposes to add 'protobuf==5.28.3' in `dev/requirements.txt`. If you have an old version in your env, you would face an error like:

```
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 5.28.3 runtime 5.28.2. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.
```

This PR is a sort of a followup of #48644

### Why are the changes needed?

So developers can easily set up their env.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

Manually installed and tested.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #48763 from HyukjinKwon/note-protobuf.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants