[Task]: Support Protobuf v4 in Python SDK.

[tvalentyn]

What needs to happen?

Upgrade upper & lower bounds to use Protobuf v4.

Issue Priority

Priority: 2

Issue Component

Component: dependencies

[tvalentyn]

Requirements are specified in: https://github.com/apache/beam/blob/1ba7821a09cb1b9a1793f7044ed64aabf55f7713/sdks/python/setup.py

Protobuf releases switched to v4 in: https://pypi.org/project/protobuf/#history

Let's try to switch to Protobuf v4, or identify (what if anything) doesn't work.

To avoid having a mismatch at startup and at runtime for this dependency, let's tighten the upper bound (e.g. protobuf<=4.21.11 and rely on dependabot for perodic upgrades, or otherwise make periodic upgrades for every release milestone if we still need special casing (which is the case for MacOS now), how we do with cloudpickle, see: #23119

[tvalentyn]

@AnandInguva would you have time sometime this/next week to take a look if Protobuf v4 just works? Thanks.

[AnandInguva]

.take-issue

[AnandInguva]

We are still blocked on Tensorflow for the protobuf update. Tensorflow is still on protobuf >= 3.9.2, < 3.20
https://github.com/tensorflow/tensorflow/blob/d5b57ca93e506df258271ea00fc29cf98383a374/tensorflow/tools/pip_package/setup.py#L102

[aucampia]

We are still blocked on Tensorflow for the protobuf update. Tensorflow is still on protobuf >= 3.9.2, < 3.20 https://github.com/tensorflow/tensorflow/blob/d5b57ca93e506df258271ea00fc29cf98383a374/tensorflow/tools/pip_package/setup.py#L102

I made an upstream issue to track this:

• protobuf 4 is not supported tensorflow/tensorflow#59221