Update Protobuf and GCP dependencies in Beam Python SDK #24599

AnandInguva · 2022-12-08T20:49:46Z

Fixes: #23355
Fixes: #24569
Fixes: #25581
Fixes: #25328
Fixes: #20991
Fixes: #21019
Fixes: #24432
Fixes: #23585
Fixes: #22742

Also look at #22319 for proto2_coder_test_messages_pb2.py generation
Tensorflow Protobuf update GH issue: tensorflow/tensorflow#59221

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

AnandInguva · 2022-12-08T21:12:47Z

Local tested error in Python 3.8 MacOS env:

Traceback (most recent call last):
  File "/Users/anandinguva/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/anandinguva/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/__init__.py", line 92, in <module>
    from apache_beam import coders
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/coders/__init__.py", line 17, in <module>
    from apache_beam.coders.coders import *
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/coders/coders.py", line 59, in <module>
    from apache_beam.coders import coder_impl
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/coders/coder_impl.py", line 63, in <module>
    from apache_beam.typehints.schemas import named_tuple_from_schema
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/typehints/__init__.py", line 38, in <module>
    from apache_beam.typehints.arrow_type_compatibility import *
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/typehints/arrow_type_compatibility.py", line 32, in <module>
    from apache_beam.portability.api import schema_pb2
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/portability/api/__init__.py", line 21, in <module>
    from .org.apache.beam.model.pipeline import v1
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/portability/api/org/apache/beam/model/pipeline/v1/__init__.py", line 17, in <module>
    from . import external_transforms_pb2
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/portability/api/org/apache/beam/model/pipeline/v1/external_transforms_pb2.py", line 14, in <module>
    from . import schema_pb2 as org_dot_apache_dot_beam_dot_model_dot_pipeline_dot_v1_dot_schema__pb2
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/portability/api/org/apache/beam/model/pipeline/v1/schema_pb2.py", line 15, in <module>
    from . import beam_runner_api_pb2 as org_dot_apache_dot_beam_dot_model_dot_pipeline_dot_v1_dot_beam__runner__api__pb2
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/portability/api/org/apache/beam/model/pipeline/v1/beam_runner_api_pb2.py", line 14, in <module>
    from . import endpoints_pb2 as org_dot_apache_dot_beam_dot_model_dot_pipeline_dot_v1_dot_endpoints__pb2
  File "/Users/anandinguva/projects/beam/sdks/python/apache_beam/portability/api/org/apache/beam/model/pipeline/v1/endpoints_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/Users/anandinguva/.pyenv/versions/deps_3.8.12/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

AnandInguva · 2022-12-08T21:15:46Z

https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

AnandInguva · 2022-12-20T18:31:33Z

protocolbuffers/protobuf#10051

AnandInguva · 2023-02-07T18:00:50Z

The urns are not getting build when protobuf is updated to 4.21.11.

Can be reproduced with

python setup.py sdist
pip install dist/apache-beam-x.xx.x.dev0.tar.gz

Update:

Looked more into it. We generate the sidecar urn files to support mypy type checker. Some of the types of Protobuf classes has been changed and they were not getting matched with the current code, hence the URNs are not getting generated.

AnandInguva · 2023-03-10T21:56:11Z

Run Python 3.7 PostCommit

AnandInguva · 2023-03-12T18:18:27Z

Run Python 3.9 PostCommit

AnandInguva · 2023-03-12T18:18:37Z

Run Python 3.7 PostCommit

AnandInguva · 2023-03-13T14:55:22Z

blocker: tensorflow_hub 0.12 (latest) is not compatible with tensorflow 2.12, which makes our TF integration tests fail. We need to wait until tensorflow_hub releases a newer version.

we can skip the TF tests for now and have a tracking issue for tensorflow_hub release and unskip them before the next release.

AnandInguva · 2023-03-15T16:23:27Z

Run Python 3.9 PostCommit

AnandInguva · 2023-03-15T17:53:37Z

Run Python 3.7 PostCommit

AnandInguva · 2023-03-15T17:54:01Z

Run Python_PVR_Flink PreCommit

tvalentyn · 2023-03-15T23:58:34Z

merging

tvalentyn · 2023-03-15T23:59:11Z

Thanks, @AnandInguva !

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

Abacn · 2023-05-15T17:30:57Z

sdks/python/container/base_image_requirements_manual.txt

@@ -32,14 +32,11 @@ cython<1
 # some versions of libraries that launch Beam pipelines, like tensorflow-transform.
 # Leaving 'future' in our containers for now prevent breaking tft users.
 future
-# TODO: Remove the upper bound once Tensorflow 2.11 is released.
-# https://github.com/apache/beam/issues/23355
-google-cloud-profiler<4.0.0


Any reason removing google-cloud-profiler dep in base image? This has broken the google-cloud-profiler support on Beam ~~v2.47.0~~

Update: confirmed that support on v2.47.0 is not affected. It only affects dev version for now

This was not intentional. Looks like Dataflow containers still include it.

we intentionally removed google-cloud-debugger from Dataflow containers, perhaps there was confusion between these two deps.

Dataflow container now does not include it. Can be seen by https://github.com/apache/beam/blob/master/sdks/python/container/py38/base_image_requirements.txt where google-cloud-profiler is no longer listed in. I now get error using profiler: #26698

ahh...this was not intentional.

@Abacn these are beam external images. Dataflow images are internal: https://source.corp.google.com/piper///depot/google3/third_party/cloud/dataflow/worker/images/base_python/Dockerfile.base-python;rcl=522192081;l=117

Thanks, this is less severe, so it only affects dev version

* Update build dependencies and mypy-protobuf * Update import * Add os.environ['PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION'] = 'python' * Update gen_protos.py * Manually generate proto file. * update proto file * Remove redundant code * Add encoding * Update protobuf * Remove tensorflow ad google cloud profiler * update dependencies * update facets-overview * Update goolge-cloud-langugage code to support breaking changes from v2.0 * Update video intelligence breaking changes * Update lower bounds on some gcp deps * Update deps * Debug * Update gen_protos.py to supoort google._upb * Change checking condition to cpp to upb * Update google-cloud-datastore ByteSize * Update Dockerfile * raise Error * Update dependencies bound * Add optional field * Add license headers * Add more optional * Changes types to make them compatible with mypy-protobuf generated stubs * Add license for grpcio-status * update mypy to 0.790 * add more optional fields to satisfy mypy type checker * Identify and solve the mypy type checks * fix lint * Revert "add more optional fields to satisfy mypy type checker" This reverts commit 1ebc815. * Revert "Add more optional" This reverts commit 3801c16. * Revert "Add optional field" This reverts commit 35631f5. * add relax_strict_optional_primitives to mypy_out * Fix up formatting * Fix up lint * Fix up formatting * Fix up lint * fix lint * Fix up docs * Add gh issue to update code and doc strings * Update lower bound of gcp dependencies * Add tensorflow rc * Add tf rc version and comment onnx tests * Update container dependencies * comment onnx task * Change bound on google-cloud-recommendations * Modify FakeMutation to support mocking proto messages * Update comment * Add todo for uncommenting onnx changes * update comment * Update dependencies * revert doc change * Refactor code/remove helper function from util * Update comment * Uncomment changes * Update tensorflow * Comment tfx-bsl tests * Update tox.ini syntax * refactor * fix requirements * Skip tensorflow tests on Python 3.7

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

github-actions bot added the python label Dec 8, 2022

AnandInguva force-pushed the update-protobuf branch from b84821d to e7a7858 Compare December 20, 2022 18:23

AnandInguva force-pushed the update-protobuf branch from e7a7858 to 04824b7 Compare December 28, 2022 15:53

AnandInguva changed the title ~~Update build dependencies and mypy-protobuf~~ Update Python SDK build dependencies Feb 7, 2023

AnandInguva added 3 commits February 14, 2023 14:10

Update build dependencies and mypy-protobuf

a3e72b7

Update import

673aaf3

Add os.environ['PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION'] = 'python'

ee1da36

AnandInguva force-pushed the update-protobuf branch from 04824b7 to ee1da36 Compare February 14, 2023 20:47

AnandInguva and others added 8 commits February 16, 2023 14:13

Update gen_protos.py

6140ff7

Manually generate proto file.

8f24747

update proto file

e6af718

Remove redundant code

9c44141

Add encoding

5860711

Update protobuf

fd4176f

Remove tensorflow ad google cloud profiler

2cf5526

update dependencies

afbe42d

github-actions bot added the docker label Feb 17, 2023

AnandInguva added 9 commits February 18, 2023 11:44

update facets-overview

ab20544

Update goolge-cloud-langugage code to support breaking changes from v2.0

990cf94

Update video intelligence breaking changes

7eab925

Update lower bounds on some gcp deps

073788d

Merge remote-tracking branch 'upstream/master' into update-protobuf

15a4d3b

Update deps

30b9b7f

Debug

882e5af

Update gen_protos.py to supoort google._upb

bf2846c

Change checking condition to cpp to upb

3a0c1b7

github-actions bot added model and removed model labels Mar 10, 2023

fix requirements

ba8b376

AnandInguva added 2 commits March 13, 2023 10:41

Skip tensorflow tests on Python 3.7

5534073

Merge remote-tracking branch 'upstream/master' into update-protobuf

a678bc4

Merge remote-tracking branch 'upstream/master' into update-protobuf

0f0c3ec

tvalentyn approved these changes Mar 15, 2023

View reviewed changes

tvalentyn merged commit 5cb1711 into apache:master Mar 15, 2023

potiuk mentioned this pull request Mar 27, 2023

Update SDKs for google provider package apache/airflow#30067

Merged

38 tasks

tvalentyn added a commit that referenced this pull request Mar 29, 2023

Add #24599 to CHANGES.md (#26020)

52531b7

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

This was referenced Mar 30, 2023

[Bug]: adapt to protobuf versioning #26043

Closed

[Bug]: Py310 unittests on MacOS are failing #23734

Closed

Cannot install apache-beam gcp and google-ads #26080

Closed

tvalentyn mentioned this pull request Apr 12, 2023

[Task]: Do an audit of Beam Python SDK dependencies and verify that bounds are set appropriately. #25655

Closed

15 tasks

jakubrauch pushed a commit to jakubrauch/beam that referenced this pull request Apr 27, 2023

Add apache#24599 to CHANGES.md (apache#26020)

1029722

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

Abacn reviewed May 15, 2023

View reviewed changes

Abacn mentioned this pull request May 15, 2023

[Bug]: Python Google cloud profiler broken on dev branch #26698

Closed

15 tasks

rezarokni pushed a commit to rezarokni/beam that referenced this pull request May 22, 2023

Add apache#24599 to CHANGES.md (apache#26020)

ccb2950

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

tvalentyn mentioned this pull request Jun 30, 2023

[Bug]: Python SDK sometimes crashes in streaming jobs running on 2.47.0+ SDK #27330

Closed

15 tasks

tvalentyn mentioned this pull request Aug 31, 2023

[Bug]: [Python SDK] Memory leak in 2.47.0 - 2.51.0 SDKs. #28246

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Protobuf and GCP dependencies in Beam Python SDK #24599

Update Protobuf and GCP dependencies in Beam Python SDK #24599

AnandInguva commented Dec 8, 2022 •

edited

Loading

AnandInguva commented Dec 8, 2022

AnandInguva commented Dec 8, 2022

AnandInguva commented Dec 20, 2022

AnandInguva commented Feb 7, 2023 •

edited

Loading

AnandInguva commented Mar 10, 2023

AnandInguva commented Mar 12, 2023

AnandInguva commented Mar 12, 2023

AnandInguva commented Mar 13, 2023

AnandInguva commented Mar 15, 2023

AnandInguva commented Mar 15, 2023

AnandInguva commented Mar 15, 2023

tvalentyn commented Mar 15, 2023

tvalentyn commented Mar 15, 2023

Abacn May 15, 2023 •

edited

Loading

tvalentyn May 15, 2023

tvalentyn May 15, 2023 •

edited

Loading

Abacn May 15, 2023

AnandInguva May 15, 2023 •

edited

Loading

Abacn May 15, 2023

Update Protobuf and GCP dependencies in Beam Python SDK #24599

Update Protobuf and GCP dependencies in Beam Python SDK #24599

Conversation

AnandInguva commented Dec 8, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

AnandInguva commented Dec 8, 2022

AnandInguva commented Dec 8, 2022

AnandInguva commented Dec 20, 2022

AnandInguva commented Feb 7, 2023 • edited Loading

AnandInguva commented Mar 10, 2023

AnandInguva commented Mar 12, 2023

AnandInguva commented Mar 12, 2023

AnandInguva commented Mar 13, 2023

AnandInguva commented Mar 15, 2023

AnandInguva commented Mar 15, 2023

AnandInguva commented Mar 15, 2023

tvalentyn commented Mar 15, 2023

tvalentyn commented Mar 15, 2023

Abacn May 15, 2023 • edited Loading

Choose a reason for hiding this comment

tvalentyn May 15, 2023

Choose a reason for hiding this comment

tvalentyn May 15, 2023 • edited Loading

Choose a reason for hiding this comment

Abacn May 15, 2023

Choose a reason for hiding this comment

AnandInguva May 15, 2023 • edited Loading

Choose a reason for hiding this comment

Abacn May 15, 2023

Choose a reason for hiding this comment

AnandInguva commented Dec 8, 2022 •

edited

Loading

AnandInguva commented Feb 7, 2023 •

edited

Loading

Abacn May 15, 2023 •

edited

Loading

tvalentyn May 15, 2023 •

edited

Loading

AnandInguva May 15, 2023 •

edited

Loading