No upper bound on python dep protobuf results in latest protobuf version breakage (protobuf>=4.21.0)

[evamaxfield]

What happened?

I was deploying a new stack when I reached a long traceback that ultimately ended with a protobuf error:

File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 560, in __new__
          _message.Message._CheckCalledFromGeneratedFile()

Log: https://github.com/CouncilDataProject/denver/runs/6604387518?check_suite_focus=true

Steps to reproduce

I can't think of a minimal reproducible example at the given moment...

Expected Behavior

Infrastructure deploy to proceed without fail. This same stack configuration completed just fine just hours prior: https://github.com/CouncilDataProject/portland/runs/6597117332?check_suite_focus=true

When I ran a text diff against the two install lists, the only difference between the installed Python packages was protobuf-3.20.1 (succeeding) and protobuf-4.21.0 (failing)

Actual Behavior

Until latest version of protobuf is supported, install should use protobuf>=3.6,<4.21.0 and succeed.

Versions used

OS: Ubuntu 20.04.4
Python: 3.9.12
Pulumi (Python): 3.33.1
pulumi/actions: 3.33.1

Full Python package installed:

Successfully installed aiohttp-3.8.1 aiosignal-1.2.0 arpeggio-1.10.2 async-timeout-4.0.2 attrs-21.4.0 cachetools-5.1.0 cdp-backend-3.0.16 certifi-2022.5.18.1 charset-normalizer-2.0.12 dataclasses-json-0.5.7 decorator-5.1.1 dill-0.3.5.1 fireo-1.5.0 frozenlist-1.3.0 fsspec-2022.5.0 gcsfs-2022.5.0 google-api-core-2.8.0 google-auth-2.6.6 google-auth-oauthlib-0.5.1 google-cloud-core-2.3.0 google-cloud-firestore-2.3.4 google-cloud-storage-2.3.0 google-crc32c-1.3.0 google-resumable-media-2.3.3 googleapis-common-protos-1.56.1 grpcio-1.46.3 grpcio-status-1.46.3 idna-3.3 marshmallow-3.15.0 marshmallow-enum-1.5.1 multidict-6.0.2 mypy-extensions-0.4.3 oauthlib-3.2.0 packaging-21.3 parver-0.3.1 proto-plus-1.20.4 protobuf-4.21.0 pulumi-3.33.1 pulumi-gcp-6.24.0 pulumi-google-native-0.19.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 rsa-4.8 semver-2.13.0 six-1.16.0 typing-extensions-4.2.0 typing-inspect-0.7.1 urllib3-1.26.9 yarl-1.7.2

Additional context

Looks like this was unpinned in #9337

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[evamaxfield]

protobuf released a new version ~4 hours ago (2022-05-25 ~6pm PST): https://pypi.org/project/protobuf/#history

[Raggey]

Just adding to this, same issue.

Current workaround is pinning protobuf==3.20.1 in my requirements.txt

Some additional error messages:

If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower). 

https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

This might be related to generate.sh ?

[wallflower762]

I ran into this issue as well. I can also confirm that reverting to the previous major version of protobuf (3.20.1) resolved the issue.

Setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python also resolved the issue, but I prefer to use the last stable release.

[Frassle]

Pinning the version would require us to do a release, at which point we could just re-generate the protobuf files to fix this.
I think it might make sense to pin the major version of protobuf as part of that.

[stack72]

Folks we are looking into this immediately - thanks for raising the issue! We see the same problem on some of our nightly acceptance test runs

[wakingrufus]

this build still failed with the proto error: https://github.com/pulumi/pulumi/runs/6608080649?check_suite_focus=true
is the issue fully fixed?

[Frassle]

No, but we're continuing to work on it. The aim is to get it fixed and then get a release out as soon as possible.

[jspiewak]

Would it be possible to quickly push out the fix that just pins the version to <4 and work on the larger upgrade of protoc afterward?
We are dead in the water across many Pulumi projects and have little desire to implement the workarounds in multiple places as a stop gap.

[Frassle]

Would it be possible to quickly push out the fix that just pins the version to <4 and work on the larger upgrade of protoc afterward?

We looked into that but setting the pinned version still caused a whole load of our test to fail with the same issue. We have multiple engineers investigating multiple approaches to fix this.

[dixler]

It looks like protobuf 4.21.0 was yanked https://pypi.org/project/protobuf/#history

[lukehoban]

It looks like protobuf 4.21.0 was yanked https://pypi.org/project/protobuf/#history

Does that prevent this issue from affecting customers? Or only from afffecting customers who have not yet ended up with the bad version pinned in a venv? What's the state of customer impact here after this was yanked and after landing #9696?

[Frassle]

#9696 didn't seem to help our tests, but I suspect that was due to our test environments doing something strange rather than the pin being wrong.

Customers that have got 4.21.0 in their venv are going to stay stuck, even if we push a new pulumi release they would have to re-install requirements to see the version clash. Having said that I think most customers impacted by this are probably building venvs from scratch each time (which is why they saw this issue straight away) and with the package now yanked they will get protobuf 3.

[evamaxfield]

I have reran my infrastructure deploy job after the protobuf release was yanked and it worked (https://github.com/CouncilDataProject/denver/runs/6613594089?check_suite_focus=true). So for the time being we are seemingly okay but definitely something to pin for the future it seems.

Thanks all for the fast response ❤️

[denniswebb]

This is back. Just hit it in our env. I also see Protobuf 4.21.0 is back on pypi.

[stack72]

Folks, we are working on a release to mitigate this issue!

[dixler]

@denniswebb patched version v3.33.2 has been pushed

[ShedPlant]

I am seeing a similar issue today

pyproject.toml

[tool.poetry]
name = "issue-573"
version = "0.1.0"
description = "Reproduce pulumi-docker issue 573"
authors = ["expend"]

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.group.dev.dependencies]
pulumi-docker = "^4.1.1"
pulumi-aws = "^5.34.0"

poetry show

arpeggio          2.0.0  Packrat parser interpreter
attrs             22.2.0 Classes Without Boilerplate
dill              0.3.6  serialize all of python
grpcio            1.53.0 HTTP/2-based RPC framework
parver            0.4    Parse and manipulate version numbers.
protobuf          4.22.1 
pulumi            3.33.1 Pulumi's Python SDK
pulumi-aws        5.34.0 A Pulumi package for creating and managing Amazon Web Services (AWS) cloud resources.
pulumi-docker     4.1.1  A Pulumi package for interacting with Docker in Pulumi programs
pyyaml            6.0    YAML parser and emitter for Python
semver            3.0.0  Python helper for Semantic Versioning (https://semver.org)
six               1.16.0 Python 2 and 3 compatibility utilities
typing-extensions 4.5.0  Backported and Experimental Type Hints for Python 3.7+

shed@Ori issue_573 % pulumi -s sandbox up
Previewing update (sandbox)

View in Browser (Ctrl+O): https://app.pulumi.com/expend/pulumi-docker-issue-573/sandbox/previews/76edadd9-e38e-4ac2-a4eb-0b0a5a65ba93

     Type                 Name                             Plan     Info
     pulumi:pulumi:Stack  pulumi-docker-issue-573-sandbox           1 error; 31 messages


Diagnostics:
  pulumi:pulumi:Stack (pulumi-docker-issue-573-sandbox):
    error: an unhandled error occurred: program exited with non-zero exit code: 1

    Traceback (most recent call last):
      File "/usr/local/bin/pulumi-language-python-exec", line 16, in <module>
        import pulumi
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/__init__.py", line 33, in <module>
        from .config import (
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/config.py", line 22, in <module>
        from . import log
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/log.py", line 22, in <module>
        from .runtime.settings import get_engine
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/__init__.py", line 29, in <module>
        from .mocks import (
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/mocks.py", line 24, in <module>
        from . import rpc, rpc_manager
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/rpc.py", line 40, in <module>
        from . import known_types, settings
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/settings.py", line 23, in <module>
        from ..runtime.proto import engine_pb2_grpc, resource_pb2, resource_pb2_grpc
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/proto/__init__.py", line 20, in <module>
        from .analyzer_pb2 import *
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/proto/analyzer_pb2.py", line 15, in <module>
        from . import plugin_pb2 as plugin__pb2
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/pulumi/runtime/proto/plugin_pb2.py", line 34, in <module>
        _descriptor.FieldDescriptor(
      File "/Users/shed/.venvs/pulumi-docker-investigations/issue_573/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 561, in __new__
        _message.Message._CheckCalledFromGeneratedFile()
    TypeError: Descriptors cannot not be created directly.
    If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
    If you cannot immediately regenerate your protos, some other possible workarounds are:
     1. Downgrade the protobuf package to 3.20.x or lower.
     2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
    More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates