Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit Google Protobuf for compatibility with bigquery client #742

Closed
sunank200 opened this issue Aug 24, 2022 · 0 comments · Fixed by #743
Closed

Limit Google Protobuf for compatibility with bigquery client #742

sunank200 opened this issue Aug 24, 2022 · 0 comments · Fixed by #743
Assignees

Comments

@sunank200
Copy link
Contributor

Describe the bug
The Biquery client does not work well with protobuf > 3.20.0 and it should be limited until we solve the problem.
Got the following error:

[2022-08-22T22:15:58.602045] Traceback (most recent call last):
[2022-08-22T22:15:58.602045] File "/app/api/run_task_utils.py", line 134, in run
[2022-08-22T22:15:58.602045] self.task_instance._run_raw_task()
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
[2022-08-22T22:15:58.602045] return func(*args, session=session, **kwargs)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1340, in _run_raw_task
[2022-08-22T22:15:58.602045] self._execute_task_with_callbacks(context)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1477, in _execute_task_with_callbacks
[2022-08-22T22:15:58.602045] result = self._execute_task(context, self.task)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1538, in _execute_task
[2022-08-22T22:15:58.602045] result = execute_callable(context=context)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/astro/sql/operators/transform.py", line 24, in execute
[2022-08-22T22:15:58.602045] super().execute(context)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/astro/sql/operators/base_decorator.py", line 60, in execute
[2022-08-22T22:15:58.602045] self.database_impl = create_database(self.conn_id)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/astro/databases/__init__.py", line 31, in create_database
[2022-08-22T22:15:58.602045] module = importlib.import_module(module_path)
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
[2022-08-22T22:15:58.602045] return _bootstrap._gcd_import(name[level:], package, level)
[2022-08-22T22:15:58.602045] File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
[2022-08-22T22:15:58.602045] File "<frozen importlib._bootstrap>", line 983, in _find_and_load
[2022-08-22T22:15:58.602045] File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
[2022-08-22T22:15:58.602045] File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
[2022-08-22T22:15:58.602045] File "<frozen importlib._bootstrap_external>", line 728, in exec_module
[2022-08-22T22:15:58.602045] File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/astro/databases/google/bigquery.py", line 8, in <module>
[2022-08-22T22:15:58.602045] from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 57, in <module>
[2022-08-22T22:15:58.602045] from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/providers/google/common/hooks/base_google.py", line 49, in <module>
[2022-08-22T22:15:58.602045] from airflow.providers.google.cloud.utils.credentials_provider import (
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/utils/credentials_provider.py", line 36, in <module>
[2022-08-22T22:15:58.602045] from airflow.providers.google.cloud._internal_client.secret_manager_client import _SecretManagerClient
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/_internal_client/secret_manager_client.py", line 23, in <module>
[2022-08-22T22:15:58.602045] from google.cloud.secretmanager_v1 import SecretManagerServiceClient
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/google/cloud/secretmanager_v1/__init__.py", line 22, in <module>
[2022-08-22T22:15:58.602045] from google.cloud.secretmanager_v1 import types
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/google/cloud/secretmanager_v1/types.py", line 23, in <module>
[2022-08-22T22:15:58.602045] from google.cloud.secretmanager_v1.proto import resources_pb2
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/google/cloud/secretmanager_v1/proto/resources_pb2.py", line 62, in <module>
[2022-08-22T22:15:58.602045] type=None,
[2022-08-22T22:15:58.602045] File "/usr/local/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 755, in __new__
[2022-08-22T22:15:58.602045] _message.Message._CheckCalledFromGeneratedFile()
[2022-08-22T22:15:58.602045] TypeError: Descriptors cannot not be created directly.
[2022-08-22T22:15:58.602045] If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
[2022-08-22T22:15:58.602045] If you cannot immediately regenerate your protos, some other possible workarounds are:
[2022-08-22T22:15:58.602045] 1. Downgrade the protobuf package to 3.20.x or lower.
[2022-08-22T22:15:58.602045] 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
[2022-08-22T22:15:58.602045]
[2022-08-22T22:15:58.602045] More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates 

Issue relates to: protocolbuffers/protobuf#10051 and apache/airflow#25886

@sunank200 sunank200 self-assigned this Aug 24, 2022
kaxil pushed a commit that referenced this issue Aug 24, 2022
The Biquery client does not work well with protobuf > 3.20.0

closes: #742
Also fixed in apache/airflow#25886
sunank200 added a commit that referenced this issue Aug 24, 2022
The Biquery client does not work well with protobuf > 3.20.0

closes: #742
Also fixed in apache/airflow#25886

(cherry picked from commit d6911b3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant