Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix circular import in DB migration job #31204

Merged
merged 1 commit into from
May 11, 2023

Conversation

LipuFei
Copy link
Contributor

@LipuFei LipuFei commented May 11, 2023

Related to #30988 and #31033

Version: 2.6.1rc1
Docker image: apache/airflow:2.6.1rc1-python3.10

The root-level import of TaskInstance will cause a circular import problem when both Kubernetes and Sentry are enabled. This will show up in the DB migration job when I use the official Helm chart 1.9.0 to deploy.

I have tested this fix by patching the image apache/airflow:2.6.1rc1-python3.10 with this kubernetes_executor.py

Here is the stack trace:

Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/cli.py", line 112, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 84, in upgradedb
db.upgradedb(
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 76, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/db.py", line 1545, in upgradedb
import_all_models()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/__init__.py", line 60, in import_all_models
__getattr__(name)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/__init__.py", line 78, in __getattr__
val = import_string(f"{path}.{name}")
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 36, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dag.py", line 82, in <module>
from airflow.models.dagrun import DagRun
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dagrun.py", line 57, in <module>
from airflow.models.taskinstance import TaskInstance as TI
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 100, in <module>
from airflow.sentry import Sentry
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py", line 195, in <module>
Sentry = ConfiguredSentry()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py", line 92, in __init__
executor_class, _ = ExecutorLoader.import_default_executor_cls()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 158, in import_default_executor_cls
executor, source = cls.import_executor_cls(executor_name)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 134, in import_executor_cls
return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 129, in _import_and_validate
executor = import_string(path)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 36, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/celery_kubernetes_executor.py", line 26, in <module>
from airflow.executors.kubernetes_executor import KubernetesExecutor
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/kubernetes_executor.py", line 49, in <module>
from airflow.models.taskinstance import TaskInstance
ImportError: cannot import name 'TaskInstance' from partially initialized module 'airflow.models.taskinstance' (most likely due to a circular import) (/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py) 

@boring-cyborg boring-cyborg bot added provider:cncf-kubernetes Kubernetes provider related issues area:Scheduler including HA (high availability) scheduler labels May 11, 2023
related: apache#30988, apache#31033

The root-level import of TaskInstance will still cause a circular import
problem when both Kubernetes and Sentry are enabled. This will show up
in the DB migration job if I use the official Helm chart 1.9.0 to
deploy.
@jedcunningham jedcunningham merged commit be8f96b into apache:main May 11, 2023
@LipuFei LipuFei deleted the fix/circular-import branch May 11, 2023 22:49
@ephraimbuddy ephraimbuddy added this to the Airflow 2.6.1 milestone May 12, 2023
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label May 12, 2023
ephraimbuddy pushed a commit that referenced this pull request May 12, 2023
related: #30988, #31033

The root-level import of TaskInstance will still cause a circular import
problem when both Kubernetes and Sentry are enabled. This will show up
in the DB migration job if I use the official Helm chart 1.9.0 to
deploy.

Co-authored-by: Lipu Fei <lipu.fei@kpn.com>
(cherry picked from commit be8f96b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes provider related issues type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants