Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

airflow db migrate crashing with new connexion == 3.0.0 #35375

Closed
1 of 2 tasks
eselyavka opened this issue Nov 2, 2023 · 4 comments
Closed
1 of 2 tasks

airflow db migrate crashing with new connexion == 3.0.0 #35375

eselyavka opened this issue Nov 2, 2023 · 4 comments
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet

Comments

@eselyavka
Copy link

Apache Airflow version

2.7.2

What happened

Team, it looks like i just discovered bug in airflow 2.7.2(latest) which were introduced by this package https://pypi.org/project/connexion/3.0.0/, which were released several hours ago
I am running my airflow image in ubuntu 20.04 x86_64 docker image under the python 3.9.16

root@cbe8e0b21de3:/# python3.9 --version
Python 3.9.18

I am installing airflow via pip

pip install apache-airflow=2.7.2

I am seeing this error when performing airflow db migrate

root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated.  Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:54:50.283-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:54:50.285-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/airflow/__main__.py", line 59, in main
    args.func(args)
  File "/usr/local/lib/python3.9/dist-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py", line 49, in initdb
    db.initdb()
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/session.py", line 77, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 739, in initdb
    _create_db_from_orm(session=session)
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 727, in _create_db_from_orm
    command.stamp(config, "head")
  File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 685, in stamp
    script.run_env()
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 579, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 117, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/environment.py", line 938, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/migration.py", line 612, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 673, in do_stamp
    return script._stamp_revs(util.to_tuple(destination_revs), rev)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 494, in _stamp_revs
    self.revision_map.filter_for_lineage(
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 680, in filter_for_lineage
    id_, branch_label = self._resolve_revision_number(check_against)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 754, in _resolve_revision_number
    self._revision_map
  File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 197, in _revision_map
    for revision in self._generator():
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 149, in _load_revisions
    script = Script._from_filename(self, dir_name, filename)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 1035, in _from_filename
    module = util.load_python_file(dir_, filename)
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/versions/0074_2_0_0_resource_based_permissions.py", line 30, in <module>
    from airflow.www.app import cached_app
  File "/usr/local/lib/python3.9/dist-packages/airflow/www/app.py", line 50, in <module>
    from airflow.www.extensions.init_views import (
  File "/usr/local/lib/python3.9/dist-packages/airflow/www/extensions/init_views.py", line 27, in <module>
    from connexion.decorators.validation import RequestBodyValidator
ModuleNotFoundError: No module named 'connexion.decorators.validation'

If i downgrade connexion to 2.14.2 db migration command succeed

root@cbe8e0b21de3:/# pip3.9 install connexion==2.14.2
Collecting connexion==2.14.2
  Downloading connexion-2.14.2-py2.py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 3.5 MB/s eta 0:00:00
Collecting clickclick<21,>=1.2 (from connexion==2.14.2)
  Downloading clickclick-20.10.2-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: jsonschema<5,>=2.5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (4.19.2)
Requirement already satisfied: PyYAML<7,>=5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (6.0.1)
Requirement already satisfied: requests<3,>=2.9.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.31.0)
Requirement already satisfied: inflection<0.6,>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (0.5.1)
Collecting werkzeug<2.3,>=1.0 (from connexion==2.14.2)
  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (23.2)
Requirement already satisfied: flask<2.3,>=1.0.4 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.2.5)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.1.2)
Requirement already satisfied: click>=4.0 in /usr/local/lib/python3.9/dist-packages (from clickclick<21,>=1.2->connexion==2.14.2) (8.1.7)
Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (6.8.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.10.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2019.11.28)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug<2.3,>=1.0->connexion==2.14.2) (2.1.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->flask<2.3,>=1.0.4->connexion==2.14.2) (3.17.0)
Installing collected packages: werkzeug, clickclick, connexion
  Attempting uninstall: werkzeug
    Found existing installation: Werkzeug 3.0.1
    Uninstalling Werkzeug-3.0.1:
      Successfully uninstalled Werkzeug-3.0.1
  Attempting uninstall: connexion
    Found existing installation: connexion 3.0.0
    Uninstalling connexion-3.0.0:
      Successfully uninstalled connexion-3.0.0
Successfully installed clickclick-20.10.2 connexion-2.14.2 werkzeug-2.2.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated.  Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:56:12.691-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:56:12.714-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 405de8318b3a
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
root@cbe8e0b21de3:/#

What you think should happen instead

root@cbe8e0b21de3:/# pip3.9 install connexion==2.14.2
Collecting connexion==2.14.2
  Downloading connexion-2.14.2-py2.py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 3.5 MB/s eta 0:00:00
Collecting clickclick<21,>=1.2 (from connexion==2.14.2)
  Downloading clickclick-20.10.2-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: jsonschema<5,>=2.5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (4.19.2)
Requirement already satisfied: PyYAML<7,>=5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (6.0.1)
Requirement already satisfied: requests<3,>=2.9.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.31.0)
Requirement already satisfied: inflection<0.6,>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (0.5.1)
Collecting werkzeug<2.3,>=1.0 (from connexion==2.14.2)
  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (23.2)
Requirement already satisfied: flask<2.3,>=1.0.4 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.2.5)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.1.2)
Requirement already satisfied: click>=4.0 in /usr/local/lib/python3.9/dist-packages (from clickclick<21,>=1.2->connexion==2.14.2) (8.1.7)
Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (6.8.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.10.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2019.11.28)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug<2.3,>=1.0->connexion==2.14.2) (2.1.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->flask<2.3,>=1.0.4->connexion==2.14.2) (3.17.0)
Installing collected packages: werkzeug, clickclick, connexion
  Attempting uninstall: werkzeug
    Found existing installation: Werkzeug 3.0.1
    Uninstalling Werkzeug-3.0.1:
      Successfully uninstalled Werkzeug-3.0.1
  Attempting uninstall: connexion
    Found existing installation: connexion 3.0.0
    Uninstalling connexion-3.0.0:
      Successfully uninstalled connexion-3.0.0
Successfully installed clickclick-20.10.2 connexion-2.14.2 werkzeug-2.2.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated.  Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:56:12.691-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:56:12.714-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 405de8318b3a
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
root@cbe8e0b21de3:/#

How to reproduce

docker run -it --rm --entrypoint /bin/bash ubuntu:focal
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.9 gcc python3.9-dev
apt install curl
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
apt install python3.9-distutils
python3.9 get-pip.py
pip install apache-airflow==2.7.2
airflow db init

Operating System

NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

Versions of Apache Airflow Providers

pip3.9 install apache-airflow==2.7.2

Deployment

Other

Deployment details

docker run -it --rm --entrypoint /bin/bash ubuntu:focal
apt update
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt update
apt install curl python3.9 gcc python3.9-dev python3.9-distutils
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.9 get-pip.py
pip install apache-airflow==2.7.2
airflow db init

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@eselyavka eselyavka added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Nov 2, 2023
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Nov 2, 2023

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@Taragolis
Copy link
Contributor

connexion already limited in:

Follow up task exists:

For avoid further error on any new package updates which might potentially break airflow it is recommend way to use constraints file

@potiuk
Copy link
Member

potiuk commented Nov 2, 2023

Also @eselyavka you should install airflow from the scratch (if you want reproducible install) following the only supported method - i.e. with constraints - https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

Then you can upgrade any new dependencies as you see fit (and test) but the "base" installation is guaranteed to not cause problems when external dependencies are changing something in their behaviours

It is SPECIFICALLY designed to make sure that such breaking releases in "future" dependencies are not affecting user installations - for users who actually follow those instructions that is. You wil make yourself a favour to use the protections we implemented for our users.

Also you might want to watch my talk https://www.youtube.com/watch?v=zPjIQjjjyHI describing why and how it works if you are interested.

@potiuk potiuk mentioned this issue Nov 2, 2023
Closed
2 tasks
@RobbeSneyders RobbeSneyders mentioned this issue Nov 3, 2023
Closed
@Taragolis Taragolis mentioned this issue Nov 4, 2023
Closed
2 tasks
@eselyavka
Copy link
Author

@potiuk thank you very much for detailed answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@potiuk @eselyavka @Taragolis