airflow 2.3.X breaks sync-users script #607

asosnovsky-sumologic · 2022-06-14T20:26:18Z

Checks

I have checked for existing issues.
This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.12-eks-a64ea69", GitCommit:"d4336843ba36120e9ed1491fddff5f2fec33eb77", GitTreeState:"clean", BuildDate:"2022-05-12T18:29:27Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

Description

We are deploying a new helm chart via argoCD, and the sync-users containers appear to failing due to an issue in the python code.

Relevant Logs

/home/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:525: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
  option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
/home/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:525: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
  option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
/home/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
Traceback (most recent call last):
  File "/mnt/scripts/sync_users.py", line 142, in <module>
    flask_app = www_app.create_app()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/app.py", line 136, in create_app
    init_appbuilder(flask_app)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/extensions/init_appbuilder.py", line 638, in init_appbuilder
    AirflowAppBuilder(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/extensions/init_appbuilder.py", line 162, in __init__
    self.init_app(app, session)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/extensions/init_appbuilder.py", line 208, in init_app
    self.sm = self.security_manager_class(self)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/security.py", line 189, in __init__
    super().__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/fab_security/sqla/manager.py", line 69, in __init__
    user_datamodel = SQLAInterface(self.user_model)
  File "/home/airflow/.local/lib/python3.10/site-packages/flask_appbuilder/models/sqla/interface.py", line 68, in __init__
    for prop in class_mapper(obj).iterate_properties:
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/base.py", line 436, in class_mapper
    mapper = _inspect_mapped_class(class_, configure=configure)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/base.py", line 415, in _inspect_mapped_class
    mapper._check_configure()
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 1874, in _check_configure
    _configure_registries({self.registry}, cascade=True)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 3384, in _configure_registries
    _do_configure_registries(registries, cascade)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 3423, in _do_configure_registries
    mapper._post_configure_properties()
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 1891, in _post_configure_properties
    prop.init()
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/interfaces.py", line 228, in init
    self.do_init()
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/relationships.py", line 2135, in do_init
    self._process_dependent_arguments()
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/relationships.py", line 2230, in _process_dependent_arguments
    self.target = self.entity.persist_selectable
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 1093, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/relationships.py", line 2097, in entity
    argument = self._clsregistry_resolve_name(self.argument)()
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/clsregistry.py", line 393, in _resolve_name
    rval = d[token]
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/util/_collections.py", line 746, in __missing__
    self[key] = val = self.creator(key)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/clsregistry.py", line 352, in _access_cls
    return _determine_container(key, decl_class_registry[key])
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/clsregistry.py", line 313, in _determine_container
    value = value.attempt_get([], key)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/orm/clsregistry.py", line 144, in attempt_get
    raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Multiple classes found for path "Permission" in the registry of this declarative base. Please use a fully module-qualified path.

Custom Helm Values

airflow:
  airflow:
    legacyCommands: false
    image:
      repository: apache/airflow
      tag: 2.3.2rc2-python3.10

    executor: KubernetesExecutor

    ###################################
    ## ENVIRONMENT VARIABLES SETUP
    ###################################
    config:
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
      AIRFLOW__CORE__LOAD_EXAMPLES: "False"
      CONNECTION_CHECK_MAX_COUNT: 10

    extraEnv:
      - name: AIRFLOW__WEBSERVER__BASE_URL
        valueFrom:
          configMapKeyRef:
            name: airflow-v2-settings
            key: webserver_url
      ## [SPEC] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-fernet-key.md
      - name: AIRFLOW__CORE__FERNET_KEY
        valueFrom:
          secretKeyRef:
            name: airflow-v2-external-secrets
            key: fernetKey
      ## [SPEC] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-webserver-secret-key.md
      - name: AIRFLOW__WEBSERVER__SECRET_KEY
        valueFrom:
          secretKeyRef:
            name: airflow-v2-external-secrets
            key: webKey

    ###################################
    ## REMOTE RESOURCES ACCESS
    ###################################
    externalDatabase:
      type: mysql
      host: "xxxxxx" 
      port: 3306
      database: airflow
      userSecret: airflow-v2-external-secrets
      userSecretKey: "db_username"
      passwordSecret: airflow-v2-external-secrets
      passwordSecretKey: "db_password"

    serviceAccount:
      create: true
      annotations:
        eks.amazonaws.com/role-arn: xxxxx

    ###################################
    ## AUTHENTICATION
    ###################################
    ## a list of users to create
    ## THIS WILL NEED TO BE DISABLED ONCE RBAC IS ENABLED
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/airflow-users.md
    users:
      - username: admin
        password: admin
        role: Admin
        email: admin@example.com
        firstName: admin
        lastName: admin

  ###################################
  ## COMPONENTS
  ###################################
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources:
    requests:
      memory: "1800Mi"
      cpu: "82m"
    values:
      memory: "1800Mi"
      cpu: "82m"

  scheduler:
    logCleanup:
      enabled: true
    securityContext:
      fsGroup: 65534
    replicas: 1
    resources:
      requests:
        memory: "1300Mi"
        cpu: "1600m"
      limits:
        memory: "1300Mi"
        cpu: "1600m"
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
    livenessProbe:
      enabled: true
      ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
      taskCreationCheck:
        enabled: false
        thresholdSeconds: 300

  web:
    securityContext:
      fsGroup: 65534
    replicas: 1
    resources:
      requests:
        memory: "3900Mi"
        cpu: "950m"
      limits:
        memory: "3900Mi"
        cpu: "950m"
    service:
      type: ClusterIP
      externalPort: 8080

  logs:
    path: /opt/airflow/logs
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-persistence.md
    persistence:
      enabled: false

  dags:
    path: /opt/airflow/dags
    persistence:
      enabled: false
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/load-dag-definitions.md
    gitSync:
      repo: "git@github.com:Sanyaku/analytics-airflow.git"
      branch: "main"
      revision: "HEAD"
      syncWait: 60 ## number of seconds to wait between syncs
      sshSecret: airflow-git-secrets
      sshSecretKey: "id_rsa"
      sshKnownHosts: ""

  ###################################
  ## Disabled Airflow Chart Services
  ###################################
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-cleanup.md
  logCleanup:
    enabled: false
  workers:
    enabled: false
  triggerer:
    enabled: false
  flower:
    enabled: false
  pgbouncer:
    enabled: false
  postgresql:
    enabled: false
  redis:
    enabled: false
  extraVolumeMounts: []
  extraVolumes: []

  ingress:
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/ingress.md
    enabled: false

The text was updated successfully, but these errors were encountered:

thesuperzapper · 2022-06-14T21:47:00Z

@asosnovsky-sumologic this issue is caused by airflow 2.3.X moving an import (which broke our snyc-users script).

We have fixed this issue in PR #592, but that PR is not part of a release yet. I plan to include the fix in chart version 8.6.1 (which I am planning to release ASAP, pending one issue related to creating new Redis ARM images).

Follow issue #572 (comment) to get notified about other 2.3.X fixes.

asosnovsky-sumologic · 2022-06-15T14:37:04Z

@thesuperzapper thank you I have downgraded for now, should have checked your compatibility table better before using 2.3.x

thesuperzapper · 2022-06-15T22:49:31Z

@asosnovsky-sumologic I really should have pushed 8.6.1 out already, I am sorry for taking so long (it's been ready for ages, I have just been hesitant to cut, as I really don't want to "encourage" upgrading yet). However, I will make it my mission to cut 8.6.1 in the next 12 hours.

Airflow 2.3.X really should have worked without needing chart changes (if airflow had not made unnecessarily breaking changes), and as of 2.3.2 I have actually worked with airflow to fix most of the issues upstream (but the sync-users will require a change on our end).

But in general, for production usage of airflow, I would recommend waiting a bit for any new minor release to stabilize before using it in anything critical (Airflow has a tendency to make breaking changes with every release, even patch ones).

thesuperzapper · 2022-06-22T09:48:26Z

@asosnovsky-sumologic the long-belated 8.6.1 is now out, and includes support for Airflow 2.3!

softestplease · 2022-09-20T09:49:26Z

Hello all,
I am aware this issue is fixed for Airflow 2.3.X with Helm chart 8.6.1
However, I am still getting this error message:
sqlalchemy.exc.InvalidRequestError: Multiple classes found for path "Permission" in the registry of this declarative base. Please use a fully module-qualified path.
I am using Airflow 2.3.3 with 8.6.1
Any help/feedback much appreciated
TIA

asosnovsky-sumologic added the kind/bug kind - things not working properly label Jun 14, 2022

thesuperzapper added the status/duplicate status - this issue or pull request already exists label Jun 14, 2022

thesuperzapper mentioned this issue Jun 14, 2022

fix: some breaking changes from airflow 2.3.0 #592

Merged

6 tasks

thesuperzapper removed the status/duplicate status - this issue or pull request already exists label Jun 14, 2022

thesuperzapper added this to the airflow-8.6.1 milestone Jun 14, 2022

thesuperzapper added this to Unsorted in Issue Triage and PR Tracking via automation Jun 14, 2022

thesuperzapper changed the title ~~sync-users sqlalchemy.exc.InvalidRequestError~~ airflow 2.3.X breaks sync-users script Jun 14, 2022

thesuperzapper moved this from Unsorted to Triage | Work Started in Issue Triage and PR Tracking Jun 14, 2022

asosnovsky-sumologic closed this as completed Jun 15, 2022

Issue Triage and PR Tracking automation moved this from Triage | Work Started to Done Jun 15, 2022

arundeep78 mentioned this issue Mar 7, 2023

sync-users scripts fails with airflow 2.3.x and 2.4.x with 8.6.1 chart version #703

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airflow 2.3.X breaks sync-users script #607

airflow 2.3.X breaks sync-users script #607

asosnovsky-sumologic commented Jun 14, 2022

thesuperzapper commented Jun 14, 2022

asosnovsky-sumologic commented Jun 15, 2022

thesuperzapper commented Jun 15, 2022

thesuperzapper commented Jun 22, 2022

softestplease commented Sep 20, 2022

airflow 2.3.X breaks sync-users script #607

airflow 2.3.X breaks sync-users script #607

Comments

asosnovsky-sumologic commented Jun 14, 2022

Checks

Chart Version

Kubernetes Version

Helm Version

Description

Relevant Logs

Custom Helm Values

thesuperzapper commented Jun 14, 2022

asosnovsky-sumologic commented Jun 15, 2022

thesuperzapper commented Jun 15, 2022

thesuperzapper commented Jun 22, 2022

softestplease commented Sep 20, 2022