Variables are getting duplicated on upgrading airflow

[vghar-bh]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

variables are getting duplicated on upgrading airflow to 2.5.3 from 2.4.1.
after upgrade when the airflow-sync-variables pod runs its not finding the existing variables and then adding it again.

logs from airflow-sync-variables:

/home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[�[34m2023-07-09 05:59:12,443�[0m] {�[34msync_variables.py:�[0m357} INFO�[0m - BEGIN: airflow variables sync�[0m
[�[34m2023-07-09 05:59:14,521�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=TEMP-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,534�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=TEMP-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,538�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=PERSPECTIVE-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,546�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=PERSPECTIVE-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,550�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=TAMR_INPUT_BUCKET_NAME is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,553�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=TAMR_INPUT_BUCKET_NAME was successfully added.�[0m
[�[34m2023-07-09 05:59:14,562�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=TAMR_OUTPUT_BUCKET_NAME is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,573�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=TAMR_OUTPUT_BUCKET_NAME was successfully added.�[0m
[�[34m2023-07-09 05:59:14,577�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=RAW-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,580�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=RAW-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,585�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=EXTRACTED-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,589�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=EXTRACTED-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,592�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=CATEGORIZED-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,596�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=CATEGORIZED-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,600�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=STRUCTURED-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,603�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=STRUCTURED-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,608�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=STANDARD-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,612�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=STANDARD-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,616�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=NPD_DATE is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,619�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=NPD_DATE was successfully added.�[0m
[�[34m2023-07-09 05:59:14,623�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=SPARK-KUBERNETES-ANNOTATION is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,631�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=SPARK-KUBERNETES-ANNOTATION was successfully added.�[0m
[�[34m2023-07-09 05:59:14,635�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=SPARK-DRIVER-MEMORY is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,638�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=SPARK-DRIVER-MEMORY was successfully added.�[0m
[�[34m2023-07-09 05:59:14,641�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=SPARK-EXECUTOR-MEMORY is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,645�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=SPARK-EXECUTOR-MEMORY was successfully added.�[0m
[�[34m2023-07-09 05:59:14,648�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=SPARK-EXECUTOR-CORES is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,653�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=SPARK-EXECUTOR-CORES was successfully added.�[0m
[�[34m2023-07-09 05:59:14,656�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=SPARK-NUM-EXECUTORS is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,659�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=SPARK-NUM-EXECUTORS was successfully added.�[0m
[�[34m2023-07-09 05:59:14,662�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=DATACURATION-OUTPUT-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,665�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=DATACURATION-OUTPUT-BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,671�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=ENT_NTFN_USER is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,676�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=ENT_NTFN_USER was successfully added.�[0m
[�[34m2023-07-09 05:59:14,694�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=ENT_NTFN_CONSUMER_COUNT is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,698�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=ENT_NTFN_CONSUMER_COUNT was successfully added.�[0m
[�[34m2023-07-09 05:59:14,710�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=MYID_BASE_URL is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,714�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=MYID_BASE_URL was successfully added.�[0m
[�[34m2023-07-09 05:59:14,727�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=DATALAKE_BUCKET is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,731�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=DATALAKE_BUCKET was successfully added.�[0m
[�[34m2023-07-09 05:59:14,735�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=BITPULSE_WEBBITS_DATALAKE_PATH is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,744�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=BITPULSE_WEBBITS_DATALAKE_PATH was successfully added.�[0m
[�[34m2023-07-09 05:59:14,749�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=UOM_DOCKER_IMAGE is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,753�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=UOM_DOCKER_IMAGE was successfully added.�[0m
[�[34m2023-07-09 05:59:14,757�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=UOM_NAMESPACE is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,760�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=UOM_NAMESPACE was successfully added.�[0m
[�[34m2023-07-09 05:59:14,765�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=UOM_MAPPING_FILE_PATH is missing, adding...�[0m
[�[34m2023-07-09 05:59:14,768�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=UOM_MAPPING_FILE_PATH was successfully added.�[0m
[�[34m2023-07-09 05:59:14,774�[0m] {�[34msync_variables.py:�[0m360} INFO�[0m - END: airflow variables sync�[0m
[�[34m2023-07-09 06:00:14,864�[0m] {�[34msync_variables.py:�[0m357} INFO�[0m - BEGIN: airflow variables sync�[0m
[�[34m2023-07-09 06:00:14,917�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=STRUCTURED-BUCKET is missing, adding...�[0m
[�[34m2023-07-09 06:00:14,921�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=STRUCTURED-BUCKET was successfully added.�[0m
[�[34m2023-07-09 06:00:14,952�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=SPARK-EXECUTOR-CORES is missing, adding...�[0m
[�[34m2023-07-09 06:00:14,958�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=SPARK-EXECUTOR-CORES was successfully added.�[0m
[�[34m2023-07-09 06:00:14,990�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=ENT_NTFN_CONSUMER_COUNT is missing, adding...�[0m
[�[34m2023-07-09 06:00:14,994�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=ENT_NTFN_CONSUMER_COUNT was successfully added.�[0m
[�[34m2023-07-09 06:00:15,013�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=MYID_REDIRECT_BASE_URL is missing, adding...�[0m
[�[34m2023-07-09 06:00:15,016�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=MYID_REDIRECT_BASE_URL was successfully added.�[0m
[�[34m2023-07-09 06:00:15,035�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=UOM_DOCKER_IMAGE is missing, adding...�[0m
[�[34m2023-07-09 06:00:15,040�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=UOM_DOCKER_IMAGE was successfully added.�[0m
[�[34m2023-07-09 06:00:15,044�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=UOM_NAMESPACE is missing, adding...�[0m
[�[34m2023-07-09 06:00:15,048�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=UOM_NAMESPACE was successfully added.�[0m
[�[34m2023-07-09 06:00:15,058�[0m] {�[34msync_variables.py:�[0m360} INFO�[0m - END: airflow variables sync�[0m
[�[34m2023-07-09 06:01:15,132�[0m] {�[34msync_variables.py:�[0m357} INFO�[0m - BEGIN: airflow variables sync�[0m
[�[34m2023-07-09 06:01:15,256�[0m] {�[34msync_variables.py:�[0m336} INFO�[0m - Variable=ENT_NTFN_CONSUMER_COUNT is missing, adding...�[0m
[�[34m2023-07-09 06:01:15,260�[0m] {�[34msync_variables.py:�[0m348} INFO�[0m - Variable=ENT_NTFN_CONSUMER_COUNT was successfully added.�[0m
[�[34m2023-07-09 06:01:15,323�[0m] {�[34msync_variables.py:�[0m360} INFO�[0m - END: airflow variables sync�[0m
[�[34m2023-07-09 06:02:15,419�[0m] {�[34msync_variables.py:�[0m357} INFO�[0m - BEGIN: airflow variables sync�[0m

What you think should happen instead

After upgrade when the airflow-sync-variables pod runs it should finding the existing variables and skip creating it again.

How to reproduce

downgrade the airflow to 2.4.1 where its working
upgrade it to airflow 2.5.3 once we sync the airflow-sync-variables pod the variables will be duplicated in the UI and in database variable table.

Operating System

N/A

Versions of Apache Airflow Providers

on airflow 2.4.1 before upgrade

apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-apache-livy 2.2.3
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-cncf-kubernetes 4.4.0
apache-airflow-providers-common-sql 1.2.0
apache-airflow-providers-docker 3.2.0
apache-airflow-providers-elasticsearch 4.2.1
apache-airflow-providers-ftp 3.1.0
apache-airflow-providers-google 8.3.0
apache-airflow-providers-grpc 3.0.0
apache-airflow-providers-hashicorp 3.1.0
apache-airflow-providers-http 4.0.0
apache-airflow-providers-imap 3.0.0
apache-airflow-providers-microsoft-azure 4.3.0
apache-airflow-providers-mysql 3.2.1
apache-airflow-providers-odbc 3.1.2
apache-airflow-providers-postgres 5.2.2
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-sendgrid 3.0.0
apache-airflow-providers-sftp 4.1.0
apache-airflow-providers-slack 5.1.0
apache-airflow-providers-sqlite 3.2.1
apache-airflow-providers-ssh 3.2.0

on 2.5.3 after upgrade

apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-apache-livy 2.2.3
apache-airflow-providers-celery 3.2.0
apache-airflow-providers-cncf-kubernetes 7.0.0
apache-airflow-providers-common-sql 1.5.1
apache-airflow-providers-docker 3.7.0
apache-airflow-providers-elasticsearch 4.5.0
apache-airflow-providers-ftp 3.4.1
apache-airflow-providers-google 10.1.1
apache-airflow-providers-grpc 3.2.0
apache-airflow-providers-hashicorp 3.4.0
apache-airflow-providers-http 4.4.1
apache-airflow-providers-imap 3.2.1
apache-airflow-providers-microsoft-azure 6.1.1
apache-airflow-providers-odbc 3.3.0
apache-airflow-providers-postgres 5.5.0
apache-airflow-providers-redis 3.2.0
apache-airflow-providers-sendgrid 3.2.0
apache-airflow-providers-sftp 4.3.0
apache-airflow-providers-slack 7.3.0
apache-airflow-providers-snowflake 4.1.0
apache-airflow-providers-sqlite 3.4.1
apache-airflow-providers-ssh 3.7.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow version 2.5.3

currently working with helm charts from Airflow Helm Chart (User Community)
https://artifacthub.io/packages/helm/airflow-helm/airflow/8.7.0

using Argocd as GitOps CD tool for Kubernetes.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct