High volume of API requests with AWS Secrets Manager backend

[john-jac]

Apache Airflow version:
1.10.12

Kubernetes version (if you are using kubernetes) (use kubectl version):

n/a

Environment:

• Cloud provider or hardware configuration: MWAA
• OS (e.g. from /etc/os-release): Amazon Linux
• Kernel (e.g. uname -a): x86_64 GNU/Linux
• Install tools:
• Others:

What happened:

When using AWS Secrets Manager backend there is a high volume of API request from Airflow to the secrets backend, regardless of whether DAGs are running or not

What you expected to happen:

Traffic to secrets backend should be limited to what's needed

How to reproduce it:
https://airflow.apache.org/docs/apache-airflow/1.10.12/howto/use-alternative-secrets-backend.html?highlight=backend#aws-secrets-manager-backend

Anything else we need to know:

n/a

[jedcunningham]

Can you try with only a simple DAG in your instance? e.g:
https://github.com/apache/airflow/blob/master/airflow/example_dags/example_latest_only.py

I wonder if it is Airflow itself or something in your DAGs (e.g. variables being used during DAG parsing).

[kaxil]

Also please test it with 2.x -- Only critical fixes would be done on 1.10.x and that too will end next month.

https://github.com/apache/airflow/blob/master/dev/PROJECT_GUIDELINES.md#support-for-airflow-110x-releases
https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html#support-for-airflow-1-10-x-releases

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.