AzureContainerInstancesOperator uses a deprecated authentication flow in apache-airflow-providers-microsoft-azure #34749
Description
Apache Airflow version
2.7.1
What happened
My previous bug report (#34736) was closed, but upon further investigation I am pretty sure that the fix should be handled in this repository. I have added the solution in a separate section below.
We currently have to hardcode our version of apache-airflow-providers-microsoft-azure to 6.2.4 in order for AzureContainerInstancesOperator to work as expected. If we bump the version to 6.3.0 or higher we get this error during execution:
[2023-10-03, 19:07:37 EEST] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 196, in execute
if self._ci_hook.exists(self.resource_group, self.name):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 198, in exists
for container in self.connection.container_groups.list_by_resource_group(resource_group):
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 123, in __next__
return next(self._page_iterator)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 75, in __next__
self._response = self._get_next(self.continuation_token)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/containerinstance/operations/_container_groups_operations.py", line 182, in get_next
pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 230, in run
return first_node.send(pipeline_request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
response = self.next.send(request)
[Previous line repeated 2 more times]
File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
self.on_request(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
self._token = self._credential.get_token(*self._scopes)
AttributeError: 'ServicePrincipalCredentials' object has no attribute 'get_token'What you think should happen instead
The hook in container_instance.py should use the azure.identity package instead of azure.common.credentials. Link to code
Microsoft posted deprecation warnings about using that package since early 2021 - azure.common.
Here is a related issue, where the solution is to start using azure.identity instead.
How to reproduce
Install airflow 2.7.1 and try creating a DAG with AzureContainerInstancesOperator and run it.
Operating System
Windows 10
Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure=6.3.0 or 7.0.0
Deployment
Docker-Compose
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct