Microsoft Azure blob storage doesn't allow managed identity connection in ObjectStore (but does in hook) #38746
Description
Apache Airflow Provider(s)
microsoft-azure
Versions of Apache Airflow Providers
No response
Apache Airflow version
9.0.1
Operating System
linux (standard airflow slim images extended with custom code running on kubernetes)
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
What happened
We use airflow connections to azure blob storage using managed identity. We would like to use the ObjectStore but this doesn't currently work.
What you think should happen instead
I propose to update the provider code to pass the required arguments to adlfs.AzureBlobFileSystem. The aim is consistent handling of the airflow connection data between WasbHook and the ObjectStoragePath for "abfs://" paths.
How to reproduce
For an existing azure blob storage container, add to airflow a connection of type "wasb" containing extra fields "account_name", "managed_identity_client_id", "workload_identity_tenant_id".
Compare results of:
WasbHook(conn_id).get_blobs_list(container_name)
to
ObjectStoragePath(f"abfs://{container_name}/", conn_id=conn_id).iterdir()
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct