Hook for managing directories and files in Azure Data Lake Storage Gen2 #28262

bharanidharan14 · 2022-12-09T16:44:12Z

Created hook for supporting ADLS gen2, which uses the WASB connection and connects to ADLS gen2 storage

Relates to #28223

tatiana · 2022-12-12T23:24:28Z

Relates to #28223

bharanidharan14 · 2022-12-13T15:25:55Z

@tatiana @luanmorenomaciel Need your review on this PR.

tatiana · 2022-12-13T00:12:09Z

airflow/providers/microsoft/azure/hooks/adls.py

+from airflow.hooks.base import BaseHook
+
+
+class AzureDataLakeStorageV2(BaseHook):


Suggested change

class AzureDataLakeStorageV2(BaseHook):

class AzureDataLakeStorageV2Hook(BaseHook):

Now I changed the Hook name now.

I'm not sure about this.
eventually v2 will be the standard and on that day it would be redundant to mention v2.

I see that all current v1 services are listed as AzueDataLakeStorage*
WDYT about having the new ones as Adls*?

Yes, the current v1 services are listed as AzueDataLakeStorage. Now it is AzureDataLakeStorageClient can it be AzureDataLakeStorageClientHook. WDYT?

since we don't have hook for adls it can be anything we want.
i suggested the pattern: Adls* so:

Current V1 Suggested V2

- AdlsClientHook

ADLSDeleteOperator AdlsDeleteOperator

ADLSListOperator AdlsListOperator

Alternatively we can also do:

microsoft/azure/hooks/adsl_v2.py microsoft/azure/operators/adsl_v2.py

then class names can have the same names as v1 because they are not in the same file
and once v1 is removed we can change the files names which is easier change then changing classes names.

tatiana · 2022-12-13T00:21:57Z

airflow/providers/microsoft/azure/hooks/adls.py

+            return DataLakeServiceClient(
+                account_url=f"https://{conn.login}.dfs.core.windows.net", credential=token_credential, **extra
+            )
+        credential = conn.password


Would it make sense to do something similar to:
https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/hooks/wasb.py#L202?

I wonder how much in common the Wasb & the AzureDataLakeStorageV2 implementation have - and if it would make sense to have a BaseAzureHook

@tatiana I am using WASB connection details to connect to ADLS gen2 storage account, apart the connection details nothing is similar I think

That's a shame, @bharanidharan14 - I didn't realise things were so disconnected in Azure

@tatiana ADLS gen2 is using different protocols and APIs and different client-type principles to connect. Currently, I have created separate connection types and connection details.

eladkal · 2022-12-15T11:16:44Z

airflow/providers/microsoft/azure/hooks/adls.py

+from airflow.providers.microsoft.azure.hooks.wasb import WasbHook
+
+
+class AzureDataLakeStorageClient(WasbHook):


#28223 description states that wasb is legacy if so what is the motivation to inhert from wsab for adsl gen2?

Only for connection details, I thought of using these WASB connection details. should I create a new connection ?

If wasb is legacy and we expect it to be removed some day then I don't think we utlize it for other services
but I'm not Azure expert - I leave this desicion to people who knows the service.

What i do want to have here is docs and specifically connection docs that explains how to make this work.
(I want to avoid questions raised down the road by users who will ask to refactor because we utlize legacy service)

Ok let me add new connection, with docs

@bharanidharan14 @eladkal did we consider having an AzureBaseHook similar to GoogleBaseHook
https://github.com/apache/airflow/blob/main/airflow/providers/google/common/hooks/base_google.py
This way, both AzureDataLakeStorageHook and WasbHook could inherit from it

@tatiana and @mateusoliveiraowshq as per our conversation, implementing the AzureBaseHook would be challenging since Azure unfortunately does not share the same connection client principles (BlobServiceClient & DataLakeServiceClient) between the WASB and ABFS protocol.

Azure Blob Storage - WASB
Azure Data Lake Storage Gen2 - ABFS

What we could do instead of is to have the following hook structure:

airflow.providers.microsoft.azure.hooks.wasb (Already Exists)

airflow.providers.microsoft.azure.hooks.azure_data_lake (Suggested)

Meaning that as per @bharanidharan14 @eladkal conversation we would recommend to inherid the hook from the airflow.providers.microsoft.azure.hooks.azure_data_lake instead of airflow.providers.microsoft.azure.hooks.wasb, that would make more sense since WASB protocol has been marked as legacy hence the, implementation process are different from each other.

@luanmorenomaciel I don't think inheriting from the airflow.providers.microsoft.azure.hooks.azure_data_lake is also not recommended because this ADLS existing hook is gen1 and it's being getting retired from feb. They may stop supporting for that too as well. so its better to inherit from BaseHook. WDYT ?

We should be fine to inherit from BaseHook

@luanmorenomaciel Currently I am inheriting from BaseHook. Can you also review this PR

tatiana · 2022-12-16T15:55:08Z

airflow/providers/microsoft/azure/hooks/adls_v2.py

+from hooks.base import BaseHook
+
+
+class AdlsClientHook(BaseHook):


Would it make sense for this to be defined inside airflow/providers/microsoft/azure/hooks/azure_data_lake?

@bharanidharan14 ^^

Addressed it

bharanidharan14 · 2022-12-20T06:42:16Z

@kaxil Need your review on this PR

kaxil · 2022-12-22T00:13:00Z

Tests are failing

airflow/providers/microsoft/azure/hooks/adls_v2.py

tatiana

@bharanidharan14 we're very close to merging this work.

There is one comment pending: https://github.com/apache/airflow/pull/28262/files#r1056265305
Rebasing and passing tests also seem to be pending.

Fix mypy issues Fix mypy issues Fix mypy issues Fix mypy issues Add adls v2 connection Fix doc Fix doc Moved hook to existing data lake hook file Moved hook test case Moved hook test case Fix doc

ADLS gen2 hook

Fix mypy issues Fix mypy issues Fix mypy issues Fix mypy issues Add adls v2 connection Fix doc Fix doc Moved hook to existing data lake hook file Moved hook test case Moved hook test case Fix doc

…o adls-gen2

boring-cyborg bot added area:providers kind:documentation provider:microsoft-azure Azure-related issues labels Dec 9, 2022

bharanidharan14 force-pushed the adls-gen2 branch 2 times, most recently from 39e6f88 to bfede61 Compare December 9, 2022 16:50

bharanidharan14 force-pushed the adls-gen2 branch from 5f3b883 to bfd98df Compare December 13, 2022 13:21

bharanidharan14 changed the title ~~[WIP]: Hook for managing directories and files in Azure Data Lake Storage Gen2~~ Hook for managing directories and files in Azure Data Lake Storage Gen2 Dec 13, 2022

bharanidharan14 marked this pull request as ready for review December 13, 2022 15:21

tatiana reviewed Dec 14, 2022

View reviewed changes

phanikumv mentioned this pull request Dec 14, 2022

Initial Support for Azure Blob Storage Object astronomer/astro-sdk#1275

Merged

4 tasks

eladkal reviewed Dec 15, 2022

View reviewed changes

bharanidharan14 force-pushed the adls-gen2 branch from 63d5201 to b742318 Compare December 16, 2022 01:32

bharanidharan14 requested a review from mik-laj as a code owner December 16, 2022 04:18

tatiana reviewed Dec 16, 2022

View reviewed changes

bharanidharan14 requested review from tatiana and luanmorenomaciel and removed request for mik-laj, tatiana and luanmorenomaciel December 20, 2022 11:49

kaxil force-pushed the adls-gen2 branch from 74b2e86 to 232be61 Compare December 21, 2022 23:03

tatiana reviewed Dec 23, 2022

View reviewed changes

airflow/providers/microsoft/azure/hooks/adls_v2.py Outdated Show resolved Hide resolved

tatiana reviewed Dec 23, 2022

View reviewed changes

bharanidharan14 force-pushed the adls-gen2 branch from 232be61 to 84211f2 Compare December 27, 2022 09:05

bharanidharan14 force-pushed the adls-gen2 branch from 197f7a7 to ce0ea8d Compare December 29, 2022 02:02

Moved hook to existing data lake hook file

fc15cc6

Fix mypy issues Fix mypy issues Fix mypy issues Fix mypy issues Add adls v2 connection Fix doc Fix doc Moved hook to existing data lake hook file Moved hook test case Moved hook test case Fix doc

bharanidharan14 force-pushed the adls-gen2 branch from ce0ea8d to fc15cc6 Compare December 29, 2022 02:04

bharanidharan14 added 3 commits December 29, 2022 07:34

ADLS gen2 hook

5725a22

ADLS gen2 hook

Add Test case

bb6f8fb

Moved hook to existing data lake hook file

e4fe4c4

Fix mypy issues Fix mypy issues Fix mypy issues Fix mypy issues Add adls v2 connection Fix doc Fix doc Moved hook to existing data lake hook file Moved hook test case Moved hook test case Fix doc

bharanidharan14 force-pushed the adls-gen2 branch from fc15cc6 to e4fe4c4 Compare December 29, 2022 02:05

Merge branch 'adls-gen2' of https://github.com/astronomer/airflow int…

c260594

…o adls-gen2

bharanidharan14 force-pushed the adls-gen2 branch from 74e55b7 to c260594 Compare December 29, 2022 05:49

bharanidharan14 requested review from tatiana, luanmorenomaciel, kaxil and eladkal and removed request for luanmorenomaciel, tatiana and kaxil December 29, 2022 08:03

kaxil approved these changes Jan 5, 2023

View reviewed changes

kaxil merged commit ad7f8e0 into apache:main Jan 5, 2023

kaxil deleted the adls-gen2 branch January 5, 2023 14:03

tatiana mentioned this pull request Jan 10, 2023

Add Support to ABFS Azure Data Lake Storage Gen2 (ADLS2) Protocol on Microsoft.Azure.Hooks #28223

Closed

2 tasks

eladkal mentioned this pull request Jan 14, 2023

Status of testing Providers that were prepared on January 14, 2023 #28938

Closed

37 tasks

rajaths010494 mentioned this pull request Feb 13, 2023

ABFS support OSS Azure provider astronomer/astronomer-providers#799

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hook for managing directories and files in Azure Data Lake Storage Gen2 #28262

Hook for managing directories and files in Azure Data Lake Storage Gen2 #28262

bharanidharan14 commented Dec 9, 2022 •

edited

tatiana commented Dec 12, 2022

bharanidharan14 commented Dec 13, 2022

tatiana Dec 13, 2022

bharanidharan14 Dec 15, 2022

eladkal Dec 15, 2022

bharanidharan14 Dec 15, 2022

eladkal Dec 15, 2022

eladkal Dec 15, 2022 •

edited

tatiana Dec 13, 2022

bharanidharan14 Dec 15, 2022

tatiana Dec 16, 2022

bharanidharan14 Dec 19, 2022 •

edited

eladkal Dec 15, 2022

bharanidharan14 Dec 15, 2022 •

edited

eladkal Dec 15, 2022

bharanidharan14 Dec 15, 2022 •

edited

tatiana Dec 16, 2022

luanmorenomaciel Dec 16, 2022

bharanidharan14 Dec 19, 2022

luanmorenomaciel Dec 20, 2022

bharanidharan14 Dec 20, 2022

tatiana Dec 16, 2022

kaxil Dec 21, 2022

bharanidharan14 Dec 29, 2022

bharanidharan14 commented Dec 20, 2022

kaxil commented Dec 22, 2022

tatiana left a comment

		from airflow.hooks.base import BaseHook


		class AzureDataLakeStorageV2(BaseHook):

	class AzureDataLakeStorageV2(BaseHook):
	class AzureDataLakeStorageV2Hook(BaseHook):

Current V1	Suggested V2
-	AdlsClientHook
ADLSDeleteOperator	AdlsDeleteOperator
ADLSListOperator	AdlsListOperator

		from airflow.providers.microsoft.azure.hooks.wasb import WasbHook


		class AzureDataLakeStorageClient(WasbHook):

		from hooks.base import BaseHook


		class AdlsClientHook(BaseHook):

Hook for managing directories and files in Azure Data Lake Storage Gen2 #28262

Hook for managing directories and files in Azure Data Lake Storage Gen2 #28262

Conversation

bharanidharan14 commented Dec 9, 2022 • edited

tatiana commented Dec 12, 2022

bharanidharan14 commented Dec 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eladkal Dec 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharanidharan14 Dec 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharanidharan14 Dec 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharanidharan14 Dec 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharanidharan14 commented Dec 20, 2022

kaxil commented Dec 22, 2022

tatiana left a comment

Choose a reason for hiding this comment

bharanidharan14 commented Dec 9, 2022 •

edited

eladkal Dec 15, 2022 •

edited

bharanidharan14 Dec 19, 2022 •

edited

bharanidharan14 Dec 15, 2022 •

edited

bharanidharan14 Dec 15, 2022 •

edited