ADLS Gen2 VFS : Confirm DFS Endpoint Usage & 403 Error

[in-da]

I get a 403 AuthorizationFailure when browsing an ADLS location via Apache Hop's ADLS Gen2 VFS connector using Shared Key authentication. (Apache Hop version : 2.17)
The storage account only has a Private Endpoint for the blob sub-resource.
I suspect Hop's ADLS Gen2 connector uses the dfs endpoint which, in my setup, has no Private Endpoint.

To be sure about next steps, I need your support to confirm the following points:

1. Does the ADLS Gen2 VFS connector use *.dfs.core.windows.net?
2. Is there an option to use the Blob API instead?

Below You will find the error message I got:
FileSystemException: Could not determine the type of file "ldz:///data/raw". Root cause: DataLakeStorageException: Status code 403 "AuthorizationFailure" - "This request is not authorized to perform this operation."

Thank you all for your support.

[hansva]

I can confirm that we are using https://%s.dfs.core.windows.net but you can set your own endpoint and that should be used, have you tried this?

[in-da]

Yes already did, same issue, error message below:
Error browsing to location: 'ldz:///data' FileSystemException: Could not determine the type of file "ldz:///data". Root cause: DataLakeStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call. If you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call. Please remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII. Status code 403, "{"error":{"code":"AuthorizationFailure","message":"This request is not authorized to perform this operation........."}}"

To be sure, I made a test with another client and it is working fine (I can say no network issues to get/send the data through the blob endpoint).
Any additional setup is required?

[hansva]

We only support hierarchical namespaces and not blob-only.

[in-da]

Already enabled! I got the same issue with Hop 2.11 and 2.17.

[hansva]

You'll need to create a private endpoint for DFS. That's what I wanted to say, we don't support the Blob service only, we need the Data Lake Storage system with the implementation we have.

[in-da]

Undersood! Thank you for the input.
I will do so and get back to you soon.

[in-da]

@hansva I confirm that it is working aftrer having both blob and dfs PE activated.
Thank you for you help.

[in-da]

@hansva Do you please have an idea if Apache Hop does support the usage of Azure Shared Access Signature?
If it is the case can you please share some insights?
Thank you for your help.

[hansva]

I think that would require us to add an extra authentication mechanism. Create a feature request for this so someone can investigate at some point in time