ADLS Blobfuse2 SPN Token generated with wrong resource/audience, auth-resource in config not registered

[red324]

Which version of blobfuse was used?

blobfuse2 version 2.0.3

Which OS distribution and version are you using?

Tested with Ubuntu 20.04.4 LTS and Ubuntu 22.04

If relevant, please share your mount command.

sudo blobfuse2 mount mycontainer --tmp-path=tmp --config-file=config.yml

What was the issue encountered?

I am getting authorization issues when using blobfuse2 and adls.

My config file (config.yml) is as follows:

allow-other: true
 
logging:
  type: base
  level: log_debug
  file-path: ~/blobfuse.log

components:
  - libfuse
  - file_cache
  - attr_cache
  - azstorage
 
libfuse:
  default-permission: 0777
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240
 
azstorage:
  type: adls
  account-name: <account_name>
  mode: spn
  container: <container_name>
  # service principal authentication
  tenantid: <tenant_id>
  clientid: <client_id>
  clientsecret: <client_secret>
  auth-resource: https://storage.azure.com

The mounting seems to succeed, but when I try to get to a directory in the file system and list the contents, I receive the error:

ls: reading directory '.': Input/output error

Looking at the blobfuse logs, I see:

Note: The Authorization is redacted so I am just guessing as to which token is being used based on the results

Earlier in the logs, I can see the tokens that are being created. The token for the blob service uri https://xxx.blob.core.windows.net/works when using postman to make calls to the directory. The token generated using the .dfs.core.windows.net does NOT work and produces this error. When I generate a token with: https://storage.azure.com as the resource, I am also able to successfully make the API call.

Taking the tokens directly from the blobfuse logs.. I can authenticate properly with the .blob.core.windows.net token:

but not the .dfs.core.windows.net token:

My service principal has Reader RBAC access to the Storage Account, and has R-X ACL access to container and RWX access on the directory I am trying to ls in. It is not possible to grant further RBAC access due to the way access partitioning is set up in the organization's data lake, so we use ACLs for authorization. I am able to ls on the container, but not on the directory underneath the container, I am also able to cd into the directory from the container.

According to the way ACLs vs. RBAC work in the Data Lake, it would appear that I should not need the storage blob data contributor role since I have r-x access at container and rwx at the directory level: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control-model#how-permissions-are-evaluated

I have tested mounting this same storage container using blobfuse version 1 and it does work as expected. I checked the token it is using and the resource appears to be: https://storage.azure.com.

So I tried to set the auth-resource in the config file for blobfuse2 to equal https://storage.azure.com so that the correct token will be generated and used, but I am still receiving access issues shown above.

I am unsure of how to 100% confirm the token that is being used for the GET call since the token is redacted, but it would seem to me that regardless of what I set as the auth-resource, it is using the https://xxx.dfs.core.windows.net endpoint since I am getting permission errors.

Conclusion:

The proper token auth resources appear to be either https://storage.azure.com or https://<account>.blob.core.windows.net as stated here: https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory#microsoft-authentication-library-msal and here: https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-azure-active-directory#use-oauth-access-tokens-for-authentication

I think there is a bug in V2 where the token is being generated with the https://account_name.dfs.core.windows.net uri as the audience/resource (and/or also not taking into account the auth-resource in the config file)

Have you found a mitigation/solution?

No

Please share logs if available.

See screen shots above. I can share more as needed.

This is a follow up from discussion: #1153

[vibhansa-msft]

In case account is configured as ADLS in yaml config file, you will see tokens being generated for both .dfs and .blob endpoints as some of the operations that blobfuse performs goes over .blob endpoint as well. If you beleive .dfs token is not needed then you can skip "type: adls" config but that might restrict some other operations on your storage account.
As far as auth_resource config goes, blobfuse does not support that as of now. While it worked for you in v1, was v1 configured to consider account as ADLS account or blob account (.blob end point by default).

[red324]

Hello @vibhansa-msft, while working with v1, yes I did mount the directory with the adls option set to true. Here is the command I was running with V1: sudo blobfuse container --tmp-path=tmp --config-file=fuse_connection.cfg -o allow_other --use-adls=true

Utilizing V1 with the use-adls=true option, I was able to traverse through the directories and list the contents of the directories that the service principal has ACL access to.

Attempting to use V2 without the type: adls set to true does not work. I don't think the directories are being registered properly as it says that directories which exist do not exist.

It is not so much that I think the .dfs. endpoint is not necessary to be used, but rather that the token generated with .dfs. as the audience does not work properly. I think the audience should be: https://storage.azure.com. I can see that blobfuse V1 uses:

I am able to get Blobfuse V2 working with other ADLS storage accounts, but not all of them so it seems inconsistent across the board.

Is there a plan to add the auth_resource support?

[vibhansa-msft]

Can you point where do you see this "aud": "htps://storage.azure.com".
V1 was built using older c++ sdk while v2 works with go-sdk which is on higher version compared to the c++ one. Hence there might be changes on how the token is retrieved or used.

[vibhansa-msft]

Will it be possible for you to checkout " vibhansa/v2/spn_resource_fix" branch and try it out. I have modified the resource url in this one to point to stroage.azure.com/.default.

[red324]

@vibhansa-msft initial testing, just checking out the ibhansa/v2/spn_resource_fix branch and changing directory to the desired directory it appears to be working now. I am able to cd into the directory and ls contents. Also able to do other actions like edit text files etc.

[vibhansa-msft]

I am doing testing from my end as well, but the problem you pointed out was not hit in any of our regular testing. Can you validate with a non-HNS account as well just to validate there is no collaterals.

[vibhansa-msft]

@red324 Can you validate some parts on your end

• We verified that giving "Reader" role to SPN alone does not work and listing of directories goes into issues.
• Instead use "Storage blob data reader" role for SPN, this will give your SPN read access across your account.
• If you do not wish to have your SPN access across the account then do not assign any role to the SPN and instead use ACLs. In this mode each directory that you wish to iterate through should have 'r-x' permissions set for the SPN.
• If you wish to read/write a file"/A/B/C/d.txt" then you need "--x" permissions on "A", "A/B', "A/B/C" and "rwx" on "A/B/C/d.txt"
• Once proper RBAC or ACLs are set, token scope related issue shall not be hit as well (in our testing this is not creating any challenge on HNS enabled account).
  Kindly correct the RBAC/ACLs and let us know if that resolves your issue.