Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]

[fabio-s-franco]

Which version of blobfuse was used?

version: 2.1.2

Which OS distribution and version are you using?

Ubuntu 22.04.3 LTS 5.15.0-1054-azure

If relevant, please share your mount command.

blobfuse2 test --log-level=LOG_DEBUG --container-name=mycont --allow-other --tmp-path=/tmp/blobfuse --file-cache-timeout-in-seconds=120 --virtual-directory=true

AZURE_STORAGE_ACCOUNT_TYPE=adls
AZURE_STORAGE_AUTH_TYPE=

What was the issue encountered?

Testing from the node agent in AKS. I don't know what else to do. Tried everything.
The storage account is only accessible via private endpoint. I can reach it:

nc -zv myaccount.blob.core.windows.net 443
Warning: inverse host lookup failed for 10.8.0.5: Unknown host
myaccount.blob.core.windows.net [10.8.0.5] 443 (https) open

Can't get anything other than:
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]

I am using managed blob driver, tried the open source, tried helm. cleaned everything. No dice.

Have you found a mitigation/solution?

No

Please share logs if available.

There are no logs, at least where it is documented they would be.

[souravgupta-msft]

Hi @fabio-s-franco, you can check the logs in the node in /var/log/blobfuse2.log.
What is the auth mode you are using?

[fabio-s-franco]

Hi @souravgupta-msft , thanks for the reply. I tried all auth modes. They all present the same behavior. About the log, there is nothing there.
At this point I am convinced it has something to do with the access via private endpoint. The error occurs only when the storage account has public network access disabled. I can access the storage account through az cli from the same source, but not with blobfuse when public network access is disabled (and firewalled to the subnets only). It seems it doesn't try to resolve the DNS from the network (via private DNS zone) and instead always tries to use the public routing.

[souravgupta-msft]

If you are using private endpoint for your storage account, you have to specify that. Please set this environment variable, AZURE_STORAGE_BLOB_ENDPOINT to the private endpoint.

[fabio-s-franco]

# echo $AZURE_STORAGE_BLOB_ENDPOINT 
***********.blob.core.windows.net

Same error, no change

[fabio-s-franco]

I was finally able to figure out the issue. I found the logfile in another location. In my case the node agent (CSI Blob Driver) log file was located on the path /root/.blobfuse2/blobfuse2.log.

Because I was using adsl, the address was not *.blob.core.windows.net, but instead *.dfs.core.windows.net
And I only had a private DNS zone for .blob.core.windows.net. Setting AZURE_STORAGE_BLOB_ENDPOINT had no effect on the call, which returned:

===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=403 This request is not authorized to perform this operation., Details: (none)
   GET https://*.dfs.core.windows.net/mycont?maxResults=2&recursive=false&resource=filesystem&timeout=901
   Authorization: REDACTED
   User-Agent: [Azure-Storage-Fuse/2.1.2 (Debian GNU/Linux 12 (bookworm)) Azure-Storage/0.1 (go1.20.5; linux)]
   X-Ms-Client-Request-Id: [REDACTED]
   X-Ms-Date: [Fri, 05 Apr 2024 11:40:17 GMT]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [194]
   Content-Type: [application/json;charset=utf-8]
   Date: [Fri, 05 Apr 2024 11:40:17 GMT]
   Server: [Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [REDACTED]
   X-Ms-Version: [2018-11-09]
]

A better error message could've saved me a lot of time (with --log-level=LOG_DEBUG, I would expect to get the most out in the output of the command).

The other part is where the log file was located. I can imagine this is something customized by the driver itself, but given both products target the same cloud service, I thought they had a somewhat concise way to operate and couldn't figure out what was happening to the logs and assumed they were not being written for some reason.

After figuring this out, either pointing to a block blob storage account type, with AZURE_STORAGE_ACCOUNT_TYPE=block or creating a private DNS zone with .dfs.core.windows.net solve the issue.

But the error message is the same if either:

• There is no connectivity
• Authentication fails (for any of the variety of reasons)
• Target container does not exist

Which makes it quite difficult to understand the root cause. I think the error message should be more specific and propate the code (ie: Code: ContainerNotFound) returned by the API call (or where there is a timeout, say it is a timeout).

I am not sure if it's the best idea to simply close the issue. But if so, maybe it is also a good idea to raise a new issue to improve the error message directly returned by the command. Otherwise, we need to go to a number of o hops and dig through other docs just to get to the daemonset for the CSI driver, to then start experimenting and try reproduce the problem and finally get to the root cause by digging a mysteriously located log file.

[bhansalivikas]

• Log file location is something directed by the Blob CSI Driver. Depending upon their custom settings the path of log file exists and blobfuse as a tool have no control over it. If you wish to provide any suggestion in this regard, blob csi driver repo would be the correct place. I can convey the same to the team internally but coming that directly from customer will have more weight associated.
• In case of private endpoints either you shall have correct DNS configured to resolve the public endpoint to private endpoint or configure endpoint URL in blobfuse config. In case of containers and AKS we have observed customers define some of these but they are not exposed to the running container or pods and this results into failure from blobfuse side.
• If any of the config is not exposed to Blobfuse, it has no way to understand whether the end point is not configured, DNS is not configured or network path is not stable. It will just try to reach out to the configured options and if that fails, it will report back the exact error it received on the network. Logs will also poiont to the same message. What we can do is to improve our documentaion and call these things out.
• For blobfuse any user configuration error is unknown and if anything fails, we can only report back what error we received on the network. There is no way for blobfuse to know what was the exact configuration, which was missing, and it can be a variety of things. Again, may be better documentation is what I take away from this discussion. Feel free to close the issue if you align to these thoughts otherwise you can share more thoughts/insights here.

[fabio-s-franco]

• Log file location is something directed by the Blob CSI Driver. Depending upon their custom settings the path of log file exists and blobfuse as a tool have no control over it. If you wish to provide any suggestion in this regard, blob csi driver repo would be the correct place. I can convey the same to the team internally but coming that directly from customer will have more weight associated.

That is indeed what I assumed, but given the customer has no control over the driver or the configuration as it is setup out of the box by managed CSI driver installation, it is difficult to assume both tools coming from same supplier, would diverge. And I would say, a word from both sides would be the most effective. Here is the issue, that with an acknowledgement from your team, would probably help getting the proper attention: kubernetes-sigs/blob-csi-driver#1333

You can also add anything I may have been missing on my issue report.

• In case of private endpoints either you shall have correct DNS configured to resolve the public endpoint to private endpoint or configure endpoint URL in blobfuse config. In case of containers and AKS we have observed customers define some of these but they are not exposed to the running container or pods and this results into failure from blobfuse side.

The problem is that there was no way for me to know (or I could not find any documentation about) that setting the environment variable AZURE_STORAGE_ACCOUNT_TYPE=adls or using the flag --use-adls=true (which is the case of configuring CSI Driver storage class), would cause blobfuse2 to automatically use .dfs.core.windows.net, instead of .blob.core.windows.net. There are two problems here in my point of view:

1 - Lack of documentation explaining this nuance
2 - No way to configure the endpoint used by blobfuse2 when using adls (for the sake of HNS enabled mount).

If any of the config is not exposed to Blobfuse, it has no way to understand whether the end point is not configured, DNS is not configured or network path is not stable. It will just try to reach out to the configured options and if that fails, it will report back the exact error it received on the network. Logs will also poiont to the same message. What we can do is to improve our documentaion and call these things out.

What I am pointing out is exactly opposite of what you are saying here. Don't get me wrong, I am not trying to point fingers, but instead trying to clarify what was happening in my situation and how to improve the process of configuration and diagnostic of blobfuse2. Two main points can be derived from this experience:

1 - Private endpoint and DNS private zone were correctly configured, I even shared my output of netcat, showing the the port was open, calling from the same container as blobfuse2 command, meaning the name was resolving to the correct IP address. The problem was here that the blobfuse2 requires (and automatically uses) the DFS endpoint. The variable you mentioned AZURE_STORAGE_BLOB_ENDPOINT had no influence on the URL called when --use-adsl=true and without proper output of the error, it becomes very difficult to pinpoint this nuance.

2 - It does not report the exact error it received from the network and that is what I am trying to point out it would be a lot more useful to output instead. blobfuse2 abstract any possible underlying networking errors in combination with any authentication errors into a single not helpful error: Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
If it instead, produce the output of the real error (like network timeout) or one of the return codes I saw on blobfuse2 log files (when called manually), I could have immediately identified the problem, as that would have been propagated by the CSI Driver into the kubernetes resource event. As I mentioned, I experienced three different types of networking errors I later identified when manually calling blobfuse2 (which in turn would show up in the logs). If it is shown in the logs, it can be made bubled up to the output. Those are quite clear on the API response or in the case of a timeout, it is also easily determined.

• For blobfuse any user configuration error is unknown and if anything fails, we can only report back what error we received on the network. There is no way for blobfuse to know what was the exact configuration, which was missing, and it can be a variety of things. Again, may be better documentation is what I take away from this discussion. Feel free to close the issue if you align to these thoughts otherwise you can share more thoughts/insights here.

Following up from my previous point, I am not really asking for blobfuse2 to identify what I have done wrong, but instead, output more relevant information. Packaging up a bunch of different underlying errors into a single error message is blindfolding the user. What is even worse, it may indicate it is an authentication issue, when in fact is a networking issue. I spent a lot of time misled by this error, thinking there was something wrong with authentication. Whether I invertedly was using workload identity authentication and whatnot.

Even doing something as simple as adding the url called and slightly change the error message would have made this endeavor a lot (an understatement) less frustrating:

Error: failed to initialize new pipeline [**network or authentication** error while trying to connect with azstorage @**** *.dfs.core.windows.net]

I can tell you, if I had seen such an error message popping up on my kubernetes resource event history, this would have transformed days of headless chicken debugging to something resolved within an hour.

I still think the error could be more clear to propagate to the output more detail coming from the network or API. Pluse a way to more easily configure the behavior for HNS enabled mounts, such as the endpoint used, in addition to some clarifying documentation is the way to go in my opinion.

[mkemmerz]

Hello, we are receiving the same error message but with a different setup. We try to use the Workload Identity on AKS 1.29.2 with the Microsoft-Managed Blob CSI Driver solution. Our Storage Account and Container are publicly reachable, Access Keys are disabled, "Storage Account Contributor" granted to the Identity and Workload Identity setup inside of the AKS is done correctly too.

We strictly followed the documentation: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/workload-identity-static-pv-mount.md (option#2: Pod with ephemeral inline volume) but without any luck.

The log of the csi-blob-node Pod. Maybe someone understand more than I do.

[fabio-s-franco]

@mkemmerz Your logs are cut off due to it being a screenshot. It would be easier to help if you copied and pasted the text in a cod block.

Having that said, if you followed my thread, the best way to see the root cause is to check the logs from blobfuse2 by running the command manually inside the csi driver container. There are more details on the issue created in the csi driver repo. You problem is likely something unrelated to this issue.

Also, make sure to read the first paragraph here:

https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/workload-identity-deploy-csi-driver.md

workload identity is NOT supported on AKS managed Blob CSI driver since the driver controller is managed by AKS control plane which is already using managed identity by default, it's not necessary to use workload identity for AKS managed Blob CSI driver.

Check whether it is your case.

[mkemmerz]

@fabio-s-franco thanks for the quick response. This is basically where it gets a bit confusing.
AKS mentions the Workload Identity support in its last release, which comes with the blob-csi-driver 1.23.3: https://github.com/Azure/AKS/releases/tag/2024-03-17

On the other hand, the blob-csi-driver mentions the workload identity support in its 1.24.0 release: https://github.com/kubernetes-sigs/blob-csi-driver/releases/tag/v1.24.0

I guess AKS made a mistake and it is not fully supported yet? The docs also mentions a support starting from 1.24.0

The full log:

E0411 10:53:36.767542   93742 utils.go:108] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
, output: 
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
I0411 10:55:38.866707   93742 utils.go:103] GRPC call: /csi.v1.Node/NodePublishVolume
I0411 10:55:38.866725   93742 utils.go:104] GRPC request: {"target_path":"/var/lib/kubelet/pods/b4b59bc1-4fa4-4f25-a559-d765e2eeec7d/volumes/kubernetes.io~csi/persistent-storage/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"clientID":"c16dc5d5-c4c9-4ca2-881b-639f112b1033","containerName":"testvol","csi.storage.k8s.io/ephemeral":"true","csi.storage.k8s.io/pod.name":"busybox","csi.storage.k8s.io/pod.namespace":"hello-world-1","csi.storage.k8s.io/pod.uid":"b4b59bc1-4fa4-4f25-a559-d765e2eeec7d","csi.storage.k8s.io/serviceAccount.name":"service-account-azure-workload-identity","csi.storage.k8s.io/serviceAccount.tokens":"{\"api://AzureADTokenExchange\":{\"token\":\"xxx\",\"expirationTimestamp\":\"2024-04-11T11:55:38Z\"}}","resourceGroup":"ReG_aksscd001hw1","storageAccount":"saaksscd001hw1","subscriptionid":"xxx"},"volume_id":"csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a"}
I0411 10:55:38.866860   93742 nodeserver.go:85] NodePublishVolume: volume(csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a) mount on /var/lib/kubelet/pods/b4b59bc1-4fa4-4f25-a559-d765e2eeec7d/volumes/kubernetes.io~csi/persistent-storage/mount with service account token, clientID: c16dc5d5-c4c9-4ca2-881b-639f112b1033
I0411 10:55:38.867085   93742 blob.go:416] parsing volumeID(csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a) return with error: error parsing volume id: "csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a", should at least contain two #
I0411 10:55:38.867098   93742 blob.go:497] volumeID(csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a) authEnv: []
I0411 10:55:38.867107   93742 blob.go:522] clientID(c16dc5d5-c4c9-4ca2-881b-639f112b1033) is specified, use service account token to get account key
I0411 10:55:39.497844   93742 nodeserver.go:385] target /var/lib/kubelet/pods/b4b59bc1-4fa4-4f25-a559-d765e2eeec7d/volumes/kubernetes.io~csi/persistent-storage/mount
protocol 

volumeId csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a
context map[clientID:c16dc5d5-c4c9-4ca2-881b-639f112b1033 containerName:testvol csi.storage.k8s.io/ephemeral:true csi.storage.k8s.io/pod.name:busybox csi.storage.k8s.io/pod.namespace:hello-world-1 csi.storage.k8s.io/pod.uid:b4b59bc1-4fa4-4f25-a559-d765e2eeec7d csi.storage.k8s.io/serviceAccount.name:service-account-azure-workload-identity csi.storage.k8s.io/serviceAccount.tokens:{"api://AzureADTokenExchange":{"token":"xxx","expirationTimestamp":"2024-04-11T11:55:38Z"}} resourceGroup:ReG_aksscd001hw1 storageAccount:saaksscd001hw1 subscriptionid:xxx]
mountflags []
mountOptions [ --empty-dir-check=false --tmp-path=/mnt/csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a --container-name=testvol --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10]
args /var/lib/kubelet/pods/b4b59bc1-4fa4-4f25-a559-d765e2eeec7d/volumes/kubernetes.io~csi/persistent-storage/mount  --empty-dir-check=false --tmp-path=/mnt/csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a --container-name=testvol --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
serverAddress saaksscd001hw1.blob.core.windows.net
I0411 10:55:39.497882   93742 nodeserver.go:167] start connecting to blobfuse proxy, protocol: , args: /var/lib/kubelet/pods/b4b59bc1-4fa4-4f25-a559-d765e2eeec7d/volumes/kubernetes.io~csi/persistent-storage/mount  --empty-dir-check=false --tmp-path=/mnt/csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a --container-name=testvol --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
I0411 10:55:39.498692   93742 nodeserver.go:176] begin to mount with blobfuse proxy, protocol: , args: /var/lib/kubelet/pods/b4b59bc1-4fa4-4f25-a559-d765e2eeec7d/volumes/kubernetes.io~csi/persistent-storage/mount  --empty-dir-check=false --tmp-path=/mnt/csi-dd224d9ed82701454e5e5fa08fafbacef80a0a8bf9f4375fd8921758128d2b9a --container-name=testvol --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
E0411 10:55:39.925577   93742 nodeserver.go:179] GRPC call returned with an error:rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
E0411 10:55:39.925624   93742 nodeserver.go:411] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]

[fabio-s-franco]

@mkemmerz The CSI driver can be setup in multiple ways:

• Managed
• Helm Chart
• Scripted

The first one is the case that note is talking about. As it states in the note "on AKS managed Blob CSI driver since the driver controller is managed by AKS control plane". We don't have direct access to the control plane and it also means it uses the node managed identity, which indicates we can't really change that. But I would raise an issue on the CSI driver repo, as a sanity check to make sure the document is not outdated.