Error HTTP 409 for Azure Blob with empty managedIdentity on Linux

[CravateRouge]

I got the HTTP error 409 Public access is not permitted on this storage account. when trying to access a file in an Azure Blob with the system assigned identity of my Linux ARM debian 12 VM which has the "StorageBlobDataContributor" role:

        {
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "apiVersion": "2024-07-01",
            "name": "[concat(parameters('virtualMachineName'), '/CustomScriptExtension')]",
            "location": "eastasia",
            "dependsOn": [
                "[resourceId('Microsoft.Compute/virtualMachines', parameters('virtualMachineName'))]",
                "[resourceId('Microsoft.Authorization/roleAssignments', guid(parameters('virtualMachineName'), 'StorageBlobDataContributor'))]"
            ],
            "properties": {
                "publisher": "Microsoft.Azure.Extensions",
                "type": "CustomScript",
                "typeHandlerVersion": "2.1",
                "autoUpgradeMinorVersion": true,
                "settings": {
                },
                "protectedSettings": {
                    "fileUris": [
                        "https://<storage name>.blob.core.windows.net/<container>/config.json"
                    ],
                    "commandToExecute": "cp config.json /tmp/.",
                    "managedIdentity" : {}
                }
            }
        }

Here is the full log:

/var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.1.13/bin/custom-script-shim: line 60: lsof: command not found
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.1.13/bin/custom-script-extension-arm64 enable
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event=start
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event=pre-check
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="comparing seqnum" path=mrseq
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="seqnum saved" path=mrseq
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="reading configuration"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="read configuration"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="validating json schema"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="json schema valid"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="parsing configuration json"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="parsed configuration json"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="validating configuration logically"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="validated configuration"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="creating output directory" path=/var/lib/waagent/custom-script/download/2
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="created output directory"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 files=1
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 event="download start"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 retry=0 info="starting download with client request ID 793d2e1a-528a-4310-8ae2-387dd25630d3"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 retry=0 error="file download failed with error 'CustomScript failed to download the file from STORAGE.blob.core.windows.net because the server returned a response code and message of \"409 Public access is not permitted on this storage account.\" Please verify the machine has network connectivity. (Service request ID: adbded22-d01e-0014-129c-a028c6000000)' : downloaded and saved 0 bytes in 423 milliseconds"
time=2025-03-29T11:17:49Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 retry=0 info="downloader download.urlDownload returned 409, skipping retries"
time=2025-03-29T11:17:51Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 retry=0 info="starting download with client request ID 0acc4fe9-37e4-41db-ba95-5e12ceac43e6"
time=2025-03-29T11:17:51Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 retry=0 error="file download failed with error 'please ensure that the blob location in the fileUri setting exists, and the specified Managed Identity has read permissions to the storage blob (Service request ID: e6078436-501e-0035-809c-a00cbd000000)' : downloaded and saved 0 bytes in 1843 milliseconds"
time=2025-03-29T11:17:51Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 retry=0 info="downloader *download.blobWithMsiToken returned 404, skipping retries"
time=2025-03-29T11:17:51Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 file=0 event="download failed" error="failed to download response and write to file: /var/lib/waagent/custom-script/download/2/config.json: please ensure that the blob location in the fileUri setting exists, and the specified Managed Identity has read permissions to the storage blob (Service request ID: e6078436-501e-0035-809c-a00cbd000000)"
time=2025-03-29T11:17:51Z version=v2.1.12/git@4636e53-dirty operation=enable seq=2 event="failed to handle" error="processing file downloads failed: failed to download file[0]: failed to download response and write to file: /var/lib/waagent/custom-script/download/2/config.json: please ensure that the blob location in the fileUri setting exists, and the specified Managed Identity has read permissions to the storage blob (Service request ID: e6078436-501e-0035-809c-a00cbd000000)"

I think there is an error while retrieving the access token and using it for the HTTP request to the blob.

However I have no issue when trying manually:

# Run the first curl command and capture its output in a variable
access_token=$(curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F' -H Metadata:true | jq -r '.access_token')

# Run the second curl command with the access token
curl "https://<STORAGE ACCOUNT>.blob.core.windows.net/<CONTAINER NAME>/<FILE NAME>" \
  -H "x-ms-version: 2017-11-09" \
  -H "Authorization: Bearer $access_token"