[BUG] ManagedIdentityCredential attempts to parse a non-success response

[cataggar]

Library name and version

Microsoft.Azure.WebJobs.Extensions.Storage.Queues 5.0.0

Describe the bug

ManagedIdentityCredential authentication sometimes fails directly after a deployment of an Azure Function. I put additional details in Azure/azure-functions-host#8623.

Exception while executing function: StartCreateHostUsage ManagedIdentityCredential authentication failed: Managed Identity response was not in the expected format. See the inner exception for details.
Status: 403 (Forbidden)



Headers:
Date: Tue, 09 Aug 2022 23:00:04 GMT
Server: Kestrel
X-CORRELATION-ID: REDACTED
Content-Length: 0



See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot Managed Identity response was not in the expected format. See the inner exception for details.
Status: 403 (Forbidden)



Headers:
Date: Tue, 09 Aug 2022 23:00:04 GMT
Server: Kestrel
X-CORRELATION-ID: REDACTED
Content-Length: 0
The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0.

Expected behavior

A better error message if the managed identity endpoint responds with a 403. It would be nice if it retried or a retry policy could be set up. Is there a way for me to wait for the system managed identity to be ready during startup?

Actual behavior

I seems like the managed identity is not quite ready yet in the App Service environment and a 403 response is returned. Even with a bad response, the code still tries to parse the response body and fails with a JsonReaderException.

Reproduction Steps

Deploy an Azure Function that uses system managed identity and try to write to a storage queue. Here is the code we are using.

type StartCreateUsage() =
    [<FunctionName(nameof StartCreateUsage)>]
    member x.Run
        // Hourly process to create usage
        ([<TimerTrigger("1 0 * * * *", RunOnStartup = false, UseMonitor = true)>] timer: TimerInfo, [<Queue(StorageQueues.createUsage)>] createUsageQueue: ICollector<CreateUsage.Params>)
        (log: ILogger): Task<unit> =
            task {
                let usageHours = getUsageHours (timer.ScheduleStatus.Last.ToUniversalTime()) DateTime.UtcNow
                if usageHours.Length > 1 then
                    let usageHours = usageHours |> List.map (fun d -> d.SortableString)
                    log.Warningf "creating usage for more than one hour: %A" usageHours
                for usageHour in usageHours do
                   CreateUsage.Params(UsageHour = usageHour) |> createUsageQueue.Add
            }

Environment

It is dotnet 6. The base docker image is:
FROM mcr.microsoft.com/azure-functions/dotnet:4.0.1.16816-dotnet6-appservice

[jsquire]

Hi @cataggar. Thank you for reaching out and we regret that you're experiencing difficulties. There's nothing that the credential can do to address the root problem, as the endpoint itself is returning a 403. (your Functions issue is a good path forward there). I'm not sure why we're attempting to parse the body of a non-success response - that seems like it may be a bug. I've updated the title to reclassify and have routed this to the team member best able to assist.

[cataggar]

Thanks. The Azure.Identity version is 1.6.0.

      "Azure.Identity": {
        "type": "Transitive",
        "resolved": "1.6.0",
        "contentHash": "EycyMsb6rD2PK9P0SyibFfEhvWWttdrYhyPF4f41uzdB/44yQlV+2Wehxyg489Rj6gbPvSPgbKq0xsHJBhipZA==",
        "dependencies": {
          "Azure.Core": "1.24.0",
          "Microsoft.Identity.Client": "4.39.0",
          "Microsoft.Identity.Client.Extensions.Msal": "2.19.3",
          "System.Memory": "4.5.4",
          "System.Security.Cryptography.ProtectedData": "4.7.0",
          "System.Text.Json": "4.7.2",
          "System.Threading.Tasks.Extensions": "4.5.4"
        }
      },

[schaabs]

@cataggar Thanks for filing this issue, and sorry for the confusion. When the ManagedIdentityCredential receives a non-success status code from the managed identity endpoint, it tries to extract an error message from the response. In this error case the response body seems to not be JSON, but also is not empty so we fail to parse. This exception is handled and an RequestFailedException is raised, but it seems we're using the exception raised when parsing as the inner exception. I see how this could be confusing as it could imply that the parsing is what caused the request to fail, but in fact we would fail in any case given we receive a 403 from the managed identity endpoint. We could improve the error message by separating the handling of exceptions thrown from parsing 200 responses from those of trying to parse error messages from non-success status codes.

The other question this raises is why the managed identity endpoint is returning a 403 Forbidden error. Unfortunately the REST reference docs I was able to find don't detail what error codes are expected from the endpoint. 403 Forbidden is usually used for authorization failures, and hence not retriable. However, if the App Service managed identity endpoint is using this to indicate that the service is not currently available perhaps we should add this to our retriable codes so we can retry as we would for other transient errors. Do you have any info that can confirm this is in fact retriable?

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[cataggar]

@schaabs, the service isn't supposed to send a 403 error, but was. Here is the internal IcM that is now resolved. It probably shouldn't be retriable. Yes, a better error message for non-success status codes is what this issue is about.