[Aspire] azd up sometimes fails because of not reaching ACR

[christiannagel]

When creating new Azure resources and publishing the app, azd up sometimes fails because it cannot access the ACR. Repeating it, everything succeeds.
For a better user experience, I think it would be useful to check for the ACR to be available and repeat the check before failing the deployment.

I had this issue when deploying to East US 2 from within Europe. This time creating the Container Apps Environment was a lot faster than previous experiences. It looks like that's why this failed.

Versions:
azd version 1.6.1 (commit eba2c97)
aspire 8.0.0-preview.3.24105.21/8.0.100

Packaging services (azd package)

SUCCESS: Your application was packaged for Azure in 5 seconds.

Provisioning Azure resources (azd provision)
Provisioning Azure resources can take some time.

Subscription: <redacted>
Location: East US 2

  You can view detailed progress in the Azure Portal:
  https://portal.azure.com/#view/HubsExtension/DeploymentDetailsBlade/~/<redacted>

  (✓) Done: Resource group: rg-codebreaker-packt12us
  (✓) Done: Log Analytics workspace: law-okccxl67xgpc6
  (✓) Done: Container Registry: acrokccxl67xgpc6
  (✓) Done: Container Apps Environment: cae-okccxl67xgpc6
  (✓) Done: Container App: redis
  (✓) Done: Azure Cosmos DB: codebreakercosmosokccxl67xgpc6

SUCCESS: Your application was provisioned in Azure in 3 minutes 16 seconds.
You can view the resources created under the resource group rg-codebreaker-packt12us in Azure Portal:
https://portal.azure.com/#@/resource/subscriptions/<redacted>

Deploying services (azd deploy)

  (x) Failed: Deploying service blazor

ERROR: failed deploying service 'blazor': failing invoking action 'deploy', logging in to registry: failed logging into container registry, token: failed getting ACR token: POST https://acrokccxl67xgpc6.azurecr.io/oauth2/exchange
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
, admin: cannot find registry with name 'acrokccxl67xgpc6' and subscriptionId '<redacted'

ERROR: error executing step command 'deploy --all': failed deploying service 'blazor': failing invoking action 'deploy', logging in to registry: failed logging into container registry, token: failed getting ACR token: POST https://acrokccxl67xgpc6.azurecr.io/oauth2/exchange
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
, admin: cannot find registry with name 'acrokccxl67xgpc6' and <redacted>

After starting the deployment again, everything succeeded.

[rajeshkamal5050]

@vhvb1989 is it related to the admin user authn error we were seeing earlier?

[christiannagel]

@rajeshkamal5050 @vhvb1989 - if #2980 is meant, I don't think this is related. Previously I had this authentication issue as well because of an old Azure account. After adding the roles as described with this issue, this was solved.

The issue here looks like it's on timing. For me, usually azd up is working with the first run. In this case, the container apps environment was completed in just 3 minutes. Usually, it takes a lot more. I think the ACR was created, but it was not accessible at that time. Running it again just after the error, the deployment succeeded.

[vhvb1989]

@vhvb1989 is it related to the admin user authn error we were seeing earlier?

No. This is a timing issue. By the time deploy runs, the Azure Container Registry is still not discoverable by azd (b/c it might take a few seconds for Azure to propagate the resource existents) .

We have the same issue for Azure Web Apps.

azd makes 3 retries by deafult, but I think we need to add some exponential delay time between retries to make it more reliable.

[weikanglim]

Hey @christiannagel , @vhvb1989 and I were reviewing this issue again, we noticed that you mentioned that the issue occurs when deploying to East US 2 from within Europe. Have you noticed it when deploying to the same geolocation? We are suspecting that geo-replication is the primary factor behind the propagation delay.

The slightly uneasy part of making an improvement here is that azd today doesn't quite know whether the ACR was recently created (this is a bicep output). The service is simply responding with 404 that "the ACR doesn't exist". If azd waits and retries, it may end up waiting longer in cases where truly, the ACR doesn't exist.

[christiannagel]

Hi @weikanglim , @vhvb1989 - I'm deploying from my local system in Austria. I didn't see this issue when deploying to West Europe, only to East US 2. However, in West Europe resource provisioning took 7+ minutes, in East US 2 the provisioning was ready after 3 minutes and 16 seconds.

[christiannagel]

Hi @weikanglim, @vhvb1989 - currently I'm seeing a timing issue more often using azd up, also when deploying from within Austria to West Europe. It's also a lot faster now (probably because of Easter). This error was curious:

? Select an Azure Subscription to use:  3. <redacted>
? Select an Azure location to use: 34. (Europe) West Europe (westeurope)

Packaging services (azd package)


SUCCESS: Your application was packaged for Azure in less than a second.

Provisioning Azure resources (azd provision)
Provisioning Azure resources can take some time.

Subscription: <redacted>
Location: West Europe

  You can view detailed progress in the Azure Portal:
  https://portal.azure.com/#view/HubsExtension/DeploymentDetailsBlade/~/overview/id/<redacted>providers%2FMicrosoft.Resources%2Fdeployments%2Fcodebreaker-08-dev-1711806663

  (✓) Done: Resource group: rg-codebreaker-08-dev
  (✓) Done: Log Analytics workspace: law-ssengodjcf2mu
  (✓) Done: Container Registry: acrssengodjcf2mu
  (✓) Done: Container Apps Environment: cae-ssengodjcf2mu

SUCCESS: Your application was provisioned in Azure in less than a second.
You can view the resources created under the resource group rg-codebreaker-08-dev in Azure Portal:
https://portal.azure.com/#@/resource/subscriptions/<redacted>/resourceGroups/rg-codebreaker-08-dev/overview

Deploying services (azd deploy)

  (✓) Done: Deploying service bot
  - Endpoint: https://bot.yellowground-22274287.westeurope.azurecontainerapps.io/

  (x) Failed: Deploying service gameapis

ERROR: failed deploying service 'gameapis': failing invoking action 'deploy', updating container app service: applying manifest: PUT https://management.azure.com/subscriptions/7221029f-b7dc-4ab8-b117-111900d0d73c/resourceGroups/rg-codebreaker-08-dev/providers/Microsoft.App/containerApps/gameapis
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidParameterValueInContainerTemplate
--------------------------------------------------------------------------------
{
  "error": {
    "code": "InvalidParameterValueInContainerTemplate",
    "message": "The following field(s) are either invalid or missing. Field 'template.containers.gameapis.image' is invalid with details: 'Invalid value: \"acrssengodjcf2mu.azurecr.io/azd-deploy-gameapis-1711806831\": Get \"https://acrssengodjcf2mu.azurecr.io/v2/\": dial tcp: lookup acrssengodjcf2mu.azurecr.io on 100.100.224.10:53: no such host';."
  }
}
--------------------------------------------------------------------------------

TraceID: 3c2685661739cea960ad29545807b2a6

ERROR: error executing step command 'deploy --all': failed deploying service 'gameapis': failing invoking action 'deploy', updating container app service: applying manifest: PUT https://management.azure.com/subscriptions/7221029f-b7dc-4ab8-b117-111900d0d73c/resourceGroups/rg-codebreaker-08-dev/providers/Microsoft.App/containerApps/gameapis
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidParameterValueInContainerTemplate
--------------------------------------------------------------------------------
{
  "error": {
    "code": "InvalidParameterValueInContainerTemplate",
    "message": "The following field(s) are either invalid or missing. Field 'template.containers.gameapis.image' is invalid with details: 'Invalid value: \"acrssengodjcf2mu.azurecr.io/azd-deploy-gameapis-1711806831\": Get \"https://acrssengodjcf2mu.azurecr.io/v2/\": dial tcp: lookup acrssengodjcf2mu.azurecr.io on 100.100.224.10:53: no such host';."
  }
}
--------------------------------------------------------------------------------

TraceID: 3c2685661739cea960ad29545807b2a6

The provisioning part was successful. With the deployment stage, the bot service was successfully deployed - thus the ACR could be reached here. When deploying the games API, the ACR could not be reached: "no such host". Probably the deployment to the games API was started earlier?
With the output it was showing Your application was provisioned in Azure in less than a second., but this of course was not correct. Deployments within the Azure portal show 41,71 seconds for resources, and 48,51 seconds for the app configuration.

Running azd up again, everything succeeded.

I had similar timing issues running this from a GitHub Actions workflow using azd deploy.

Versions currently used:
azd version 1.7.0 (commit 49d6adc)
aspire 8.0.0-preview.6.24177.4/8.0.100 SDK 8.0.200

[vhvb1989]

@christiannagel , for the versions you mentioned, seems like you are trying the last azd release 1.7.0 with a daily aspire version (preview-6) ?
We don't cover such combination.

There are currently breaking changes for the upcoming aspire P5. The fixes for azd are in this PR: #3612 . If you are trying daily aspire builds, please install azd from that PR or wait until it is merged to take azd's daily version.