Self-hosted Azure runner failed to create

[dennisameling]

I just kicked off a run to build the ARM64 version of 2.43.0, but it looks like the deployment of the self-hosted Azure runner failed:

Error: ERROR: ***"status":"Failed","error":***"code":"DeploymentFailed","target":"/subscriptions/***/resourceGroups/***/providers/Microsoft.Resources/deployments/deploy-actions-runner-20231121103205908804162","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[***"code":"ResourceDeploymentFailure","target":"/subscriptions/***/resourceGroups/***/providers/Microsoft.Compute/virtualMachines/actions-runner-20231121103205908804162","message":"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.","details":[***"code":"OSProvisioningTimedOut","message":"OS Provisioning for VM 'actions-runner-20231121103205908804162' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. For details on how to check current provisioning state of Windows VMs, refer to https://aka.ms/WindowsVMLifecycle and Linux VMs, refer to https://aka.ms/LinuxVMLifecycle."***]***]***

I don't have access to these resources, so can't see what exactly failed here. @dscho would you mind checking your VM overview in Azure? 🙏🏼

[dscho]

Hmm. From a cursory look, it seems that all the resources were provisioned all right. The VM was running. I deleted those resources and kicked off a re-run of the workflow spinning up a new VM. Let's 🤞 that it will work this time.

[dscho]

The pkg is building!

[dennisameling]

Thank you and sorry for bothering! I forgot that we have the "delete azure-self-hosted-runner" workflow that I could've used to manually remove the VM instead of you having to go into the Azure Portal. Great to see it's working again 😊

[dennisameling]

Hmm, things seem to be very unstable.

• Another failed run: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/6945021981/attempts/1 (I deleted the VM manually afterwards)
• Another pending job that's been running for 20 mins and will likely time out after 40 mins as well: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/6945021981/attempts/2. I'll make sure to remove it manually if it fails again.

I'll try to replicate this in my own Azure tenant, but have an open ticket with Azure support to enable arm-based runners in my tenant. That'll take a while.

[dscho]

No worries about bothering me, all good. One thing I noticed too late, recently, is that the VMs are created without auto-shutdown. I noticed only after three weeks of the VM running, running through half of my monthly credit...

Do you think you could add an auto shut-down after, say, 6h?

[dscho]

Another idea: we should be able to teach GitForWindowsHelper to detect failed create-self-hosted-runner runs and automatically kick off the delete run, no? And maybe also try re-running the failed run, up to five times?