Appservice deployment failed #60
Comments
|
Do you have the full AppService log file that you could email me? asdkconfigurator @ outlook . com - I’m curious if there’s anything else hidden in there. Is the environment internet connected without a proxy or anything like that? Thanks! |
|
I've send the log files. The server is hosted in a datacenter and has a corporate firewall in front of it. |
|
Note that: I'm not sure if this is a structural error. It could very well be that a redeploy would work fine, if issue #59 is resolved. In that case, there should simply be some sort of retry for the VM deployment. |
|
OK, after reviewing your logs, I don't think your original issue was due to the database. WHile there are many points in the large log file that say "Waiting for Hosting database to be ready", on all occasions, eventually, that was successful. The error you have is around provisioning of CN0-VM, specifically "failed to provision vm extensions for vm cn0-vm" which i think may have been a random transient issue. I'm not sure how the App Service handles reinstalling over a partially failed attempt, so what I may do is write some cleanup of the resource group itself, and if there has been a failed run previously, I just completely clean the App Service resource group (and the SQL DB cleanup) and then it tries a fresh run again. I'll work on that later. |
|
So my challenge is, I don't know how the AppService.exe handles the re-run. We know it fails almost immediately if the database exists, and so we have accounted for that by cleaning the DB. But, if you look in the appservice-infra resource group, and click deployments, you'll see that there are 9 separate deployments that take place: AppService.DeployCloud I don't know at what stage, any of these deployments write to the DB. In a default deployment on the ASDK, 7 of the 9 deployments listed above (from LargeWorker down to DeployStorage) finish in less than 2 minutes. If all of those succeed, but say, the AppService.DeployCloud fails, if we're cleaning up the database before the next attempt, will that break anything associated with the steps that had previously completed? That's the question I don't know. It may be 'safer' and more predictable, to completely delete the appservice-infra resource group if it already exists, and then start a fresh for each attempt. I'll look into it a bit more though. |
|
Let's retest with the latest 1809.2 version of the DeployAppService.ps1 - as mentioned here: #59 (comment) - this should clean up failed runs and retry from the start again. |
|
Unfortunately it still fails with the new script. I've send the log to your email address. |
|
Yep, looking at your logs, my script is not catching the fact that you have a failed deployment within the appservice-infra resource group. Can you confirm that you still see the appservice-infra RG and that there is a failed deployment within it? I'll work on adjusting the code to make it work. |
|
Could you run this in your environment? # Check if there is a previous failure for the App Service deployment - easier to completely clean the RG and start fresh
# Login to Azure Stack first
$azsLocation = (Get-AzsLocation).Name
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
$appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra" -Name "AppService.DeployCloud" -ErrorAction Stop)
$appServiceFailCheck.ProvisioningState
if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. Starting cleanup..."
}and post the output here? Does it print the 'There is evidence...' message? |
|
The appservice-infra resource group doesn't exist. I do see appservice-fileshare and appservice-sql resource groups. I just ran the commands you send, but it doesn't seem to print. |
|
OK, this is strange: PS C:\Windows\system32> $appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra"
-Name "AppService.DeployCloud" -ErrorAction Stop)
Get-AzureRmResourceGroupDeployment : Resource group 'appservice-infra' could not be found.If you look in the admin portal, under Resource Groups, is there one that shows "appservice-######"? What is the result when you run: Get-AzureRmResourceGroup |
|
I don't have a resource group which is literally called "appservice-######" Output from |
|
The output above, is from Azure. You need to make sure you're logged into Azure Stack first, then run Get-AzureRmResourceGroup. So do this: $azsLocation = (Get-AzsLocation).Name
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
Get-AzureRmResourceGroupAnd see what comes back. I'm curious why you don't have the appservice-infra, and if there's another random one that's been created. Aside from the appservice-sql and appservice-fileshare (which my script creates for you), if you have no other appservice- here, my 1809.2 script must have cleaned it up successfully, so now we have to figure out why your rerun is failing. |
|
The output from that command is: |
|
OK, so that shows you have no left-over appservice-infra stuff, so there shouldn't be a problem re-deploying the App Service. As a test, i crashed the App Service deployment within my new ASDK. Within the appservice-infra resource group, mine clearly showed 'failed' for the AppService.DeployCloud deployment. I then reran the script. My script successfully cleaned up the resuorce group, and successfully redeployed the App Service, so it must be something in your environment. Can you try one more time, ensuring you've downloaded this version: https://raw.githubusercontent.com/mattmcspirit/azurestack/1809.2/deployment/powershell/DeployAppService.ps1 and copied it into your ConfigASDK\Scripts folder, overwriting the one that's there. Then run the main script and see if it fails. NOTE on your launch script, enclose your passwords in single quotes 'P@ssw0rd!' rather than double-quotes. If it does fail, can you please email me the log file from the app service itself (not the log files in the ConfigASDK\Logs folder). This appservice log file will be within the downloadPath\ASDK\appservice folder. The name of the log will be something like AppServiceLog1118-145337 but your numbers will be different. If it still fails with the 'network password' issue, we'll do some additional cleanup and try again. |
|
I've send you the logs, as it still fails with the 'network password' issue. |
|
OK, here's what we're going to do.
So, we've essentially cleaned up your temp folder, cleaned up certs and old txt files, and removed your app. We need to ensure that the script will run the 'AppServicePreReqs stage again, so we'll need to do this:
$sqlServerInstance = '(localdb)\MSSQLLocalDB'
$databaseName = "ConfigASDK"
$tableName = "Progress"
$progressStage = "AddAppServicePreReqs"
Invoke-Sqlcmd -Server $sqlServerInstance -Query "USE $databaseName UPDATE Progress SET $progressStage = 'Incomplete';" -Verbose:$false -ErrorAction Stop
Read-SqlTableData -ServerInstance $sqlServerInstance -DatabaseName "$databaseName" -SchemaName "dbo" -TableName "$tableName" -ErrorAction SilentlyContinue -Verbose:$falseThen run it in the ISE. The output from the last command should show the AddAppServicePreReqs stage as 'Incomplete'. Did that work? If so, close the ISE, ensure all PS windows are closed, then open a fresh PS Console/ISE and you should be good to try a rerun of the main script, and remember to use single quotes for containing passwords. |
|
I followed all those steps, however the rerun didn't go very well:
Shortly after it stopped running. Seems like it's not finding the script files, even thought they are there: |
|
This is bizarre, and I have no idea why that wouldn't work. There is no reason for that to fail.
|
|
Yes, that is all the case. I'm retrying it again in a new PowerShell window again. Instead of using the CD command I used set-environment, I'm not sure if that matters. Seems to work sort of better: |
|
It failed at the VM provisioning again, just like it did when I opened this issue. |
|
Could you email the AppService log from within the ASDK\appservice folder? I don’t think think this is script related. Either something is wrong with the base Windows Server 2016 Datacenter image, or the virtualization stack. I’ll review the logs and if required, share with the App Service team. |
|
I've send the log file. |
|
OK, I've sent it to out internal App Service on Azure Stack team, to help us figure out what's happening here. I'll follow up when I have more info, but don't delete anything just yet, so we can investigate further. Thanks! |
|
I think we can close this issue here? Since it also happens when installing App Services manually, so it's not script related. Then again, you may want to keep it open to avoid other people from reporting the same issue ;) |
|
Closed for now, we can always reopen later :) |
I've ran the post deployment script on a fresh ASDK with the following arguments:
The post deployment step for DeployAppService failed.
After checking the AppServiceLog it, it seems that a virtual machine is being deployed. But the deployment of this virtual machine failed.
From the: AppServiceLog1116-011218.txt:
The text was updated successfully, but these errors were encountered: