Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED #471

qmfrederik · 2015-08-31T09:14:23Z

I have the following build pipeline:

MSBuild task to build my web app using the /target:Package additional build arguments
Publish the .zip package as a build artifact (not really used in this scenario)
Azure Web App Deployment task where I deploy my web app to a test environment.

The Azure Web App Deployment task fails with the error message:
Web Deploy experienced a connection problem with the server and had to terminate the connection. Contact your server administrator if the problem persists. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.

The page that is linked does not contain any information about an error ERROR_CONNECTION_TERMINATED.

The text was updated successfully, but these errors were encountered:

qmfrederik · 2015-08-31T09:14:53Z

Here are the log files:

2015-08-31T08:58:24.9892263Z Executing the powershell script: C:\vsoagent\tasks\AzureWebPowerShellDeployment\1.0.15\Publish-AzureWebDeployment.ps1
2015-08-31T08:58:25.2860804Z Looking for Azure PowerShell module at C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1
2015-08-31T08:58:26.0360271Z AzurePSCmdletsVersion= 0.9.3
2015-08-31T08:58:26.0828987Z Get-ServiceEndpoint -Name <my-guid-here> -Context Microsoft.TeamFoundation.DistributedTask.Agent.Worker.Common.TaskContext
2015-08-31T08:58:26.1141468Z subscription= Quamotion (Production)
2015-08-31T08:58:26.1141468Z Get-X509Certificate -CredentialsXml <xml>
2015-08-31T08:58:26.1766425Z azureSubscriptionId= <my-guid-here>
2015-08-31T08:58:26.1766425Z azureSubscriptionName= Quamotion (Production)
2015-08-31T08:58:26.1766425Z azureServiceEndpoint= https://management.core.windows.net/
2015-08-31T08:58:26.1922669Z Set-AzureSubscription -SubscriptionName Quamotion (Production) -SubscriptionId <my-guid-here> -Certificate <cert> -Environment AzureCloud
2015-08-31T08:58:26.2703860Z Select-AzureSubscription -SubscriptionId <my-guid-here>
2015-08-31T08:58:26.2860095Z ConnectedServiceName= <my-guid-here>
2015-08-31T08:58:26.2860095Z WebSiteName= quamotion
2015-08-31T08:58:26.2860095Z Package= D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip
2015-08-31T08:58:26.3016339Z Slot= 
2015-08-31T08:58:26.3016339Z AdditionalArguments= 
2015-08-31T08:58:26.3016339Z packageFile= Find-Files -SearchPattern D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip
2015-08-31T08:58:26.3172576Z packageFile= D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip
2015-08-31T08:58:26.3172576Z Get-AzureWebSite -Name quamotion -ErrorAction SilentlyContinue
2015-08-31T08:58:32.8066037Z Publish-AzureWebsiteProject -Name "quamotion" -Package "D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip" 
2015-08-31T08:59:09.8584050Z ##[error]Web Deploy experienced a connection problem with the server and had to terminate the connection.  Contact your server administrator if the problem persists.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.

bryanmacfarlane · 2015-09-03T10:29:30Z

Was this a one time failure or is it persistent? Is it still reproducing?

qmfrederik · 2015-09-03T20:32:55Z

Hi Bryan,

The issue still reproduces, every time.

Frederik.

chrispat · 2015-09-04T02:44:35Z

Could you try and deploy your package using that command let from a powershell on your machine? I am not able to reproduce this.

bryanmacfarlane · 2015-09-04T13:36:25Z

@qmfrederik - also want to confirm that the build machine has network access to azure. is this on-prem TFS or VSO? We still can't repro so anything interesting network wise will help.

qmfrederik · 2015-09-04T15:10:51Z

@bryanmacfarlane - The project is part of a VSO (West Europe) account, the build agent is running on an Azure IaaS VM, also hosted in West Europe. The agent has been provisioned using Azure RM + PowerShell DSC, you can find the scripts here in case they should contain anything interesting: https://github.com/quamotion/powershell-dsc/blob/master/vso-agent/VsoAgent.ps1

@chrisrpatterson - Thanks, I'll give that a try and let you know.

qmfrederik · 2015-09-04T16:43:20Z

@chrisrpatterson I ran the Publish-AzureWebsiteProject command on the VM and that completed successfully.

PS D:\> Publish-AzureWebsiteProject -Name "quamotion" -Package "D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip"

VERBOSE: [Start]    Publishing package D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip
VERBOSE: [Complete] Publishing package D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip

So that seems to rule out a network connectivity

I authenticated to Azure using the Add-AzureAccount command. The VSO task uses the certificate to authenticate, so perhaps that triggers different behavior.

I've tried to authenticate to Azure using the certificate. However, I don't have the Get-X509Certificate command available in a standard Azure PowerShell. How would you suggest I go about and manually run the deployment in PowerShell, while mimicking as close as possible the commands that the task actually uses?

I did gave it a try on the Hosted agent, and did proceed. So at least the setup of the Azure subscription in VSO seems to be correct.

I'm kind of at loss, so I guess being either able to run the exact same PowerShell commands the task executes would help, or some additional logging in the tasks themselves.

chrispat · 2015-09-04T17:59:55Z

The fact it works in hosted build and on your work station but not form your build VM to me does indicate network on some level.

You can run the exact commands https://github.com/Microsoft/vso-agent-tasks/blob/master/Tasks/AzureWebPowerShellDeployment/Publish-AzureWebDeployment.ps1

From: Frederik Carlier [mailto:notifications@github.com]
Sent: Friday, September 4, 2015 12:43 PM
To: Microsoft/vso-agent-tasks vso-agent-tasks@noreply.github.com
Cc: Chris Patterson Chris.Patterson@microsoft.com
Subject: Re: [vso-agent-tasks] Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED (#471)

@chrisrpattersonhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fchrisrpatterson&data=01%7c01%7cChris.Patterson%40microsoft.com%7cbb51be3448e140a1668908d2b547f24e%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=8Ow92uPRFGjXwOtTmV0BgoDkZVklZxiDmmd51E%2bE9tU%3d I ran the Publish-AzureWebsiteProject command on the VM and that completed successfully.

PS D:> Publish-AzureWebsiteProject -Name "quamotion" -Package "D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip"

VERBOSE: [Start] Publishing package D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip

VERBOSE: [Complete] Publishing package D:\Build\d0a91b73\artifacts\drop\Quamotion.Website.zip

So that seems to rule out a network connectivity

I authenticated to Azure using the Add-AzureAccount command. The VSO task uses the certificate to authenticate, so perhaps that triggers different behavior.

I've tried to authenticate to Azure using the certificate. However, I don't have the Get-X509Certificate command available in a standard Azure PowerShell. How would you suggest I go about and manually run the deployment in PowerShell, while mimicking as close as possible the commands that the task actually uses?

I did gave it a try on the Hosted agent, and did proceed. So at least the setup of the Azure subscription in VSO seems to be correct.

I'm kind of at loss, so I guess being either able to run the exact same PowerShell commands the task executes would help, or some additional logging in the tasks themselves.

—
Reply to this email directly or view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fMicrosoft%2fvso-agent-tasks%2fissues%2f471%23issuecomment-137787475&data=01%7c01%7cChris.Patterson%40microsoft.com%7cbb51be3448e140a1668908d2b547f24e%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=v1RHqpJyKTHNhOt1kHybk9yY9a1G1J9PGw2g2q05edY%3d.

adriaanthomas · 2015-10-06T10:55:01Z

It looks like we're running into the same issue.

I have one build agent on Azure (West-Europe, VSO in West-Europe too) that works without any problems, but when I add a second one it keeps running into the above error. I compared the log output between the two (from the Azure Web App Deployment task) but they are exactly the same (apart from working directory which was on the D drive for the new agent, but I also tested that on C like the working agent, same result).

I am also able to run Publish-AzureWebsiteProject with the artefact from the build, on the faulty build VM (also after running Add-AzureAccount), so it does not appear to be a networking issue. Also, the hosted agent works, so it does not appear a configuration issue either.

When I compare the agent capabilities in VSO, they appear exactly the same. I also installed the build agent software from the same zip as the first agent, using the same VSO account, but it still does not work.

How can I help?

adriaanthomas · 2015-10-06T17:00:53Z

I do not have this issue when I create a build agent with size Standard_D2, but I do with size Standard_D1.

purplepiranha · 2015-10-10T14:02:06Z

This is definitely stressing me out!

I have the issue with my build agent running on the Basic_A1 size, but not with Basic_A2. I don't really want to have to run such a large instance purely because of deployment. I already limit it to running during daylight hours to save costs.

I can run everything ok through powershell, but via the build agent it fails.

It looks like it's a timeout issue on the Azure Wep App, but the logs tell me nothing.

purplepiranha · 2015-10-10T15:35:47Z

Also tried creating my own Azure Powershell script to do this rather than using the 'Azure Web App Deployment' build step. Still the same problem...

Works fine on the box, just not through VSO.

I'm wondering whether this a deliberate attempt by MS to make us either use their hosted build service (at a cost) or pay more for an Azure VM?

Why should web deploy need so much processor power? Surely it just copies the zip file across and then does all the work at the other end?

I'm using West Europe for both agent VM and the site that I'm deploying to. Has anyone had the same issue anywhere else?

qmfrederik · 2015-10-11T21:06:30Z

Hi @chrisrpatterson, @bryanmacfarlane

So, there's three of us now who can each consistently reproduce the problem with Azure VMs.
All of us host our VMs in the West Europe region.

May be related to the VM size (Basic_A1, Standard_A1 apparently don't work, whereas Basic_A2 and Standard_A2); at the same time, running the command from PowerShell does work.

Any thoughts? Can you reproduce this problem on, say, a Basic_A1 VM running in West Europe?

yaananth · 2015-10-30T19:00:52Z

sorry wasn't able to get to this. I created a VM to test this, i am on vacation for few weeks, but will try my best to atleast repro and we can get to it from there.

yaananth · 2015-12-18T16:07:20Z

So I did this -
Standard A1 size VM
West Europe

Against my own account in South Central US.

I didn't see any problem and webapp is deployed. So not sure what else we can do here.

I found a blog regarding this,
http://i1.blogs.msdn.com/b/benjaminperkins/archive/2015/09/15/error-connection-terminated-msdeploy-web-deploy.aspx

but, ofcourse no explanation other than saying some network problem

AndreaPic · 2015-12-22T10:40:52Z

Hi,
I'have the same issue

My environment is:

App Service plan/pricing tier

Pricing Tier
Free

Location

West Europe

Subscription name

Visual Studio Professional with MSDN

In Visual Studio On Line the endpoint configuration is configured with :
Type: Azure
Connecting to service using certificate

The same thing (Azure Web App Deployment) executed by a "Release Definition" has no error.

Thanks for any suggestions

yaananth · 2015-12-23T18:49:27Z

@AndreaPic what build agent are you using?

AndreaPic · 2016-01-07T17:25:46Z

Hi @yaananth
sorry for the late reply but I was on holiday.

I'm using Visual Studio On Line with free hosted agent, zero private agent then I have a free shared agent.
I haven't my private VM for the agent.

qmfrederik · 2016-01-07T18:21:20Z

Hi @yaananth

I believe we have also started seeing this issue using the hosted Visual Studio build service (so no custom VM). Our account is hosted in the Europe region.

I believe we've opened a ticket with product support for this, @bartsaintgermain should be able to give you the ticket # if that helps.

Frederik.

yaananth · 2016-01-08T00:10:45Z

@qmfrederik sure that would help. We are seeing some perf issues lately.

AndreaPic · 2016-01-13T16:44:57Z

Hi @yaananth,@qmfrederik,@bartsaintgermain
I think I found the problem.
I've made a build process with "Copy Publish Artifact" task before "Azure Deployment" task and in this configuration I reproduce the problem ("Copy Publish Artifact" work fine and "Azure Deployment" goes in "ERROR_CONNECTION_TERMINATED").
If I move the "Copy Publish Artifact" after "Azure Deployment", "Azure Deployment" work fine but "Copy Publish Artifact" is not executed (it remain gray).
It seems that there is an incompatibility between "Copy Publish Artifact" and"Azure Deployment".
What do you think ?

chrispat · 2016-01-13T16:56:22Z

Can you ping your account and send the result. I would be interested to know which scale unit you are on. Also which azure region are you targeting for your deployment?

From: AndreaPic [mailto:notifications@github.com]
Sent: Wednesday, January 13, 2016 11:45 AM
To: Microsoft/vso-agent-tasks vso-agent-tasks@noreply.github.com
Cc: Chris Patterson Chris.Patterson@microsoft.com
Subject: Re: [vso-agent-tasks] Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED (#471)

Hi @yaananthhttps://github.com/yaananth,@qmfrederikhttps://github.com/qmfrederik,@bartsaintgermainhttps://github.com/bartsaintgermain
I think I found the problem.
I've made a build process with "Copy Publish Artifact" task before "Azure Deployment" task and in this configuration I reproduce the problem ("Copy Publish Artifact" work fine and "Azure Deployment" goes in "ERROR_CONNECTION_TERMINATED").
If I move the "Copy Publish Artifact" after "Azure Deployment", "Azure Deployment" work fine but "Copy Publish Artifact" is not executed (it remain gray).
It seems that there is an incompatibility between "Copy Publish Artifact" and"Azure Deployment".
What do you think ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/471#issuecomment-171359113.

qmfrederik · 2016-01-13T17:01:29Z

Hi @chrisrpatterson

We’re on tfsprodweusu4.cloudapp.net and deploying to West Europe.

Hi @yaananth

Issue # was 116010513545526

AndreaPic · 2016-01-14T08:08:47Z

Hi @chrisrpatterson
We reproduce the problem on tfsprodweusu4.cloudapp.net [191.233.96.220] in West Europe.
Thanks

purplepiranha · 2016-01-14T11:43:55Z

It's good to see that this is being looked into. I haven't tried again recently as I've had a lot on.

I'm also on tfsprodweusu4.cloudapp.net and was deploying to West Europe.

KrishnaAdityaB · 2016-07-12T12:50:20Z

@qmfrederik @adriaanthomas @purplepiranha If you are still seeing this issue:

try the root cause mentioned at Azure web App deployment fails with "An error occured when request was processed on the remote computer" #1555
reopen this

bryanmacfarlane assigned yaananth Oct 6, 2015

yaananth added the Investigate label Oct 30, 2015

codito mentioned this issue Apr 22, 2016

Azure web App deployment fails with "An error occured when request was processed on the remote computer" #1555

Closed

bryanmacfarlane added the Area: Azure label Jun 12, 2016

KrishnaAdityaB closed this as completed Jul 12, 2016

bryanmacfarlane assigned KrishnaAdityaB Sep 9, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED #471

Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED #471

qmfrederik commented Aug 31, 2015

qmfrederik commented Aug 31, 2015

bryanmacfarlane commented Sep 3, 2015

qmfrederik commented Sep 3, 2015

chrispat commented Sep 4, 2015

bryanmacfarlane commented Sep 4, 2015

qmfrederik commented Sep 4, 2015

qmfrederik commented Sep 4, 2015

chrispat commented Sep 4, 2015

adriaanthomas commented Oct 6, 2015

adriaanthomas commented Oct 6, 2015

purplepiranha commented Oct 10, 2015

purplepiranha commented Oct 10, 2015

qmfrederik commented Oct 11, 2015

yaananth commented Oct 30, 2015

yaananth commented Dec 18, 2015

AndreaPic commented Dec 22, 2015

yaananth commented Dec 23, 2015

AndreaPic commented Jan 7, 2016

qmfrederik commented Jan 7, 2016

yaananth commented Jan 8, 2016

AndreaPic commented Jan 13, 2016

chrispat commented Jan 13, 2016

qmfrederik commented Jan 13, 2016

AndreaPic commented Jan 14, 2016

purplepiranha commented Jan 14, 2016

KrishnaAdityaB commented Jul 12, 2016

Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED #471

Azure Web App Deployment fails with ERROR_CONNECTION_TERMINATED #471

Comments

qmfrederik commented Aug 31, 2015

qmfrederik commented Aug 31, 2015

bryanmacfarlane commented Sep 3, 2015

qmfrederik commented Sep 3, 2015

chrispat commented Sep 4, 2015

bryanmacfarlane commented Sep 4, 2015

qmfrederik commented Sep 4, 2015

qmfrederik commented Sep 4, 2015

chrispat commented Sep 4, 2015

adriaanthomas commented Oct 6, 2015

adriaanthomas commented Oct 6, 2015

purplepiranha commented Oct 10, 2015

purplepiranha commented Oct 10, 2015

qmfrederik commented Oct 11, 2015

yaananth commented Oct 30, 2015

yaananth commented Dec 18, 2015

AndreaPic commented Dec 22, 2015

yaananth commented Dec 23, 2015

AndreaPic commented Jan 7, 2016

qmfrederik commented Jan 7, 2016

yaananth commented Jan 8, 2016

AndreaPic commented Jan 13, 2016

chrispat commented Jan 13, 2016

qmfrederik commented Jan 13, 2016

AndreaPic commented Jan 14, 2016

purplepiranha commented Jan 14, 2016

KrishnaAdityaB commented Jul 12, 2016