Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure App Service: Failed to Deploy. The underlying connection was closed: An unexpected error occurred on a send. #12444

Closed
sanar-microsoft opened this issue Feb 27, 2020 · 18 comments
Assignees

Comments

@sanar-microsoft
Copy link

@sanar-microsoft sanar-microsoft commented Feb 27, 2020

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Azure App Service deploy(4.163.3)

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

Environment

  • Server - Azure Pipelines or TFS on-premises?

    • If using Azure Pipelines, provide the account name, team project name, build definition name/build number: Account name: DevDiv, DevDiv, Release: Mindaro-RP/Rel-20200224.3-4
  • Agent - Hosted or Private:

    • If using private agent, provide the OS of the machine running the agent and the agent version: Windows_NT, 2.165.0

Issue Description

Task name: Deploy Azure App Service
Environment: Canary - East US 2
The Deploy Azure App Service task fails due to connection problems, but when the same code is published using Visual Studio things work.

Only happening for web apps in Canary (East US 2 EUAP) region

Task logs

##[debug]Exit code 4294967295 received from tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[debug]Exit code 4294967295 received from tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[warning]Make sure the machine is using TLS 1.2 protocol or higher. Check https://aka.ms/enableTlsv2 for more information on how to enable TLS in your machine.
##[error]Error: Error: Could not complete the request to remote agent URL 'https://.scm.azurewebsites.net/msdeploy.axd?site='.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error count: 1.

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

##[error]Error: Error: Could not complete the request to remote agent URL 'https://.scm.azurewebsites.net/msdeploy.axd?site='.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error count: 1.

@cynthiashlezar

This comment has been minimized.

Copy link

@cynthiashlezar cynthiashlezar commented Feb 27, 2020

Hi! We're getting this error on our release pipeline when trying to deploy a function app to Europe West

@chshrikh chshrikh assigned SumiranAgg, niadak and arjgupta and unassigned kmkumaran Feb 28, 2020
@NickolaiL

This comment has been minimized.

Copy link

@NickolaiL NickolaiL commented Feb 28, 2020

Hello, we are getting the same errors when trying to deploy a Web App on West Europe too.

@NikitaKudin

This comment has been minimized.

Copy link

@NikitaKudin NikitaKudin commented Feb 28, 2020

We are experiencing the same issue now. Help!

@mschouveller

This comment has been minimized.

Copy link

@mschouveller mschouveller commented Feb 28, 2020

same issue since yesterday

[error]Failed to deploy App Service.
[error]Error: Could not complete the request to remote agent URL 'https://xxxx.scm.azurewebsites.net/msdeploy.axd?site=xxxx'.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295

@willynagel

This comment has been minimized.

Copy link

@willynagel willynagel commented Feb 28, 2020

Same issue here, but only with the deployment of function apps.
Changing TLS level to 1.0 works as a workaround.

@rdeveen

This comment has been minimized.

Copy link

@rdeveen rdeveen commented Feb 28, 2020

Changing TLS level to 1.0 works as a workaround.

How did you do that?

Answer: In the Azure Portal, Blade TLS/SSL settings, changing the Minimal TLS version to 1.0.

Update: don't do this, insecure! See #12444 (comment)

@willynagel

This comment has been minimized.

Copy link

@willynagel willynagel commented Feb 28, 2020

Indeed, in the portal or in our case in the ARM templates before deploying the webapp.

@mschouveller

This comment has been minimized.

Copy link

@mschouveller mschouveller commented Feb 28, 2020

Thanks willynagel and rdeveen
Works for me

@michaelspinks

This comment has been minimized.

Copy link

@michaelspinks michaelspinks commented Feb 28, 2020

Thanks worked for us today. Fine on 1.2 prior to this.

@jayeshpatel6

This comment has been minimized.

Copy link

@jayeshpatel6 jayeshpatel6 commented Mar 2, 2020

The work around worked for us, anyone know if the issue has been fixed?

@rdeveen

This comment has been minimized.

Copy link

@rdeveen rdeveen commented Mar 2, 2020

This script is using [Net.ServicePointManager]::SecurityProtocol to check if Tls12 is enabled. To check if Tls12 is enabled on your build machine you can run this command in Powershell console.

PS C:\> [Net.ServicePointManager]::SecurityProtocol
Tls, Tls11, Tls12
PS C:\>

To enforce Tls12 you can add this keys in the registry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

No need to change TLS level to 1.0 in the Azure Portal.

@jvanderbiest

This comment has been minimized.

Copy link

@jvanderbiest jvanderbiest commented Mar 2, 2020

@rdeveen Strange on our build server we get the same protocols when running your PowerShell command, but we have the same issue.

@jaeger-csj

This comment has been minimized.

Copy link

@jaeger-csj jaeger-csj commented Mar 2, 2020

+1 for @rdeveen 's solution. Per this article, I believe that the Self-hosted Windows build agent software is targeting an earlier version of .Net. Therefore, it executes pipeline deploys using the TLS settings that are default to that version of .Net.

In that article, it suggests modifying the OS registry with the values referenced by @rdeveen and two more (all of which copied below). After I do that and re-run the [Net.ServicePointManager]::SecurityProtocol PS command, I get back "SystemDefault". As my OS is on .Net 4.7 (which uses TLS 1.2 by default), the build agent uses TLS 1.2 and we no longer have any pipeline deploy issues ran through this agent pool.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001
@pollax

This comment has been minimized.

Copy link

@pollax pollax commented Mar 4, 2020

In my case the script returns SystemDefault.

PS C:\> [Net.ServicePointManager]::SecurityProtocol
SystemDefault

My agent is on .NET 4.8, so I just rebooted the build agent and... 🎉 now it works again. Wtf?

@miguelEsteban

This comment has been minimized.

Copy link

@miguelEsteban miguelEsteban commented Mar 5, 2020

Same issue on build agent with 2012 R2

@JennyLawrance

This comment has been minimized.

Copy link

@JennyLawrance JennyLawrance commented Mar 6, 2020

We are aware of the issue. This was a result of App Service enforcing the minTlsVersion on the web app for the SCM endpoint. As folks pointed out already, this is a breaking change from a behavior perspective, even though it was well intended.

As some of you already figured it out, there are multiple ways to get unblocked while App Service team address this

  1. Update the build environment to use TLS 1.2.
  2. Consider updating the devops pipeline to use zip-deploy. The tools for zip deploy is a simple CURL, which has enough configuration on the client side tooling to pick a secure TLS version.
  3. Changing the site's minTlsVersion to 1.0 is not recommended, since that comes with its own issues. TLS 1.0 is not considered secure any more by any one. Use that option as a last resort knowing what it really implies.
@frankwinfaber

This comment has been minimized.

Copy link

@frankwinfaber frankwinfaber commented Mar 6, 2020

Had the same issue when trying to run a deployment on a App Service in West Europe, but only for our production environment.
The output of [Net.ServicePointManager]::SecurityProtocol on our build-server was "Ssl3, Tls", but after updating the registry is changed to "SystemDefault".

Temporarely switched to an Hosted Build Agent, but after reboot of the build server the deployment is running as smooth as before. :-)

@SumiranAgg

This comment has been minimized.

Copy link
Contributor

@SumiranAgg SumiranAgg commented Mar 9, 2020

Thanks @JennyLawrance for the update. Closing the issue.

@SumiranAgg SumiranAgg closed this Mar 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.