Azure App Service Deploy fails for .net core 2.0 with ERROR_FILE_IN_USE

[IvanAlekseev]

I am trying to deploy an Asp.Net Core 2.0 web app to Azure App Service and receive this error

2017-09-07T15:20:07.7379097Z ##[section]Starting: Deploy Azure App Service
2017-09-07T15:20:07.7569101Z ==============================================================================
2017-09-07T15:20:07.7569101Z Task         : Azure App Service Deploy
2017-09-07T15:20:07.7569101Z Description  : Update Azure Web App Services, Web App On Linux , Function Apps, Mobile Apps using Web Deploy / Kudu REST APIs
2017-09-07T15:20:07.7569101Z Version      : 3.3.14
2017-09-07T15:20:07.7569101Z Author       : Microsoft Corporation
2017-09-07T15:20:07.7569101Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-09-07T15:20:07.7569101Z ==============================================================================
2017-09-07T15:20:11.9902809Z Got connection details for Azure App Service:'██████████████'
2017-09-07T15:20:16.5780995Z [command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='d:\a\r1\a\██████████████\drop\██████████████.zip' -dest:auto,ComputerName='https://██████████████.scm.azurewebsites.net:443/msdeploy.axd?site=██████████████',UserName='********',Password='********',AuthType='Basic' -setParam:name='IIS Web Application Name',value='██████████████' -enableRule:AppOffline -retryInterval:6000 -retryAttempts:10 -enableRule:DoNotDeleteRule -userAgent:VSTS_e29e8f6a-6b5e-463d-ba6e-72dd7866010e_release_10_50_99_1
2017-09-07T15:20:17.1910281Z Info: Using ID '████████████████████████████████████' for connections to the remote server.
2017-09-07T15:20:21.1058805Z Info: Using ID '████████████████████████████████████' for connections to the remote server.
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\appsettings.Development.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\appsettings.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\Connected Services\Application Insights\ConnectedService.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.Contract.dll).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.Contract.pdb).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.deps.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.dll).
2017-09-07T15:21:24.7852633Z ##[error]Failed to deploy web package to App Service.
2017-09-07T15:21:24.7862628Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-09-07T15:21:24.7862628Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file '██████████████.dll' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

2017-09-07T15:21:24.7862628Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
2017-09-07T15:21:26.2210992Z Successfully updated deployment History at https://██████████████.scm.azurewebsites.net/deployments/501504797684782
2017-09-07T15:21:26.2321177Z ##[section]Finishing: Deploy Azure App Service

I've tried some workarounds from similar issue without any luck #1607

Any ideas on what should I do to fix this?

[vincent1173]

@IvanAlekseev , Can you please share the full logs for the release?

[IvanAlekseev]

@vincentdass added more info. Will it be enough?

[vincent1173]

@IvanAlekseev , Is there any web jobs enabled in App Service (including Application Insights)?
Did you enable Rename locked files option too?
Please share your app service name without reporting it explicitly

[IvanAlekseev]

@vincentdass - this is a dummy service name - http://errorfileinuse-msexample.azurewebsites.net and we have a problem with recommendations api.
We have app insights enabled and no other webjobs.
I will try with Rename locked files option and tell you the results, for now we deploy to staging and then swap with prod

[nphmuller]

Same issue here: http://dummy-fileinuse.azurewebsites.net/

Happens on both the staging and production variant of the api app. I have Rename locked files and Take App Offline enabled. Still get the error. Retrying the deployment usually works.
Application Insights is enabled in our application. Not as an App Service extensions. No Web Jobs either.

I have tried -retryAttempts:10 but not -retryInterval:6000. Will probably work as a workaround when I try the deployment with both.

[pan-wang]

could you please collect event log from AppService portal and share it? In general, a windows event will be logged once the file change of app_offline.htm is received and the IIS component try to gracefully shutdown the backend asp.net process.

[jeffputz]

There are at least three versions of this issue floating around, all with workarounds that don't work for everyone. It became an issue again with .NET Core v2 release time... perhaps the thing that changed has something to do with it? Having to change your build process and endure a lot of pain with every release isn't great for productivity.

[rmarinho]

Ok let's just talk here :) i have the same issue, workaround is manual turn off the slot.. i don't like that kills productivity ..

[hkusulja]

It worked for me before, and it stopped working about 2 months ago, and still have issues. My app is asp.net core v1 including the asp.net 4.6.1 , I did not change my app or deployment procedure. Issue still persist. Second build always passes. I have same issue for multiple VS TS projects / and azure web sites.

[vincent1173]

@hkusulja , Is App offline flag enabled?
Can you please try adding -retryAttempts:10 -retryInterval:6000 in Additional arguments of Web Deploy?
In this way, we increase the retry time for deployment.

[jeffputz]

I did that and it had no effect.

[TroySchmidt]

Having the same issue. Using a continuous deployment release definition in VSTS to AWS instance running IIS 8.5.9600.x + Windows 2012 R2 + .NET Core 1.1.2 x64. It randomly fails deploying because the file is still in use. I have tried the retryAttemps, retryInterval, put stop and start management tasks around the deploy. This is still happening regardless of any workarounds other than manually redeploy until it works.

[tomkerkhove]

Same issue here

[ColinM9991-zz]

Having the same issue on an Azure App Service environment with no fix on the retryAttempts and retryInterval.

It works fine with "app_offline.htm", but not "App_Offline.htm".
This is apparently fixed in a later version of the ASP.NET Core Module for IIS, but this isn't something that can exactly be installed on an Azure App Service node, and the "Rename locked files" option doesn't appear in VSTS.

Setting the 'MSDEPLOY_RENAME_LOCKED_FILES' app setting does the trick, but ideally it would be better to display a message to end users when the service is being updated.

[vincent1173]

@davidebbo, @pan-wang , to comment on the issue.
The casing issue in app_offline is fixed long back.

[pan-wang]

@ColinM9991 the casing issue was fixed this Jan. When did you hit this casing issue last time?

[ColinM9991-zz]

@panwang, it was hit both on Sunday, and yesterday (Monday).

[fmdufour]

@pan-wang I'm hitting this issue right now.

[pan-wang]

@fmdufour could you please collect server side windows event log (during the deploy) through portal and share it ?

[ColinM9991-zz]

@pan-wang

I've retried following an environment rebuild and it seems to be working now, I will put it under a bit of load once I am at home again and attempt a redeployment.

[PaitoAnderson]

Encountering this error as well today about an hour ago, it wasn't happening for over a month though.

I do have MSDEPLOY_RENAME_LOCKED_FILES enabled, and Sync Retry Attempts at 20.

[fmdufour]

@pan-wang Are you refering to eventlog.xml file? I couldn't find any relevant information on it.

[TroySchmidt]

I updated to .NET Core v1.1.4 and that seems to have fixed it. It does appear that I was having the casing issue otherwise.

[davidebbo]

@thomasrousset WEBSITE_RUN_FROM_ZIP has no chance of working unless you deploy using zipdeploy. Please see my previous responses.

[PrimeHydra]

@thomasrousset you'll also need to create a new release from your saved definition. (If you re-deploy an existing release it will use the older release definition.) That tripped me up once, just though I'd mention it.

I don't know if "Remove additional files at destination" might also be trying to delete locked files, maybe uncheck that as well.

Beyond that, I'm out of ideas--good luck! :)

[thomasrousset]

@davidebbo when you said 'using zipdeploy' I thought you were talking about 'using webdeploy'. That's not what you meant ? If not where do I set it ?

@PrimeHydra that's ok that's what I did. Thank you anyway :)

[davidebbo]

@thomasrousset zipdeploy is a different thing (details here).

Problem is that VSTS consuming zipdeploy is still work in progress as @SumiranAgg pointed at above.

So I probably caused confusion by bringing it up before VSTS is ready for it (my bad). But things will get better once VSTS supports it.

[BowserKingKoopa]

Zip deploy sounds like the solution to this, as well as just a simpler and cleaner way to deploy apps in general. I'll be sitting tight until Visual Studio supports it (excruciating Stop > Publish > Start cycle until then). Did we ever get an answer to why this is broken in Core but always worked before in previous version of asp.net and publish?

[davidebbo]

@BowserKingKoopa it's because Core loads assemblies in-place, while regular ASP.NET shadow copies everything (so they're not locked in-place). So in Core, it relies on App_Offline.htm to unload things, and evidently that does not work reliably enough.

[DaleMckeown]

@BowserKingKoopa Not that I'm aware of. I'm waiting for @pan-wang to report his ANCM findings based on his repro.

[thomasrousset]

@davidebbo may you show me where do you set the 'using zipdeploy' option please ?

[davidebbo]

@thomasrousset: @vincentdass or others working on VSTS can better answer this (I'm on the App Service side).

[vincent1173]

@thomasrousset , we have added this feature in current sprint and will be available in next deployment of 4.* task version.
As a workaround, you can use the curl command provided here to use the Zip Deploy command.
You can also pull from the repo and upload the task to your account.

[thomasrousset]

@vincentdass ok thank you for your answer. When do you plan to publish it ? Is it possible to share it whith us in private mode ?

[frankfuu]

@thomasrousset, I'm also waiting for this feature.

I think VSTS tries to release every 3 weeks and their last release was on 19th June
so i'm hoping it would be around 10th July?

[vincent1173]

@thomasrousset , you can pull from master branch, build the task and upload to your account.
To upload task, use tfx-cli

[vincent1173]

@frankfuu , @thomasrousset , Zip Deploy & RunFromZip feature is available in 4.* task version.
Please try out the same and share your feedback.

[frankfuu]

hi @vincentdass, just tried using Zip Deploy but it came back with an error

Publish using zip deploy option is not supported for msBuild package type.

Here is my msbuild command

Msbuild.exe "C:\vsts\MyTestPool\my-test-bd-agent-2\_work\1\s\MyCompany\src\UI\My.Project\MyCompany.My.Project.csproj" 
/nologo /nr:false 
/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true 
/p:SkipInvalidConfigurations=true 
/p:PackageLocation="C:\vsts\MyTestPool\my-test-bd-agent-2\_work\1\a\\WebApps\\MyCompany.My.Project\\"
/p:PrecompileBeforePublish=true /p:UseMerge=true 
/p:SingleAssemblyName=AppCode /p:WarningLevel=1 /p:platform="AnyCPU" /p:configuration="UAT" 
/p:VisualStudioVersion="15.0" /p:_MSDeployUserAgent="VSTS_15d68490-21eb-479b-8bf3-95d04d04d581_build_15_0"

Does it have to be in any special format? Any documentation?

[vincent1173]

@frankfuu , Can you please edit the build as follows:

• In Build Solution task, change the MSBuild Arguments to /p:DeployOnBuild=true /p:DeployDefaultTarget=WebPublish /p:WebPublishMethod=FileSystem /p:DeleteExistingFiles=True /p:publishUrl="$(System.DefaultWorkingDirectory)\\WebAppContent"
  

• Add Archive Task and change the inputs as follows:

  • Change Root folder or file to archive to $(System.DefaultWorkingDirectory)\\WebAppContent
  • Disable Prepend root folder name to archive paths option

[thomasrousset]

Hi,
@vincentdass I just tried the v4 with zipDeploy deployment and it works well on my side.
We'll keep on test it. Thank's for your work and the message to warn us about the v4 avaibility.

[frankfuu]

Hi @vincentdass ,

You're a champ, thanks for the detailed instructions. Got it working ;)

[vincent1173]

@pan-wang , Can you please share your findings on why App offline flag is not honored by IIS?

[pan-wang]

@vincentdass ANCM does honor appoffline. However, it depends on file change notification from OS. If FCN was not emitted or missed, nothing can be done by ANCM. This is the scenario for AppService.
In the past, WebDeploy only creates appoffline.htm file once instead of every deployment. This is why the retry logic does not help. We made change to webdeploy. That is webdeploy touches the appoffline file after deploying the appoffline and force more FCN. The new version of Webdeploy has already been shipped and we hope it will mitigate this deployment issue.

[ssdevref]

Hi I have the same issue deploying, I have tried:
<AppOffline>true</AppOffline>
<RetryAttemptsForDeployment>100</RetryAttemptsForDeployment>
<EnableMSDeployAppOffline>True</EnableMSDeployAppOffline>
I have two projects: a library and the site. The library is locked when I come to do a web deploy.

[vincent1173]

PR #8457 for consuming new version of MSDeploy as per Web Deploy team recommendation.

[vincent1173]

New Version of Web Deploy is used in App Service Deploy to avoid FILE_IN_USE as per web deploy team recommendation.
If you still face FILE_IN_USE error, we recommend you to move from Web Deploy to Run From Package.

Run From Package support is added in 4.* task version of Azure App Service Deploy task.

Run From Package helps in avoiding the FILE_IN_USE error.

If you face the below issue,
Publish using zip deploy option is not supported for msBuild package type., please follow the steps provided here to change the package format.

#8710 - Added recommended message in the error logs to move from Web Deploy to Run From Package.

Closing the issue as the above recommendations should fix this. Please feel free to re-open or start a new thread, if any assistance required.