ZipDelpoy on azure web app linux fails during kudu sync

[SumiranAgg]

We are deploying a PHP Zip file to Azure App Service for Linux on a Premium (P1V2) Linux App Service Plan. It fails during the Kudu Sync step while processing files:

2019-05-08T04:06:36.2389539Z Command: "/home/site/deployments/tools/deploy.sh"
2019-05-08T04:06:36.2389838Z Handling Basic Web Site deployment.
2019-05-08T04:06:36.2390093Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot’
…
….
2019-05-08T04:06:36.2403894Z Processed 11560 files...
2019-05-08T04:06:36.2404111Z Processed 14025 files...
2019-05-08T04:06:36.2404338Z An unknown error has occurred. Check the diagnostic log for details.
2019-05-08T04:06:36.2478968Z ##[error]Failed to deploy web package to App Service.

Note that we deploy this exact same Zip file to identical (Premium P1V2) Windows instances without issue. This problem is intermittent, and specific to App Service For Linux deployments. We have seen this occur in multiple Azure Regions - so this isn't a flaky instance or zone-specific issue. It seems like a bug in the Azure App Service Deploy task itself, which only occurs on App Service For Linux deployments. Azure Support staff have mentioned that it appears to be causing a Docker Container restart, which is killing the deployment process.

[suwatch]

@rramachand21 can you assist this user?

[Ajay-MS]

Team,

We have got one more customer DTS on same scenario. Can you please take a look into this at priority

Snippet of logs

2019-05-13T23:48:15.2050106Z Error: From directory doesn't exist​
2019-05-13T23:48:15.2050349Z An error has occurred during web site deployment.​
2019-05-13T23:48:15.2050418Z Kudu Sync failed​
2019-05-13T23:48:15.2050481Z \n/opt/Kudu/bin/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"​
2019-05-13T23:48:15.2050604Z App container will begin restart within 10 seconds.​
2019-05-13T23:48:15.2107637Z ##[error]Failed to deploy web package to App Service.​
2019-05-13T23:48:15.2116824Z ##[debug]Processed: ##vso[task.issue type=error;]Failed to deploy web package to App Service.​
2019-05-13T23:48:15.2127512Z ##[debug]Deployment Failed with Error: Error: Package deployment using ZIP Deploy failed. Refer logs for more details.​
2019-05-13T23:48:15.2127716Z ##[debug]task result: Failed​
2019-05-13T23:48:15.2128148Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.​
2019-05-13T23:48:15.2128563Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.​
2019-05-13T23:48:15.2129561Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.​
2019-05-13T23:48:15.2148489Z ##[debug][POST]https://management.azure.com/subscriptions/<subscription Id>/resourceGroups/spreevel/providers/Microsoft.Web/sites/Spreedix2/config/appsettings/list?api-version=2016-08-01​

[katerd]

We are also having the same problem deploying to our Linux B1 app services. There are eight in separate app service plans, and have seen random failures across all of them. It is an intermittent issue and re-running the deployment eventually does succeed.

Example log:

2019-07-01T16:29:52.0074562Z Package deployment using ZIP Deploy initiated.
2019-07-01T16:30:07.7763815Z Fetching changes.
2019-07-01T16:30:08.1841003Z Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/7e26b992-ac80-4f1c-89ac-027684a25a66.zip (1.27 MB) to /tmp/zipdeploy/extracted
2019-07-01T16:30:08.1901108Z ##[error]Failed to deploy web package to App Service.
2019-07-01T16:30:08.1909376Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2019-07-01T16:30:11.6780546Z Successfully added release annotation to the Application Insight : xxx-yyy-zzz
2019-07-01T16:30:12.2726654Z Successfully updated deployment History at https://xxx-yyy-zzz.scm.azurewebsites.net/api/deployments/111111111111
2019-07-01T16:30:12.6429723Z App Service Application URL: https://xxx-yyy-zzz.azurewebsites.net
2019-07-01T16:30:12.6633655Z ##[section]Finishing: Deploy Site

[ManuelGalindo]

Hello Team

We have this problem is there a workaround?

2019-08-26T21:07:10.3457838Z ##[debug][GET]https://111111111.scm.azurewebsites.net/api/deployments/1111111111/log
2019-08-26T21:07:10.6186452Z ##[debug]getDeploymentLogs. Data: {"statusCode":200,"statusMessage":"OK","headers":{"transfer-encoding":"chunked","content-type":"application/json; charset=utf-8","server":"Kestrel","date":"Mon, 26 Aug 2019 21:07:09 GMT","connection":"close"},"body":[{"log_time":"2019-08-26T21:07:06.8214615Z","id":"11111","message":"Updating submodules.","type":0,"details_url":null},{"log_time":"2019-08-26T21:07:06.8681795Z","id":"99c3b156-2642-46b2-884f-fa65f7563e48","message":"Preparing deployment for commit id '70431e9f2f'.","type":2,"details_url":"https://111111111.scm.azurewebsites.net/api/deployments/111111/log/1111111"},{"log_time":"2019-08-26T21:07:07.6680215Z","id":"5e7e269a-6571-4ba0-b75d-71f8e268a541","message":"App container will begin restart within 10 seconds.","type":0,"details_url":null}]}
2019-08-26T21:07:10.6187508Z Updating submodules.
2019-08-26T21:07:10.6187756Z Preparing deployment for commit id '70431e9f2f'.
2019-08-26T21:07:10.6188204Z ##[debug]setting affinity cookie ["ARRAffinity=caee1681588a93469f1c8dc7165bba44fcf2d3c8ef87385b9a2f16ce75474466;Path=/;HttpOnly;Domain=11111111.scm.azurewebsites.net"]
2019-08-26T21:07:10.6188646Z ##[debug][GET]https://111111111111.scm.azurewebsites.net/api/deployments/11111111/log/11111111
2019-08-26T21:07:10.9967360Z ##[debug]getDeploymentLogs. Data: {"statusCode":200,"statusMessage":"OK","headers":{"transfer-encoding":"chunked","content-type":"application/json; charset=utf-8","server":"Kestrel","date":"Mon, 26 Aug 2019 21:07:10 GMT","connection":"close"},"body":[{"log_time":"2019-08-26T21:07:07.3547516Z","id":"","message":"Object reference not set to an instance of an object.","type":2,"details_url":null}]}
2019-08-26T21:07:10.9968511Z Object reference not set to an instance of an object.
2019-08-26T21:07:10.9968789Z App container will begin restart within 10 seconds.
2019-08-26T21:07:11.0044078Z ##[error]Failed to deploy web package to App Service.
2019-08-26T21:07:11.0054504Z ##[debug]Processed: ##vso[task.issue type=error;]Failed to deploy web package to App Service.
2019-08-26T21:07:11.0054886Z ##[debug]Deployment Failed with Error: Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2019-08-26T21:07:11.0055048Z ##[debug]task result: Failed)

[suwatch]

@ManuelGalindo your issue seemed different - please open a new gissue.

[ibuyauction]

We are also facing Same Issue. Works well with Widows WebApp and failed with Linux WebApp.

[JonathanBaileyTC]

Also having the same issue. Log snippet:

2019-10-07T17:52:10.6075020Z Processed 96267 files...
2019-10-07T17:52:10.6075086Z Processed 97524 files...
2019-10-07T17:52:10.6075200Z Processed 99527 files...
2019-10-07T17:52:10.6075282Z An unknown error has occurred. Check the diagnostic log for details.
2019-10-07T17:52:10.6160866Z ##[error]Failed to deploy web package to App Service.
2019-10-07T17:52:10.6171123Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2019-10-07T17:52:13.3119324Z Successfully added release annotation to the Application Insight : xxxxxxx-b9eb
2019-10-07T17:52:14.0541850Z Successfully updated deployment History at https://xxxxxxx.scm.azurewebsites.net/api/deployments/1791570470733307
2019-10-07T17:52:14.2915923Z App Service Application URL: http://xxxxxxx.azurewebsites.net
2019-10-07T17:52:14.3153074Z ##[section]Finishing: Deploy Azure Function App

[suren1525]

The same :(

2020-02-29T17:06:08.2983939Z Package deployment using ZIP Deploy initiated.
2020-02-29T17:26:43.0604255Z Updating submodules.
2020-02-29T17:26:43.0609942Z Preparing deployment for commit id '5990ad07ae'.
2020-02-29T17:26:43.0611103Z Repository path is /tmp/zipdeploy/extracted
2020-02-29T17:26:43.0611819Z Running oryx build...
2020-02-29T17:26:46.0687104Z Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform nodejs --platform-version 12 -i /tmp/8d7bd3a35c34740 -p compress_node_modules=tar-gz --log-file /tmp/build-debug.log
2020-02-29T17:26:46.0689704Z Build orchestrated by Microsoft Oryx, https://github.com/Microsoft/Oryx
2020-02-29T17:26:46.0690529Z You can report issues at https://github.com/Microsoft/Oryx/issues
2020-02-29T17:26:46.0691156Z
2020-02-29T17:26:46.0691731Z Oryx Version : 0.2.20191105.2, Commit: 67e159d71419415435cb5d10c05a0f0758ee8809, ReleaseTagName: 20191105.2
2020-02-29T17:26:46.0692333Z Build Operation ID: |7ga48LrNSwY=.8449380c_
2020-02-29T17:26:46.0692830Z Repository Commit : 5990ad07ae024887ac6441fac1124b1e
2020-02-29T17:26:46.0693129Z
2020-02-29T17:26:46.0693782Z Using intermediate directory '/tmp/8d7bd3a35c34740'.
2020-02-29T17:26:46.0694106Z
2020-02-29T17:26:46.0694482Z Copying files to the intermediate directory...
2020-02-29T17:26:46.0694845Z Done in 8 sec(s).
2020-02-29T17:26:46.0695080Z
2020-02-29T17:26:46.0695375Z Source directory : /tmp/8d7bd3a35c34740
2020-02-29T17:26:46.0695775Z Destination directory: /home/site/wwwroot
2020-02-29T17:26:46.0696060Z
2020-02-29T17:26:46.0696232Z
2020-02-29T17:26:46.0696537Z Using Node version:
2020-02-29T17:26:46.0696871Z v12.13.0
2020-02-29T17:26:46.0697002Z
2020-02-29T17:26:46.0697169Z Using Yarn version:
2020-02-29T17:26:46.0697370Z 1.17.3
2020-02-29T17:26:46.0697891Z
2020-02-29T17:26:46.0698086Z Configuring Yarn cache folder...
2020-02-29T17:26:46.0698336Z yarn config v1.17.3
2020-02-29T17:26:46.0698892Z success Set "cache-folder" to "/usr/local/share/yarn-cache".
2020-02-29T17:26:46.0699205Z Done in 0.62s.
2020-02-29T17:26:46.0699344Z
2020-02-29T17:26:46.0699740Z Running 'yarn install --prefer-offline'...
2020-02-29T17:26:46.0700059Z
2020-02-29T17:26:46.0700231Z yarn install v1.17.3
2020-02-29T17:26:46.0700451Z [1/4] Resolving packages...
2020-02-29T17:26:46.0700780Z [2/4] Fetching packages...
2020-02-29T17:26:46.0701440Z info There appears to be trouble with your network connection. Retrying...
2020-02-29T17:26:46.0701879Z info There appears to be trouble with your network connection. Retrying...
2020-02-29T17:26:46.0702293Z info There appears to be trouble with your network connection. Retrying...
2020-02-29T17:26:46.0703158Z info There appears to be trouble with your network connection. Retrying...
2020-02-29T17:26:46.0706927Z info There appears to be trouble with your network connection. Retrying...
2020-02-29T17:26:46.0746701Z info There appears to be trouble with your network connection. Retrying...
2020-02-29T17:26:46.0747312Z info fsevents@2.1.2: The platform "linux" is incompatible with this module.
2020-02-29T17:26:46.0754650Z info "fsevents@2.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
2020-02-29T17:26:46.0755529Z info fsevents@1.2.11: The platform "linux" is incompatible with this module.
2020-02-29T17:26:46.0756908Z info "fsevents@1.2.11" is an optional dependency and failed compatibility check. Excluding it from installation.
2020-02-29T17:26:46.0757538Z [3/4] Linking dependencies...
2020-02-29T17:26:46.0762029Z warning " > next-ga@2.3.4" has incorrect peer dependency "next@^6.0.0 || ^7.0.0".
2020-02-29T17:26:46.0763157Z warning " > react-clamp@1.0.5" has incorrect peer dependency "react@^0.14.3".
2020-02-29T17:26:46.0767314Z warning " > react-clamp@1.0.5" has incorrect peer dependency "react-dom@^0.14.3".
2020-02-29T17:26:46.0770416Z warning " > react-google-maps@9.4.5" has unmet peer dependency "@types/googlemaps@^3.0.0".
2020-02-29T17:26:46.0771552Z warning " > react-google-maps@9.4.5" has unmet peer dependency "@types/markerclustererplus@^2.1.29".
2020-02-29T17:26:46.0772491Z warning " > react-google-maps@9.4.5" has unmet peer dependency "@types/react@^15.0.0 || ^16.0.0".
2020-02-29T17:26:46.0773470Z warning "react-pdf > pdfjs-dist@2.1.266" has unmet peer dependency "webpack@^3.0.0 || ^4.0.0-alpha.0 || ^4.0.0".
2020-02-29T17:26:46.0774618Z warning "react-pdf > pdfjs-dist > worker-loader@2.0.0" has unmet peer dependency "webpack@^3.0.0 || ^4.0.0-alpha.0 || ^4.0.0".
2020-02-29T17:26:46.0777172Z warning " > react-simple-video-player@1.0.0" has unmet peer dependency "glamorous@^4.11.4".
2020-02-29T17:26:46.0778250Z warning " > react-simple-video-player@1.0.0" has unmet peer dependency "react-player@^1.1.1".
2020-02-29T17:26:46.0779210Z warning " > eslint-config-airbnb@18.0.1" has unmet peer dependency "eslint-plugin-react-hooks@^1.7.0".
2020-02-29T17:26:46.0779952Z An unknown error has occurred. Check the diagnostic log for details.
2020-02-29T17:26:46.0815835Z ##[error]Failed to deploy web package to App Service.
2020-02-29T17:26:46.0827631Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.

[mattmelton]

Currently crippled by this issue

[suwatch]

@sanchitmehta please assist.

[BjoernLaemmerzahl]

Facing the same issue.

I am amazed that just nobody gets any support.

[jamesfaceface]

We are also experiencing this occasionally.

Looking into the deployment logs in Kudu at the specified time, the suspicious entries are as follows:

--   | step title="**Deployment Lock Failure**" date="2020-07-22T07:18:40.081"!-- duration: 311ms --   | step title="Deployment [temp name removed] is Failed at 2020-07-22T07:18:40.2849353Z" date="2020-07-22T07:18:40.399" !-- duration: 6ms --   | /step !-- duration: 474ms --

step title="Error occurred" date="2020-07-22T07:39:26.085" type="error" text="Object reference not set to an instance of an object." stackTrace=" at Kudu.Contracts.Infrastructure.LockExtensions.LockOperation(IOperationLock lockObj, Action operation, String operationName, TimeSpan timeout) in /tmp/KuduLite/Kudu.Contracts/Infrastructure/LockExtensions.cs:line 51 at Kudu.Core.Deployment.FetchDeploymentManager.c__DisplayClass12_0.PerformBackgroundDeploymentb__1() in /tmp/KuduLite/Kudu.Core/Deployment/FetchDeploymentManager.cs:line 341"
<-- duration: 19ms -- step

Running the deployment a second time was successful.

[Gerrit-K]

@suwatch since you've replied first once this was reported last year, can you give us an overview of the current state of investigation of this issue?
Or @rramachand21 did you perhaps take a look at this in the meantime?
It seems to affect a notable number of customers and I can as well confirm that it's still an issue to this day.

[sanchitmehta]

@jamesfaceface We are actively working on fixing the above issue by next App Service release cycle which would be around end of September.

@Gerrit-K Can you paste the error trace ?

To give an idea of the issue: App Service uses an NFS based Volume to host the App's /home directory. Kudu on Linux uses this mounted NFS share to host locks used during deployment for various purposes (to ensure there is just on active deployment at a time, to lock the writes to the deployment status file). These locks don't work well when Kudu is hosted on multi-instances (mostly due to latency in NFS Based calls and our previous implementation of these locks being flock files), especially when devops polls the status of the deployment frequently while status is being continuously changed. We are working on revamping the locking system and make it more reliant.

Meanwhile you could try using the Run from package feature WEBSITE_RUN_FROM_PAKCAGE=1. Please note this makes the mounted share for the app as read only but improves the deployment reliability and the deployment time.

[Gerrit-K]

@sanchitmehta Thanks for the prompt response! It's good to hear that this issue is being actively worked on.

Unfortunately due to several reasons I am not able to share the trace, but I've seen an error similar to the one reported by @jamesfaceface: Deployment Lock Failure. In our case, we're running multiple apps on one service plan and if we deploy everything, most deployments fail at more or less exactly the 20 minute mark (which, based on your explanation, is probably a lock timeout?).

I've tested WEBSITE_RUN_FROM_PACKAGE=1 for those apps that don't need a writable file system and it seems to improve the situation currently, but that might be a volatile observation as these failures weren't deterministic for us. Still, thanks for the hint!

Edit: I have to withdraw my report on WEBSITE_RUN_FROM_PACKAGE=1. The deployments indeed went smoothly, but unfortunately it seems that this disables remote builds, because all of our apps are now complaining about missing dependencies (node_modules). We're configuring them with:

{
...
  SCM_DO_BUILD_DURING_DEPLOYMENT: 'true',
  ENABLE_ORYX_BUILD: 'true',
  BUILD_FLAGS: 'UseExpressBuild',
...
}

Looking forward for any updates (and of course hopefully a fix) on this topic.

[Gerrit-K]

Moving each app to its own dedicated app service plan reduced the occurrence frequency of this issue, but didn't completely diminish it. Out of 8 parallel app deployments, there was still one that failed. This time I also retrieved the exception stack trace:

Click here to view

2020-08-19T12:51:12    Error occurred, type: error, text: Object reference not set to an instance of an object., stackTrace:    at Kudu.Contracts.Infrastructure.LockExtensions.LockOperationAsync(IOperationLock lockObj, Func`1 operation, String operationName, TimeSpan timeout) in /tmp/KuduLite/Kudu.Contracts/Infrastructure/LockExtensions.cs:line 76
   at Kudu.Services.Deployment.DeploymentController.Deploy(String id) in /tmp/KuduLite/Kudu.Services/Deployment/DeploymentController.cs:line 135
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Kudu.Services.Web.Tracing.TraceMiddleware.Invoke(HttpContext context) in /tmp/KuduLite/Kudu.Services.Web/Tracing/TraceMiddleware.cs:line 64
2020-08-19T12:51:13      Outgoing response, type: response, statusCode: 400, statusText: BadRequest

@sanchitmehta is this the issue you've described and does the trace help you in any way or were you already aware of it?

[mattmelton]

I'm confident that we have NFS locking issues during deployment. For example, files in .python_packages are locked, even from rm:

WEBSITE_RUN_FROM_PACKAGE=1 is a poor option for us because we can't programatically rotate the function host keys with a read only file system or benefit from pycache/grpc files.

[Gerrit-K]

We are actively working on fixing the above issue by next App Service release cycle which would be around end of September.

@sanchitmehta
Any news on this? It's now October, but I didn't dare to revert back to our previous setup and test because this issue seems almost untouched since then :/

[Anmol007]

@sanchitmehta @SumiranAgg @suwatch We are also facing the problem having a random error for Kudu sync for subsequent deployments for Linux Python Azure Function App. This issue seems to sit open since last year. It is adversely affecting our developments and deployments on Linux Python Azure Function Apps. Could you please updates on this or provide a workaround solution? Thanks in advance!

@Gerrit-K @SumiranAgg @mattmelton Have you found any workaround for this issue?

Attempt 1

2020-11-04T17:06:57.4362725Z ##[section]Starting: Deploy Azure Function App
2020-11-04T17:06:57.4374239Z ==============================================================================
2020-11-04T17:06:57.4374594Z Task : Azure Functions
2020-11-04T17:06:57.4375018Z Description : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2020-11-04T17:06:57.4375400Z Version : 1.163.7
2020-11-04T17:06:57.4375674Z Author : Microsoft Corporation
2020-11-04T17:06:57.4375996Z Help : https://aka.ms/azurefunctiontroubleshooting
2020-11-04T17:06:57.4376379Z ==============================================================================
2020-11-04T17:06:58.0771407Z Got service connection details for Azure App Service:'name of app'
2020-11-04T17:06:59.0305223Z Trying to update App Service Application settings. Data: {"WEBSITES_ENABLE_APP_SERVICE_STORAGE":"true","WEBSITE_RUN_FROM_PACKAGE":"true"}
2020-11-04T17:06:59.2751323Z App Service Application settings are already present.
2020-11-04T17:07:01.0818186Z Package deployment using ZIP Deploy initiated.
2020-11-04T17:09:45.9907407Z Updating submodules.
2020-11-04T17:09:45.9908306Z Preparing deployment for commit id 'commit id'.
2020-11-04T17:09:45.9909209Z Generating deployment script.
2020-11-04T17:09:48.8441340Z Using cached version of deployment script (command: 'azure -y --no-dot-deployment -r "/tmp/zipdeploy/extracted" -o "/home/site/deployments/tools" --basic --sitePath "/tmp/zipdeploy/extracted"').
2020-11-04T17:09:48.8442007Z Running deployment command...
2020-11-04T17:09:50.4397280Z Command: "/home/site/deployments/tools/deploy.sh"
2020-11-04T17:09:50.4397690Z Handling Basic Web Site deployment.
2020-11-04T17:09:50.4398324Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2020-11-04T17:09:50.4398860Z Copying file: 'azure-pipelines.yml'
2020-11-04T17:09:50.4401765Z Copying file: 'azure-pipelines2.yml'
2020-11-04T17:09:50.4402719Z Deleting file: 'host.json'
2020-11-04T17:09:50.4403761Z Deleting directory: 'worker_venv/lib/python3.6/site-packages/statsmodels/tsa/base/pycache'
2020-11-04T17:09:50.4405095Z Error: ENOENT: no such file or directory, rmdir '/home/site/wwwroot/worker_venv/lib/python3.6/site-packages/statsmodels/tsa/base/pycache'
2020-11-04T17:09:50.4405948Z An error has occurred during web site deployment.
2020-11-04T17:09:50.4406474Z Kudu Sync failed
2020-11-04T17:09:50.4407066Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2020-11-04T17:09:50.4452537Z ##[error]Failed to deploy web package to App Service.
2020-11-04T17:09:50.4476186Z ##[error]To debug further please check Kudu stack trace URL : https://'name of app'.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
2020-11-04T17:09:50.4477867Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2020-11-04T17:09:51.5189816Z Successfully added release annotation to the Application Insight : ddw-test-ana-idm-func-code
2020-11-04T17:09:54.5333501Z Successfully updated deployment History at https://'name of app'.scm.azurewebsites.net/api/deployments/'deployment id'
2020-11-04T17:09:54.8685348Z App Service Application URL: http://'name of app'.azurewebsites.net
2020-11-04T17:09:54.8845492Z ##[section]Finishing: Deploy Azure Function App

---

Attempt 2

2020-11-04T17:06:57.4362725Z ##[section]Starting: Deploy Azure Function App
2020-11-04T17:06:57.4374239Z ==============================================================================
2020-11-04T17:06:57.4374594Z Task : Azure Functions
2020-11-04T17:06:57.4375018Z Description : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2020-11-04T17:06:57.4375400Z Version : 1.163.7
2020-11-04T17:06:57.4375674Z Author : Microsoft Corporation
2020-11-04T17:06:57.4375996Z Help : https://aka.ms/azurefunctiontroubleshooting
2020-11-04T17:06:57.4376379Z ==============================================================================
2020-11-04T17:06:58.0771407Z Got service connection details for Azure App Service:'name of app'
2020-11-04T17:06:59.0305223Z Trying to update App Service Application settings. Data: {"WEBSITES_ENABLE_APP_SERVICE_STORAGE":"true","WEBSITE_RUN_FROM_PACKAGE":"true"}
2020-11-04T17:06:59.2751323Z App Service Application settings are already present.
2020-11-04T17:07:01.0818186Z Package deployment using ZIP Deploy initiated.
2020-11-04T17:09:45.9907407Z Updating submodules.
2020-11-04T17:09:45.9908306Z Preparing deployment for commit id 'commit id'.
2020-11-04T17:09:45.9909209Z Generating deployment script.
2020-11-04T17:09:48.8441340Z Using cached version of deployment script (command: 'azure -y --no-dot-deployment -r "/tmp/zipdeploy/extracted" -o "/home/site/deployments/tools" --basic --sitePath "/tmp/zipdeploy/extracted"').
2020-11-04T17:09:48.8442007Z Running deployment command...
2020-11-04T17:09:50.4397280Z Command: "/home/site/deployments/tools/deploy.sh"
2020-11-04T17:09:50.4397690Z Handling Basic Web Site deployment.
2020-11-04T17:09:50.4398324Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2020-11-04T17:09:50.4398860Z Copying file: 'azure-pipelines.yml'
2020-11-04T17:09:50.4401765Z Copying file: 'azure-pipelines2.yml'
2020-11-04T16:46:35.0746125Z Deleting file: 'worker_venv/lib/python3.6/site-packages/openpyxl/chart/surface_chart.py'
2020-11-04T16:46:35.0748217Z Error: ENOENT: no such file or directory, unlink '/home/site/wwwroot/worker_venv/lib/python3.6/site-packages/openpyxl/chart/surface_chart.py'
2020-11-04T16:46:35.0748713Z An error has occurred during web site deployment.
2020-11-04T16:46:35.0748960Z Kudu Sync failed
2020-11-04T17:09:50.4407066Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2020-11-04T17:09:50.4452537Z ##[error]Failed to deploy web package to App Service.
2020-11-04T17:09:50.4476186Z ##[error]To debug further please check Kudu stack trace URL : https://'name of app'.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
2020-11-04T17:09:50.4477867Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.**
2020-11-04T17:09:51.5189816Z Successfully added release annotation to the Application Insight : ddw-test-ana-idm-func-code
2020-11-04T17:09:54.5333501Z Successfully updated deployment History at https://'name of app'.scm.azurewebsites.net/api/deployments/'deployment id'
2020-11-04T17:09:54.8685348Z App Service Application URL: http://'name of app'.azurewebsites.net
2020-11-04T17:09:54.8845492Z ##[section]Finishing: Deploy Azure Function App

[Gerrit-K]

@Gerrit-K @SumiranAgg @mattmelton Have you found any workaround for this issue?

@Anmol007 Unfortunately no. The only "workaround" was to use dedicated app service plans for each app service. Since then the locking errors drastically reduced in frequency (but still appear from time to time). But this of course comes means less flexibility and more costs, so it's not acceptable in the long run.

[mattmelton]

Have you found any workaround for this issue?

Environment independent ACR + Docker image deployment using the Python AppService base (
mcr.microsoft.com/azure-functions/python:3.0-python3.8-appservice).

We invested in vscode dev containers too. I'm confident I'll never go back to Oryx/Kudu deployments.

[appasaheb4]

facing issue

[NameGrey]

I had absolutely the same issue. At the beginning I had it time to time and 1 day it started to appear constantly. I was not able to find the solution and had to migrate to App Service with windows and make all the settings for my NextJs app like in the following article - https://parveensingh.com/next-js-deployment-on-azure-app-service.

[PolygeneLubricants]

Check if you have an environment variable set on your web app, under key PROJECT. This seemed to cause this issue for me.
After removing the PROJECT environment variable (and restarting the app), I could confirm in the generated deploy.sh that the value was removed, and deployment was successful again.

I started seeing the problem upgrading from .Net 6 to .Net 7 on a Blazor WebAssembly application.

[asbjornu]

I've just been hit by a similar problem to this as well:

Error: ENOTDIR: not a directory, scandir '/home/site/wwwroot/node_modules/classnames'
An error has occurred during web site deployment.
Kudu Sync failed
\n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
Deployment Failed. deployer = GITHUB_ZIP_DEPLOY deploymentPath = ZipDeploy. Extract zip.
Error: Failed to deploy web package to App Service.
Error: Deployment Failed, Package deployment using ZIP Deploy failed. Refer logs for more details.

[0Dmitry]

Not sure it is the same issue but I started to get strange error during web app startup. I use zip deployment with WEBSITE_RUN_FROM_PACKAGE = 1. The exact same package I deploy to azure works perfectly on the same node version locally and worked fine on azure a week ago. Now I am getting this error:

2023-09-26T09:05:13.182028937Z Starting OpenBSD Secure Shell server: sshd.
2023-09-26T09:05:13.393802701Z Starting periodic command scheduler: cron.
2023-09-26T09:05:13.471027445Z Could not find build manifest file at '/home/site/wwwroot/oryx-manifest.toml'
2023-09-26T09:05:13.471094746Z Could not find operation ID in manifest. Generating an operation id...
2023-09-26T09:05:13.471109446Z Build Operation ID: 29d27ec9-8757-4105-bd40-e747750d8bd8
2023-09-26T09:05:13.805856371Z Environment Variables for Application Insight's IPA Codeless Configuration exists..
2023-09-26T09:05:13.816235512Z Writing output script to '/opt/startup/startup.sh'
2023-09-26T09:05:13.936853343Z Running #!/bin/sh
2023-09-26T09:05:13.936885043Z 
2023-09-26T09:05:13.936898443Z # Enter the source directory to make sure the script runs where the user expects
2023-09-26T09:05:13.936903543Z cd "/home/site/wwwroot"
2023-09-26T09:05:13.936908343Z 
2023-09-26T09:05:13.936912743Z export NODE_PATH=/usr/local/lib/node_modules:$NODE_PATH
2023-09-26T09:05:13.936917443Z if [ -z "$PORT" ]; then
2023-09-26T09:05:13.936921943Z 		export PORT=8080
2023-09-26T09:05:13.936926844Z fi
2023-09-26T09:05:13.936931044Z 
2023-09-26T09:05:13.936935444Z PATH="$PATH:/home/site/wwwroot" node --unhandled-rejections=strict ./packages/cosmos-webservice/dist/app.js
2023-09-26T09:05:38.000471984Z /home/site/wwwroot/node_modules/@graphql-tools/load/cjs/load-typedefs/load-file.js:83
2023-09-26T09:05:38.000605186Z                 throw errors[0];
2023-09-26T09:05:38.001275094Z                 ^
2023-09-26T09:05:38.001286695Z 
2023-09-26T09:05:38.001291895Z Error: ENOTDIR: not a directory, scandir '/home/site/wwwroot/node_modules/yargs/yargs'
2023-09-26T09:05:38.001297795Z     at Object.readdirSync (node:fs:1527:3)
2023-09-26T09:05:38.001314495Z     at readdirWithFileTypes (/home/site/wwwroot/node_modules/@nodelib/fs.scandir/out/providers/sync.js:16:33)
2023-09-26T09:05:38.001320295Z     at Object.read (/home/site/wwwroot/node_modules/@nodelib/fs.scandir/out/providers/sync.js:10:16)
2023-09-26T09:05:38.001325795Z     at SyncReader.scandirSync [as _scandir] (/home/site/wwwroot/node_modules/@nodelib/fs.scandir/out/index.js:18:17)
2023-09-26T09:05:38.001331395Z     at SyncReader._handleDirectory (/home/site/wwwroot/node_modules/@nodelib/fs.walk/out/readers/sync.js:28:34)
2023-09-26T09:05:38.001337595Z     at SyncReader._handleQueue (/home/site/wwwroot/node_modules/@nodelib/fs.walk/out/readers/sync.js:23:18)
2023-09-26T09:05:38.001343195Z     at SyncReader.read (/home/site/wwwroot/node_modules/@nodelib/fs.walk/out/readers/sync.js:15:14)
2023-09-26T09:05:38.001348395Z     at SyncProvider.read (/home/site/wwwroot/node_modules/@nodelib/fs.walk/out/providers/sync.js:11:29)
2023-09-26T09:05:38.001353795Z     at ReaderSync.walkSync [as _walkSync] (/home/site/wwwroot/node_modules/@nodelib/fs.walk/out/index.js:20:21)
2023-09-26T09:05:38.001359295Z     at ReaderSync.dynamic (/home/site/wwwroot/node_modules/fast-glob/out/readers/sync.js:13:21) {
2023-09-26T09:05:38.001364896Z   errno: -20,
2023-09-26T09:05:38.001370196Z   syscall: 'scandir',
2023-09-26T09:05:38.001375496Z   code: 'ENOTDIR',
2023-09-26T09:05:38.001380896Z   path: '/home/site/wwwroot/node_modules/yargs/yargs'
2023-09-26T09:05:38.001385596Z }
2023-09-26T09:05:38.001404096Z 
2023-09-26T09:05:38.001409796Z Node.js v18.17.1

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano
Azure App Service

[PrinceTegaton]

Check if you have an environment variable set on your web app, under key PROJECT. This seemed to cause this issue for me. After removing the PROJECT environment variable (and restarting the app), I could confirm in the generated deploy.sh that the value was removed, and deployment was successful again.

I started seeing the problem upgrading from .Net 6 to .Net 7 on a Blazor WebAssembly application.

9 months after this comment, its what fixed my issues. Thanks @PolygeneLubricants