Zip deploy fails randomly on node_modules files for Linux web app

[AltarBeastiful]

Repro steps.

I'm building a Nodejs 10.14 webservice using Azure Devops and deploying on Azure web app.
It works in local or in Azure container, now i'm trying to deploy it in a classic Web App for the sake of simplicity.

Right now my Zip deploys are flaky, giving me errors 3 out of 4 times, always on node_modules files.

My webservice is a classic express backend. The bcrypt module is the only fancyness in it.
I'm deploying a zip containing all node_modules installed and files built.

Project structures.

The closest boilerplate I could find :
https://github.com/icebob/vue-express-sql-boilerplate

The log/error given by the failure.

~75% of my zip deployments including a node_module ends with an error on a random file:

2019-03-15T10:00:28.6009617Z Command: "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6010078Z Handling Basic Web Site deployment.
2019-03-15T10:00:28.6010454Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-03-15T10:00:28.6010711Z Copying file: '.dockerignore'
....
2019-03-15T10:00:28.6026397Z Copying file: 'dist/models/index.js'
2019-03-15T10:00:28.6026635Z Omitting next output lines...

2019-03-15T10:00:28.6026871Z Error: ENOENT: no such file or directory, utime '/home/site/wwwroot/node_modules/.bin/escodegen'

2019-03-15T10:00:28.6027124Z An error has occurred during web site deployment.
2019-03-15T10:00:28.6027354Z Kudu Sync failed
2019-03-15T10:00:28.6027580Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6027824Z App container will begin restart within 10 seconds.
2019-03-15T10:00:28.6113580Z ##[error]Failed to deploy web package to App Service.
2019-03-15T10:00:28.6123485Z ##[error]Error: Package deployment using ZIP Deploy failed. 

In the 5 failed deploy i had errors on: node_modules/.bin/uglifyjs, node_modules/.bin/sshpk-conv, node_modules/.bin/escodegen

Debug your Azure website remotely.

My sub is cb539346a and the site name qa***pi

Other informations

I also tried to exclude the node_modules and add npm install in my post deployment steps. This doesn't work as bcrypt fails to compile. I didn't try much to make it work as I found documentation indicating the prefered way is to publish the node_modules folder.

Right now I find it hard to publish a node app on Azure, is anybody else experiencing this ?

Thanks all !

[suwatch]

@michimune this is npm on Linux - could you help investigate this?

[satheeshpayoda]

I'm also facing this issue.
For me, KuduSync not able to copy following files under /home/site/wwwroot/node_modules/.bin
acorn
atob
errno

Even i modified the deployment script to delete whole folder using this command " rm -rf /home/site/wwwroot/node_modules "
Gettings this error:
rm: cannot remove '/home/site/wwwroot/node_modules/.bin': Directory not empty

Then i tried to ls from .bin folder:
total 0
-????????? ? ? ? ? ? acorn
-????????? ? ? ? ? ? atob
-????????? ? ? ? ? ? errno

Anyway to solve this?

---

this is npm on Linux - could you help investigate this?
Yes for me.

[blueelvis]

Try mounting a storage file share from a Storage account and see how it goes.

[satheeshpayoda]

Try mounting a storage file share from a Storage account and see how it goes.

How can i do that? No Storage account is associated with App service. I tried to check the /etc/fstab. But i can't check that from Kudu's bash.

[blueelvis]

Open Azure portal, find the web app and go to its application settings. Inside application settings, scroll to the bottom and you will see an option to mount storage.

…

On Tue 19 Mar, 2019, 9:21 AM Satheeshkumar Ravichandran, < ***@***.***> wrote: Try mounting a storage file share from a Storage account and see how it goes. How can i do that? No Storage account is associated with App service. I tried to check the /etc/fstab. But i can't check that from Kudu's bash. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2946 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGqYI0VeLNq2MaJVBKAP2lScCPcJM-iMks5vYF61gaJpZM4b2sK_> .

[satheeshpayoda]

Open Azure portal, find the web app and go to its application settings. Inside application settings, scroll to the bottom and you will see an option to mount storage.
…
On Tue 19 Mar, 2019, 9:21 AM Satheeshkumar Ravichandran, < @.***> wrote: Try mounting a storage file share from a Storage account and see how it goes. How can i do that? No Storage account is associated with App service. I tried to check the /etc/fstab. But i can't check that from Kudu's bash. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2946 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AGqYI0VeLNq2MaJVBKAP2lScCPcJM-iMks5vYF61gaJpZM4b2sK_ .

i don't see any mounted partition there. Sorry but Why do you want to mount Storage Account?

[satheeshpayoda]

I fixed this issue by increasing the Instance Count to 2 and after few mins i reverted back to 1. So old instance got removed and new one stayed. Now the same deployment script is working fine.

[AltarBeastiful]

Didn't try the instance count hack. We ended up going for Windows Web app.
Still it would be nice to fix this issue on Zip deploy for Linux.

[simkessy]

I'm getting this error all of a sudden:

ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/.bin/npm'

[ethubert]

I second this thread, any idea on how to solve this issue?

[suwatch]

@rramachand21, could you help triage this?

[brother-donkey]

Also experiencing this. Different node module but same problem.

2019-06-30T00:43:28.7646515Z Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/.bin/is-ci'

[rramachand21-zz]

@suwatch, we will investigate this.

[ivan133]

I also have this problem:

Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/bundle/programs/server/node_modules/.bin/js-yaml'

[sanchitmehta]

@AltarBeastiful @ivan133 @BrotherDonkey @ethubert @blueelvis : did it work before and is a recent problem, if yes since when did you start getting this problem? Could you please share your App Names so that we can investigate further ?

Alternatively, you could skip the local/pipeline build and add this App Setting SCM_DO_BUILD_DURING_DEPLOYMENT= true and Kudu will build the artifacts

[blueelvis]

@sanchitmehta - This has been happening since quite some time. The seems to arise because of so many files in an npm/php based project. When projects with less number of files is deployed using Zip-deployed, it doesn't happen. Seems like some sort of locking is done. We used to build locally and then push but it still used to fail.

I don't have access to the website now as it was with an earlier client.

[ivan133]

I'm pretty sure it's connected with symlinks that are used in npm modules

[satheeshpayoda]

@ivan133 i'm also suspecting the same. Because when i tried to delete whole folder, except symlink everything get deleted. Also when i changed the App Service Plan from Standard to Premium i can able to delete those symlink files without any issue.

[Laureian]

I'm also facing similar issue. I'm trying to deploy standard generated strapi.io project from Azure DevOps to Azure Web App. It's node.js app.
My steps:

1. Build package

• get project from github
• npm install
• create zip package
• publish zip package

2. Release project

• release zip package
• run npm start as a start command.
  Every task finishes successfully but when I try to access my app I always get errors like "Cannot find module '../package.json'".

What I found already:

• trying to release project without node_modules and then run npm install on Kudu but npm install on Kudu is terribly slow even for small projects
• changing Agent Poll from Hosted VS2017 to Hosted Ubuntu 1604 doesn't fix anything
• when I download zip, unzip it locally and try to run it's works on win 10 but not on macOS - when I delete node_modules/.bin it starts working on macOS
• it's for sure issue connected with differences between agent poll and kudu environment.

[brother-donkey]

I don't think this is the underlying issue, but my particular version of this problem was solved through the following steps:

1. Looking through the build, noticed that the build machine was a Windows image.
2. Changed it to hosted Ubuntu (16*), the same as the machine the app service is using.
3. Redeployed code via pushing to Github.

It did not work if yarn install was used to install things on the build machine. Again, can't say exactly why. Even though the project works well locally with yarn, the same commands on the build machine yield a different effect.

[t3custom]

I am having this exact same issue, and none of the suggestions are working for me. I suspect it has to do with Symlinks as well, but I can't confirm that.

[sanchitmehta]

We are investigating this issue and we would reply soon.

@Laureian are you using App service on Linux? we recently optimized the npm install times(via Kudu) on Linux platform. Have you tried Kudu Builds recently?

[Laureian]

Yup, I was using App Service on Linux. For now, I switched to Web App for Containers and it's working fine.

[danjrwalsh]

@sanchitmehta Any update on the investigation of this? I encountered this issue last week and can't seem to find a resolution. Switching to the Windows App Service isn't an option for my current project, unfortunately.

[Laureian]

I'm also waiting for an update as this is a blocker for Kudu Web apps. :(

[ethubert]

Context : I had to create a CI/CD pipeline for a project. The deployment failed on a linux machine on Azure.
Fix : Create a new service plan on a Windows Machine. Kudu has a problem with linux, from what I remember, the messages, while not being precise enough, seemed to indicate 1 of/or 2 things :

• No write rights for certain node modules
• Path length was too big, which causes the deployment to fail

Hope this helps anyone having problems with it

[fabiano]

I was having a similar problem with a PHP project that is using composer:

Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/vendor/bin/yii'

I had to remove the folders with symlinks from the zip package:

- task: DeleteFiles@1
  displayName: Delete vendor/bin folder from the staging directory
  inputs:
    sourceFolder: $(Build.StagingDirectory)
    contents: vendor/bin

[ustun]

@nshumoogum WEBSITES_RUN_FROM_PACKAGE should be WEBSITE_RUN_FROM_PACKAGE

[nshumoogum]

@ustun - i am embarrassed, yes this worked for me; i should have double checked the variable name

Thanks everyone who tried to help me :)

[jwr456]

this fixed my Python Flask issue

• script: |
  python3.6 -m --copies venv .env
  source .env/bin/activate
  pip3.6 install setuptools
  pip3.6 install -r requirements.txt

[RamunasAdamonis]

I was having the same issue (no such file or directory, open '/home/site/wwwroot/node_modules/.bin/sshpk-conv), and then switched to WEBSITE_RUN_FROM_PACKAGE=1. But after this app was not working anymore. Reason being that app startup failed with error: Cannot find module './oryx-appinsightsloader.js'. My Web App had application insights enabled and this was injecting appinsightsloader script to confugure app insights. But this does not work if you run from package. Workaround was to disable Application Insights on Web App and configure it manually in code.

[nitinmb]

@sanchitmehta - We are noticing similar error related to a missing module during the app startup - after using zip deploy option. The same version of code works fine in one environment, but gives an error related to a missing module in other.

Per your recommendation , setting "WEBSITE_RUN_FROM_PACKAGE=1" helped fix the issue. But this makes the FS read only.

Following up with you to see if the team has identified the root cause for the issue with zip deploy and possible timeline for fix rollout.

Thanks for your help.

[hicksdavies]

@sanchitmehta - We are too now experiencing the (no such file or directory, open '/home/site/wwwroot/node_modules/.bin/xxx ) error. I have tried everything, recreated the Linux App, added WEBSITE_RUN_FROM_PACKAGE=1 into app settings, but nothing fixes the issue. Any ideas ? This is also not intermittant it seems to be every time ? Thanks John

[hicksdavies]

@nitinmb have you got any further in solving ? After i added the setting "WEBSITE_RUN_FROM_PACKAGE=1" I too get a missing module error ?

[HoLengZai]

Facing the same issue. With Azure Web App on Linux

2020-02-13T13:20:28.940273214Z > svg-sprite --config sprite-config.json src/svgs/*.svg
2020-02-13T13:20:28.940277614Z
2020-02-13T13:20:29.588164612Z internal/modules/cjs/loader.js:797
2020-02-13T13:20:29.588189713Z throw err;
2020-02-13T13:20:29.588201314Z ^
2020-02-13T13:20:29.588206014Z
2020-02-13T13:20:29.588210314Z Error: Cannot find module '../lib/svg-sprite'
2020-02-13T13:20:29.588214615Z Require stack:
2020-02-13T13:20:29.588218715Z - /home/site/wwwroot/node_modules/.bin/svg-sprite
2020-02-13T13:20:29.588223015Z at Function.Module._resolveFilename (internal/modules/cjs/loader.js:794:15)
2020-02-13T13:20:29.588227315Z at Function.Module._load (internal/modules/cjs/loader.js:687:27)
2020-02-13T13:20:29.588231516Z at Module.require (internal/modules/cjs/loader.js:849:19)
2020-02-13T13:20:29.588235616Z at require (internal/modules/cjs/helpers.js:74:18)
2020-02-13T13:20:29.588239916Z at Object. (/home/site/wwwroot/node_modules/.bin/svg-sprite:25:15)
2020-02-13T13:20:29.588244616Z at Module._compile (internal/modules/cjs/loader.js:956:30)
2020-02-13T13:20:29.588248817Z at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
2020-02-13T13:20:29.588253017Z at Module.load (internal/modules/cjs/loader.js:812:32)
2020-02-13T13:20:29.588268718Z at Function.Module._load (internal/modules/cjs/loader.js:724:14)
2020-02-13T13:20:29.588272718Z at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10) {
2020-02-13T13:20:29.588276618Z code: 'MODULE_NOT_FOUND',
2020-02-13T13:20:29.588280918Z requireStack: [ '/home/site/wwwroot/node_modules/.bin/svg-sprite' ]
2020-02-13T13:20:29.588284919Z }
2020-02-13T13:20:29.601757090Z npm ERR! code ELIFECYCLE
2020-02-13T13:20:29.602693244Z npm ERR! errno 1

Any progress on this issue? I went to the Bash console, and the file exist

Thanks for your help.

[pat-flew]

I am having a similar issue when using WEBSITE_RUN_FROM_PACKAGE=1. Some of the symbolic links are not being set correctly during deployment

e.g.

/node_modules/html-webpack-plugin/node_modules/loader-utils/node_modules/.bin/json5

is being linked to

../../../json5/lib/cli.js

where it should be

../../../../../json5/lib/cli.js

consequently, the deploy fails with Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/html-webpack-plugin/node_modules/loader-utils/node_modules/.bin/json5'

[pat-flew]

I cleaned up the build artifact to only include packages necessary for production which had the side-affect of fixing this, as we don't need webpack there. But is the above expected behaviour? This is a build that was working previously (at least until it suddenly stopped working which is what led me to this issue)

[jbeckton]

Azure Sucks... thats the problem. On AWS people can get work done plain and simple. Microsoft's warped architectures does nothing but hold you back. Maybe we just need a few more layers of crap on top of the other layers of crap to fix it?

[capraynor]

@jbeckton
Agreed. Totally agreed.

[sanchitmehta]

@pat-flew we don't extract the zip package and recreate the symlinks. We use fuse-zip file system to mount this zip, when the app is run using RunFromPackage

[dsherret]

I've had so much trouble with Azure App Service on Linux. It's not a great experience setting up a node app there compared to deploying a .net application on a Windows service. I'm kind of upset about how much time I've wasted on this (I usually wouldn't express this on a GitHub issue, but it's a paid product). I'm concerned about what issues I may encounter in the future if setup was this bad.

For this specific problem I encountered, I managed to workaround this after a lot of trial and error.

Problem 1: After doing a deploy with symlinks, any deploy I tried after even without symlinks would fail, and web ssh didn't work. Was getting a EHOSTUNREACH error.
Solution: Open up the App Service in the portal, Development Tools -> Advanced Tools, then click on the "Bash" tab. For me, that worked for some reason and I was displayed a prompt. From there I was able to rm -rf the contents of the /home/site/wwwroot folder.

Problem 2: Since a deploy with symlinks wrecked the server, what could be done to not deploy a zip with symlinks?
Solution: I tried a lot of different things (including different deployment strategies) and ran into so many different problems. The only thing that ended up working for me was to delete all the symlinks and copy the target of the symlink to where the symlink used to be, then zip the folder for deployment to the server. This wasn't a big issue for me to do since the size of the files was small.

I used this powershell script in my azure pipeline to do this:

# Azure App Service Zip Deploy doesn't handle symlinks so inline
# all of them (https://github.com/projectkudu/kudu/issues/2946)
$links = dir . -recurse -force | ?{$_.LinkType}
foreach ($link in $links) {
  # store these before removing the link, otherwise the properties will go to null
  $source = $link.Target
  $dest = $link.FullName

  echo "Removing symbolic link at ${dest}..."
  (Get-Item $dest).Delete()

  echo "Copying ${source} to ${dest}..."
  Copy-Item -Path $source -Destination $dest -Recurse
}

[jake-subvrsive]

For people still running into this while trying to deploy a Node app from Github, I thought I'd add the easiest work around I've found.

I just created a startup.sh script in the root of my projects which runs npm rebuild to recreate all the node_modules symlinks followed by my actual project's run command. I then go into the app service configuration in the Azure portal and change the custom startup command to startup.sh.

This way the container will regenerate all of the symlinks before actually running your project. An example startup.sh script follows:

npm rebuild

npm run production

[sanchitmehta]

We have been working on a new Feature called App Cache. This feature is designed specifically to create a per instance copy of the App - while ensuring the symlinks are created during the extraction. This would also solve the Kudu Sync and rsync issues that we see on this thread. Currently, the copy of the app code is only mounted in a R/O volume but we plan to make this R/W in coming months. Please try this and open an Azure Support case with us so we can help you immediately

[BenjaminGolba]

Thanks @jake-subvrsive, I ended up with your solution which was indeed the simplest of all that I tried.

[stambunan]

@sanchitmehta Are there any updates on this? Do we still need to rely on the App Cache to get symlinks to work?

[georgeOsdDev]

@sanchitmehta
Is kudulite zipdeploy support extacting symLinks ?

Kudu version:
1.0.0.7 (e59ed50ca2)

When I deployed zip file contains symlinks, extracted symlink will result in a file with one line of content.

For example node_modules/.bin/nest is linked to ../@nestjs/cli/bin/nest.js will become a file contains ../@nestjs/cli/bin/nest.js

At local or GitHub Action Agent

$ ls -la node_modules/.bin | grep nest
lrwxrwxrwx   1 toshida toshida    26 Aug 15 10:26 nest -> ../@nestjs/cli/bin/nest.js

At Kudulite or Application Container

kudu_ssh_user@d22111dda621:~/site/wwwroot$  ls -la node_modules/.bin | grep nest
-rwxrwxrwx 1 nobody nogroup    26 Aug 18 07:11 nest
kudu_ssh_user@d22111dda621:~/site/wwwroot$ echo `cat node_modules/.bin/nest`
../@nestjs/cli/bin/nest.js
kudu_ssh_user@d22111dda621:~/site/wwwroot$ 

This file is not executable and cause error like below.

2022-08-18T07:24:09.794285335Z > my-new-nest-app@0.0.1 start /home/site/wwwroot
2022-08-18T07:24:09.794291835Z > nest start
2022-08-18T07:24:09.794295935Z 
2022-08-18T07:24:09.932729587Z /home/site/wwwroot/node_modules/.bin/nest: 1: /home/site/wwwroot/node_modules/.bin/nest: ../@nestjs/cli/bin/nest.js: not found
2022-08-18T07:24:09.943299691Z npm info lifecycle my-new-nest-app@0.0.1~start: Failed to exec start script

I found that there are WEBSITE_ZIP_PRESERVE_SYMLINKS option but it is not described official document.
https://github.com/Azure-App-Service/KuduLite/blob/dev/Kudu.Services/Deployment/PushDeploymentController.cs#L903

BTW, I followed below document and have workaround by using dist/main.js instead of using node_modules/.bin/nest
https://azureossd.github.io/2022/02/11/Nest-Deployment-on-App-Service-Linux/index.html

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano
Azure App Service