npm install task (not msbuild) fails with EPERM on hosted build #323

Closed
rosieks opened this Issue Jul 14, 2015 · 102 comments

Projects

None yet
@rosieks
rosieks commented Jul 14, 2015

It looks like there are some problems with permissions while installing various packages over npm install.

When I run my build for the 1st time I received the following errors:

npm ERR! Error: EPERM, open 'C:\NPM\Cache\bb33797c-ache-minimatch-2-0-8-package-tgz.lock'
npm ERR! { [Error: EPERM, open 'C:\NPM\Cache\bb33797c-ache-minimatch-2-0-8-package-tgz.lock']
npm ERR! errno: 50,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\\NPM\\Cache\\bb33797c-ache-minimatch-2-0-8-package-tgz.lock' }
npm ERR! 

After that I just run my build for the 2nd time without any modifications and I got the following errors:

npm ERR! Error: EPERM, open 'C:\NPM\Cache\e360d466-Cache-inherits-2-0-1-package-tgz.lock'
npm ERR! { [Error: EPERM, open 'C:\NPM\Cache\e360d466-Cache-inherits-2-0-1-package-tgz.lock']
npm ERR! errno: 50,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\\NPM\\Cache\\e360d466-Cache-inherits-2-0-1-package-tgz.lock' }
npm ERR! 

I receiving that errors on VSO

@bryanmacfarlane
Member

the front page of the agent repo talks about the npm issue
https://github.com/Microsoft/vso-agent

@rosieks
rosieks commented Jul 14, 2015

Front page talks about installing vsoagent, but I have that issue on hosted pool/agent while running npm install task during build?

@bryanmacfarlane
Member

Ahhh. interesting. Are you installing from npm globally? (-g)

@rosieks
rosieks commented Jul 16, 2015

I don't think so. I just add Npm Install task to my build and setup working directory to folder where is my package.json

@lsoltys
lsoltys commented Jul 16, 2015

I have the same problem with 'Npm install' build step on Visual Studio Online using hosted pool agent to build.

@bryanmacfarlane
Member

OK - we will investigate today

@densto88

Hey guys, sorry you're hitting this issue. I think I see what the problem is, it should be resolved shortly.

Thanks,
Dennis

@bryanmacfarlane
Member

Closing since the fix is about to rollout to hosted

@bryanmacfarlane
Member

Re-opening for Dennis to confirm

@densto88

It looks like my original theory wasn't correct. Can either of you provide me your visualstudio.com account name so I can look up which data center you're account is in? You can email me at densto@microsoft.com if you're uncomfortable posting it here.

Other questions that may help us diagnose the issue:
Have you been able to do this exact step successfully in the past? If so, when did it last succeed and when did you first start seeing it fail?

Does the step fail for all npm packages or just certain ones? If so, which have you seen it succeed for and which failed?

Thanks,
Dennis

@lsoltys
lsoltys commented Jul 16, 2015

Hi, My VSO account name is rcs-pol. For the first time I used 'npm install' build step yesterday and I had these problems from this first time. The only successful step was about 3 hours ago. I thought you had solved the problem. However now this step fails again. Error from build log:
2015-07-16T13:06:32.9426115Z ##[error]npm ERR! EEXIST, open 'C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock'
2015-07-16T13:06:32.9436102Z ##[error]File exists: C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock
2015-07-16T13:06:32.9436102Z ##[error]Move it away, and try again.
2015-07-16T13:06:32.9446106Z ##[error]npm ERR! System Windows_NT 6.2.9200
2015-07-16T13:06:32.9456106Z ##[error]npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
2015-07-16T13:06:32.9496111Z ##[error]npm ERR! cwd C:\a\4f767487\EuradSB\src\EuradWebClient
2015-07-16T13:06:32.9516111Z ##[error]npm ERR! node -v v0.10.32
2015-07-16T13:06:32.9526111Z ##[error]npm ERR! npm -v 1.4.28
2015-07-16T13:06:32.9546110Z ##[error]npm ERR! path C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock
2015-07-16T13:06:32.9576107Z ##[error]npm ERR! code EEXIST
2015-07-16T13:06:32.9716105Z ##[error]npm ERR! errno 47
2015-07-16T13:06:33.6864108Z ##[error]npm
2015-07-16T13:06:34.1110534Z ##[error]Unexpected exit code 47 returned from tool npm.cmd

@bryanmacfarlane
Member

Our theory is currently this: http://stackoverflow.com/questions/26491311/lock-file-gets-in-the-way-when-trying-to-generate-yeoman-chrome-extension

If that is the issue, we need to update npm on our build image

@cwoolum
Contributor
cwoolum commented Jul 20, 2015

I've gotten this issue randomly as well. I've run into an issue previously where if multiple builds that call "npm install" run at the same time, the lock file from the first npm install will cause the second one to fail. I didn't have much time to try and fix the issue with NPM and ended up only being able to limit builds to one concurrent NPM install at once. I did read that you can override the global cache folder per run using --cache=/folder but didn't get much farther on it.

npm/npm#2500

@bryanmacfarlane
Member

We're you running multiple agents on the same machine (that might be another workaround). Interesting that you can set cache folder - we could set local to each agent. Although it seems like it's npm's job to protect's it's cache. We'll look at the workaround. thx

@cwoolum
Contributor
cwoolum commented Jul 20, 2015

I think local to each agent would be the best bet. My specific use case was with Jenkins and multiple agents but the idea was the same.

@AbraaoAlves

This happened to me too. A quick solution was to make a agent for a Linux machine already set up. The problem is that we have no control over the npm. Eg: install npm try to compile many packages by OS and depending on the package it may require admin access, installed tools on machine, or even a more advanced npm version (like node-sass package that requires npm ^ 2.x ). Things we can not do in HostedBuild

One of the things that can put an end to these problems will be docker support , which will give us control of the environment even in the hosted environment.

@brutaldev

Also having the same issues here, I have been unable to use npm as a pre-build step as it always fails for the same error number 4048 (but a different file path each time).

EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\gulp\3.9.0\package\package.json' [d:\a\src\AppProject.csproj]
     npm ERR!     at Error (native)
     npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\gulp\3.9.0\package\package.json']
     npm ERR!   errno: -4048,
     npm ERR!   code: 'EPERM',
     npm ERR!   path: 'C:\\NPM\\Cache\\gulp\\3.9.0\\package\\package.json',
     npm ERR!   parent: 'AppProject' }
     npm ERR! 
     npm ERR! Please try running this command again as root/Administrator.
@thechrisjohnson
Contributor

The image with this fix went out on Friday/Monday. Are you still seeing the permission issue?

@brutaldev

@thechrisjohnson Yes, I just queued up another build and still got the same failure message.

Build started 7/30/2015 4:39:31 PM.
...
npm install
npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! path C:\NPM\Cache\gulp\3.9.0\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048

Anything else you need from me to investigate further?

@thechrisjohnson
Contributor

I'm unable to repro this issue just running npm install on a test project I have. What packages are you trying to install when this occurs?

@brutaldev

@thechrisjohnson It works on my machine, but only fails when running the build in VSO using the Hosted Build Controller.

@thechrisjohnson
Contributor

Oh, this is a xaml based build?

@chrisrpatterson
Member

Can you provide a package.json file that repros that we can try?

From: Werner van Deventer [mailto:notifications@github.com]
Sent: Thursday, July 30, 2015 2:08 PM
To: Microsoft/vso-agent-tasks vso-agent-tasks@noreply.github.com
Subject: Re: [vso-agent-tasks] npm install task fail on installing various packages (#323)

@thechrisjohnsonhttps://github.com/thechrisjohnson It works on my machine, but only fails when running the build in VSO using the Hosted Build Controller.


Reply to this email directly or view it on GitHubhttps://github.com/Microsoft/vso-agent-tasks/issues/323#issuecomment-126421083.

@brutaldev

@thechrisjohnson
Created a codeless project just to build with Gulp, it has no assembly output.

Just some simple MSBUILD Exec tasks in the .csproj file.

<Project ToolsVersion="12.0" DefaultTargets="GulpBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
...
<PropertyGroup>
  <CompileDependsOn>
    $(CompileDependsOn);
    GulpBuild;
  </CompileDependsOn>
</PropertyGroup>
...
<Target Name="GulpBuild">
    <Exec Command="if not exist C:\Users\buildguest\AppData\Roaming\npm mkdir C:\Users\buildguest\AppData\Roaming\npm" />
    <Exec Command="npm install" />
    <Exec Command="node_modules\.bin\bower install" />
    <Exec Command="gulp release" />
  </Target>

Deploy copy steps and so forth omitted since it can't get past the npm install. This was based on a number of different sites:
http://www.codecadwallader.com/2015/03/15/integrating-gulp-into-your-tfs-builds-and-web-deploy/
http://blogs.msdn.com/b/nicktrog/archive/2015/03/05/adding-gulp-to-visual-studio-online-builds.aspx
http://www.davepaquette.com/archive/2015/04/08/integrating-gulp-and-bower-with-visual-studio-online-hosted-builds.aspx

You can use any packages I guess, it fails with different names on each build so it appears random as to when exactly it fails. Here is a sample of my package.json.

{
  "name": "MyProject",
  "private": "true",
  "version": "0.0.1",
  "description": "",
  "dependencies": {},
  "devDependencies": {
    "bower": "^1.4.1",
    "browser-sync": "^2.7.12",
    "del": "~1.2.0",
    "gulp": "^3.9.0",
    "gulp-autoprefixer": "~2.3.1",
    "gulp-concat": "~2.5.2",
    "gulp-consolidate": "~0.1.2",
    "gulp-csso": "~1.0.0",
    "gulp-declare": "~0.3.0",
    "gulp-ext-replace": "^0.2.0",
    "gulp-foreach": "^0.1.0",
    "gulp-handlebars": "^4.0.0",
    "gulp-hb": "^2.6.0",
    "gulp-if": "~1.2.5",
    "gulp-imagemin": "~2.2.1",
    "gulp-jsbeautifier": "0.0.8",
    "gulp-jshint": "~1.11.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-minify-html": "~1.0.3",
    "gulp-minify-inline": "^0.1.0",
    "gulp-newer": "~0.5.0",
    "gulp-ng-annotate": "^1.0.0",
    "gulp-plumber": "~1.0.1",
    "gulp-print": "~1.1.0",
    "gulp-sass": "^2.0.3",
    "gulp-uglify": "~1.2.0",
    "gulp-usemin": "~0.3.11",
    "gulp-wrap": "~0.11.0",
    "imagemin-jpegtran": "^4.2.0",
    "imagemin-pngquant": "~4.1.0",
    "jshint-stylish": "~2.0.0",
    "merge-stream": "^0.1.7",
    "run-sequence": "~1.1.1"
  }
}
@thechrisjohnson
Contributor

Thanks! I was able to repro the issue and will push out a fix within the next day or so.

@brutaldev

Awesome! Let me know when it's done and I'll give it a test.

@bryanmacfarlane
Member

It would also be good to try the same thing from the new build system (+ definition from the web view)
http://aka.ms/vsopreview

@rosieks
rosieks commented Jul 30, 2015

I've tried to run my orginal build (based on new build system) and now NPM part works.

@bryanmacfarlane
Member

Awesome, thanks for confirming

@brutaldev

Unfortunately not as lucky for me.

Gets a lot further but still eventually gets caught out in the middle of the npm install after 9 minutes:

##[error]EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json'
2>EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json' [C:\a\c930acd7\MyProject\MyProject.csproj]
npm ERR!     at Error (native)
npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json']
npm ERR!   errno: -4048,
npm ERR!   code: 'EPERM',
npm ERR!   path: 'C:\\NPM\\Cache\\process-nextick-args\\1.0.2\\package\\package.json',
npm ERR!   parent: 'readable-stream' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.
@chrisrpatterson
Member

It looks to me like you are running NPM from within MSBuild. Is that the case? If so can you try adding /m:1 to the MSBuild args in your step?

From: Werner van Deventer [mailto:notifications@github.com]
Sent: Thursday, July 30, 2015 4:08 PM
To: Microsoft/vso-agent-tasks vso-agent-tasks@noreply.github.com
Cc: Chris Patterson Chris.Patterson@microsoft.com
Subject: Re: [vso-agent-tasks] npm install task fail on installing various packages (#323)

Unfortunately not as lucky for me.

Gets a lot further but still eventually gets caught out in the middle of the npm install after 9 minutes:

##[error]EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json'

2>EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json' [C:\a\c930acd7\MyProject\MyProject.csproj]

npm ERR! at Error (native)

npm ERR! { [Error: EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json']

npm ERR! errno: -4048,

npm ERR! code: 'EPERM',

npm ERR! path: 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json',

npm ERR! parent: 'readable-stream' }

npm ERR!

npm ERR! Please try running this command again as root/Administrator.


Reply to this email directly or view it on GitHubhttps://github.com/Microsoft/vso-agent-tasks/issues/323#issuecomment-126466001.

@brutaldev

Thanks for the suggestion. I've changed a couple things and made sure that it builds perfectly on my machine with the same arguments as the new build/deploy setup.

Unfortunately it still doesn't get past the npm install step:

  npm ERR! Windows_NT 6.3.9600
  npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
  npm ERR! node v0.12.7
  npm ERR! npm  v2.11.3

  npm ERR! shasum check failed for C:\Users\BUILDG~1\AppData\Local\Temp\npm-2864-a6acc8bb\registry.npmjs.org\browser-sync-ui\-\browser-sync-ui-0.5.13.tgz
  npm ERR! Expected: b3c1661824efa957fc6a7a6dcb8db95369b6d38a
  npm ERR! Actual:   cd71ba0b316389da3b236879b63eb2a328065a38
  npm ERR! From:     https://registry.npmjs.org/browser-sync-ui/-/browser-sync-ui-0.5.13.tgz
  npm ERR! 
EXEC(0,0): Error at: 
EXEC : npm ERR! If you need help, you may report this error at:  [C:\a\06a92b97\MyProject\MyProject.csproj]
  npm ERR!     <https://github.com/npm/npm/issues>

@brutaldev

Forced another build just to check it it was a dodgy download causing the SHA failure. Back with the permission errors again.

EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\once\1.3.2\package\package.json'
EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\once\1.3.2\package\package.json' [C:\a\06a92b97\MyProject\MyProject.csproj]
  npm ERR!     at Error (native)
  npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\once\1.3.2\package\package.json']
  npm ERR!   errno: -4048,
  npm ERR!   code: 'EPERM',
  npm ERR!   path: 'C:\\NPM\\Cache\\once\\1.3.2\\package\\package.json',
  npm ERR!   parent: 'end-of-stream' }
  npm ERR! 
  npm ERR! Please try running this command again as root/Administrator.
@chrisrpatterson
Member

I believe we have this tracked down now. For some reason the permissions on that directory were modify instead of full control. We have a new image building and will be rolling in out tonight.

@brutaldev

Just wanted to give some more feedback. Tested another build an hour ago and still getting the permission error. Different package this time but it's usually quite random.

EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json'
EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json' [C:\a\06a92b97\MyProject\MyProject.csproj]
  npm ERR!     at Error (native)
  npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json']
  npm ERR!   errno: -4048,
  npm ERR!   code: 'EPERM',
  npm ERR!   path: 'C:\\NPM\\Cache\\inherits\\2.0.1\\package\\package.json',
  npm ERR!   parent: 'readable-stream' }
  npm ERR! 
  npm ERR! Please try running this command again as root/Administrator.

  npm ERR! Please include the following file with any support request:
  npm ERR!     C:\a\06a92b97\MyProject\npm-debug.log

MyProject.csproj(156,5): Error MSB3073: The command "npm install" exited with code -4048.

Let me know if there is anything else I need to provide to assist you in identifying a fix.

@Peter-Juhasz

Same here:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--loglevel" "error"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! path C:\NPM\Cache\inherits\2.0.1\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! Error: EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\\NPM\\Cache\\inherits\\2.0.1\\package\\package.json',
npm ERR! parent: 'readable-stream' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.
npm ERR! Please include the following file with any support request:
npm ERR! C:\a\83f54761\REMOVED\npm-debug.log
Unexpected exit code -4048 returned from tool npm.cmd
@Zenuka
Zenuka commented Aug 27, 2015

Any updates on this issue? We're having the same problems with our VSO vNext build.

@Peter-Juhasz

+1 Sometimes succeeds but most of the time fails.

@thechrisjohnson
Contributor

We're working on deploying a fix to our hosted agent pools to fix this issue, it should be going out over the next few days.

@Peter-Juhasz

@thechrisjohnson Can you notify us when the deployment is complete? Thanks.

@Bartmax
Bartmax commented Sep 1, 2015

I'm experiencing the same issues right now. Is it random? ill try a couple more builds

@bryanmacfarlane
Member

@Bartmax - are you experiencing on the hosted pool windows build?

@Bartmax
Bartmax commented Sep 1, 2015

@bryanmacfarlane
I'm new to build so the terminology I'm not sure.
I'm using build definitions not xaml definitions.
Picked from deployment Azure Website.
and using default hosted queue

I confirm that queuing again worked (after 3 time permission error in a row). (so I think it fails sometimes)

Let me know if you need more info please, I'm more than glad to help.

@Bartmax
Bartmax commented Sep 1, 2015

and I think it's windows because it calling program files directories for npm install if that was your question.

@ChristianRygg

We are still experiencing this problem intermittently. When setting /m:1 it has become less frequent, but we still see it now and then.

I assume the fix mentioned earlier has been fully rolled out by now?

@bryanmacfarlane
Member

Yes, the fix has been rolled out. Are you seeing EPERM intermittently? I also have folks that see intermittent connection issues. Hosted build is more susceptible to that being a new machine everytime as opposed to incremental build which would go to npm much less often.

@ChristianRygg

Yes, we are seeing this sometimes (In fact, as of late it occurs more often than not):

2015-09-08T08:20:06.7233736Z EnsureNode:
2015-09-08T08:20:06.7243740Z   NodeJS path is: 'C:\NPM\Modules'
2015-09-08T08:20:06.7243740Z RunNpm:
2015-09-08T08:20:06.7243740Z   "C:\NPM\Modules\npm" install --loglevel error --production
2015-09-08T08:21:43.8125851Z   npm ERR! Windows_NT 6.3.9600
2015-09-08T08:21:43.8146306Z   npm ERR! argv "node" "C:\\NPM\\Modules\\node_modules\\npm\\bin\\npm-cli.js" "install" "--loglevel" "error" "--production"
2015-09-08T08:21:43.8146306Z   npm ERR! node v0.12.7
2015-09-08T08:21:43.8155847Z   npm ERR! npm  v2.14.2
2015-09-08T08:21:43.8155847Z   npm ERR! path C:\NPM\Cache\readable-stream\1.0.33\package\package.json
2015-09-08T08:21:43.8165844Z   npm ERR! code EPERM
2015-09-08T08:21:43.8175844Z   npm ERR! errno -4048
2015-09-08T08:21:43.8175844Z   
2015-09-08T08:21:43.8356335Z ##[error]EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\readable-stream\1.0.33\package\package.json'
2015-09-08T08:21:43.8365846Z EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\readable-stream\1.0.33\package\package.json' [C:\[Projectpath]\[ProjectName].csproj]
2015-09-08T08:21:43.8376459Z   npm ERR!     at Error (native)
2015-09-08T08:21:43.8386574Z   npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\readable-stream\1.0.33\package\package.json']
2015-09-08T08:21:43.8396077Z   npm ERR!   errno: -4048,
2015-09-08T08:21:43.8405934Z   npm ERR!   code: 'EPERM',
2015-09-08T08:21:43.8405934Z   npm ERR!   path: 'C:\\NPM\\Cache\\readable-stream\\1.0.33\\package\\package.json',
2015-09-08T08:21:43.8416463Z   npm ERR!   parent: 'bl' }
2015-09-08T08:21:43.8426216Z   npm ERR! 
2015-09-08T08:21:43.8426216Z   npm ERR! Please try running this command again as root/Administrator.
2015-09-08T08:21:44.1325862Z   
2015-09-08T08:21:44.9570537Z   npm ERR! Please include the following file with any support request:
2015-09-08T08:21:44.9570537Z   npm ERR!     C:\[Projectpath]\npm-debug.log
2015-09-08T08:21:44.9610534Z ##[error]packages\MSBuild.Npm.0.4.4\build\MSBuild.Npm.targets(11,9): Error MSB3073: The command ""C:\NPM\Modules\npm" install --loglevel error --production" exited with code -4048.
@bryanmacfarlane
Member

Are you doing it as an npm task (in the build definition) or is npm being called as part of your msbuild step?

If it's an npm task:
As an experiment (if possible) try two npm tasks back to back with the first being continue on error (checkbox on task). What I'm curious about is when it fails - is it something consistent on that VM (permissions/state) or is it simply something intermittent in npm?

@ChristianRygg

It is being called as a part of the build definition. The build runs a solution with three different web apps with different sets of modules. So we do already call npm three times in a build.

What we see, is that if varies what web app that gives this error, and sometimes more than one fails this way. There is no pattern: Sometimes the first project built gives the error, other times it's the second or last.

I could imagine that running the builds in paralell would cause problems, but even with /m:1 the problems continue.

@rdnscr
rdnscr commented Sep 16, 2015

I am facing the same issues are there any fixes available soon or will it anyhow be gone with the next node version?

Also on my environment removing parallel builds did not help. With this option it's just a little bit less likely the build will fail because of the described error.

@bryanmacfarlane bryanmacfarlane changed the title from npm install task fail on installing various packages to npm install task (not msbuild) fails with EPERM on hosted build Sep 16, 2015
@bryanmacfarlane
Member

This issue is tracking the npm task that this repo delivers.

The error given on the last report is from msbuild in an msbuild target.

##[error]packages\MSBuild.Npm.0.4.4\build\MSBuild.Npm.targets(11,9): Error MSB3073: The command ""C:\NPM\Modules\npm" install --loglevel error --production" exited with code -4048

@ichivers

I have a NPM Install build step which gives this error, but the package varies:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! path C:\NPM\Cache\inflight\1.0.4\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! Error: EPERM, rename 'C:\NPM\Cache\inflight\1.0.4\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM, rename 'C:\NPM\Cache\inflight\1.0.4\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\NPM\Cache\inflight\1.0.4\package\package.json',
npm ERR! parent: 'glob' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.
npm ERR! Please include the following file with any support request:
npm ERR! C:\a\74efc3b8\G101\src\G101.App\npm-debug.log
Unexpected exit code -4048 returned from tool npm.cmd

Have tried adding another NPN Install step below the first one and setting the first one to continue on error, but still get the same result. Only had 1 build succeed so far!

@Peter-Juhasz

Our success rate is 17 out of 38, less than 50%. Most of the time "run as administrator" error, sometimes "ECONNRESET".

At the moment our UI builds are completely unreliable. Really frustrating...

@bryanmacfarlane
Member

Just trying to narrow. As a troubleshooting mechanism, can you setup your own agent ( http://aka.ms/tfbuild has instructions ) in your own pool and try a bunch of builds. I'm trying to narrow down whether it's the networking in the hosted pool or it's something with the build process and/or npm.

you can email me and I can take each case and drill into specifics (hard to do here in a forum env)

@Peter-Juhasz

@bryanmacfarlane What's your e-mail?

@bryanmacfarlane
Member

my alias is bryanmac and you know where I work :)

@bryanmacfarlane
Member

We've made quite a bit of progress in at least understanding the issue. Workarounds are at the bottom as we wait for the root issue.

Here is the related npm issue:
npm/npm#9696

That issue has the repro and the logs etc.. The issue is when you have an npm graph with a high number of duplicates (bunch of gulp tasks is the easy repro) then when it pulls the package, as npm is writing to the cache folder, something parallel/async is not synchronized and it tries to rename the same package.json in the cache folder and it fails with an EPERM. We also noticed this happens more frequently on slower I/O machines (VMs etc...). My fast local SSD box on Win10 never reproduces it.

Workarounds that avoid the issue on hosted:

  1. Pass --force to the npm install cmd. This force it to install from the service everytime and it avoids the cache (which is where the issue is).
  2. Invoke npm install task multiple times in your build definition. On all but the last check continue on error. It will get further each time and if not using --force, it will get farther with the aid of the cache.
  3. Deploy your agent on your own azure VM. Since the agent is stable (not blown away each build) the cache gets warm and mitigates the issue. This also has the benefit of speeding up the build avoiding the redownloading of packages on every build.

We will also be adding retries within the task next sprint.

@Bartmax
Bartmax commented Sep 21, 2015

Can you point me on directions for workaround #3? Thanks.

@Peter-Juhasz

@Bartmax I recommend a DS virtual machine with Premium Storage for better I/O. We have a very high success rate with it so far. Lots of or build have failed with a regular VM.

@Bartmax
Bartmax commented Sep 21, 2015

Thanks for avoiding me the huslle. Quite expensive for my needs. I think I'm moving away from tfs ci. Not worth it yet. I hope it will improve with time.

@Peter-Juhasz

@Bartmax It's not really a VSO issue. It is definitely an NPM issue. We have experimented with a Linux machine as well. It seems to be more stable, but as soon as you want to run multiple builds (NPM install) at the same time on the same machine the same locking problem appears (different file system locations, different sessions, etc...).
When we run the npm install command manually on our dev machines (Windows 8.1 + SSD), or in the cloud (Windows Server 2012 R2), we get some fails, but mostly it succeeds. Regular VMs have produced a lot of failed builds. Only the high speed VM worked for us. And on the other hand by upgrading our build machine, our builds are up to 60% faster. Which is cool, because we have ~50 CI builds.

@bryanmacfarlane
Member

@Bartmax - instructions for #3 are @ http://aka.ms/tfbuild
Also note that #1 is a simple workaround that didn't slow down the build (hosted build). Of course #3 with high performance I/O will get you faster builds (and incremental which is key here) but at a cost.

As @Peter-Juhasz noted, these are just workarounds for the core issue - an issue in npm writing to the cache folder when you have a complex dependency graph (typical with gulp). Slower I/O exposes the issue and it will happen with any build system. I reproduced it from the command line on a plain new VM per my write-up in the npm issue.

@bryanmacfarlane
Member

Also, we should note that next sprint we will put retries in the task so you don't have to do #1 - each invocation will get you farther as you are protected by the cache (even on hosted build where the machine is cleaned).

@pescado
pescado commented Nov 5, 2015

It seems this thread may be going cold as I haven't seen anything here or on the npm thread Bryan linked to since late September. The --force workaround mentioned above doesn't always work either. Below is a build log from yesterday with the failure in the npm install step with the --force arg. I'm using the hosted agent pool in VSO and the out-of-the-box npm install build step. You'll notice this isn't the EPERM error, rather the ECONNRESET error, but we get either one intermittently and I'm guessing it's the same issue. Has there been any progress on this that anyone knows of? Thanks!

2015-11-05T14:28:22.3720730Z Executing the powershell script: C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\default\tasks\NpmInstall\0.1.3\NpmInstall.ps1
2015-11-05T14:28:23.4800859Z C:\Program Files\nodejs\npm.cmd install --force
2015-11-05T14:28:35.8044835Z ##[warning]npm WARN using --force I sure hope you know what you are doing.
2015-11-05T14:29:41.3481288Z > phantomjs@1.9.18 install C:\a\1\s\EP.WebClient\node_modules\phantomjs
2015-11-05T14:29:41.3491294Z > node install.js
2015-11-05T14:29:42.1581291Z Downloading https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-windows.zip
2015-11-05T14:29:42.1591294Z Saving to C:\Users\BUILDG1\AppData\Local\Temp\phantomjs\phantomjs-1.9.8-windows.zip
2015-11-05T14:29:42.1601298Z Receiving...
2015-11-05T14:29:42.6556257Z Received 7292K total.
2015-11-05T14:29:42.6576249Z Extracting zip contents
2015-11-05T14:29:44.0906265Z Removing C:\a\1\s\EP.WebClient\node_modules\phantomjs\lib\phantom
2015-11-05T14:29:44.0916263Z Copying extracted folder C:\Users\BUILDG
1\AppData\Local\Temp\phantomjs\phantomjs-1.9.8-windows.zip-extract-1446733782656\phantomjs-1.9.8-windows -> C:\a\1\s\EP.WebClient\node_modules\phantomjs\lib\phantom
2015-11-05T14:29:46.3083449Z Writing location.js file
2015-11-05T14:29:46.3103450Z Done. Phantomjs binary available at C:\a\1\s\EP.WebClient\node_modules\phantomjs\lib\phantom\phantomjs.exe
2015-11-05T14:30:13.9478260Z > utf-8-validate@1.2.1 install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate
2015-11-05T14:30:13.9499145Z > node-gyp rebuild
2015-11-05T14:30:14.0018258Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:30:47.9372367Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:01.2509324Z validation.cc
2015-11-05T14:31:13.4701084Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\validation.vcxproj]
2015-11-05T14:31:20.5208976Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.exp
2015-11-05T14:31:20.5268996Z Generating code
2015-11-05T14:31:21.2963213Z Finished generating code
2015-11-05T14:31:23.0896089Z validation.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.node
2015-11-05T14:31:23.1806074Z ##[error]npm ERR! Windows_NT 6.3.9600
2015-11-05T14:31:23.1816091Z ##[error]npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "--force"
2015-11-05T14:31:23.1816091Z ##[error]npm ERR! node v0.12.7
2015-11-05T14:31:23.1826093Z ##[error]npm ERR! npm v2.11.3
2015-11-05T14:31:23.1836088Z ##[error]npm ERR! code ECONNRESET
2015-11-05T14:31:23.1836088Z ##[error]npm ERR! errno ECONNRESET
2015-11-05T14:31:23.1846092Z ##[error]npm ERR! syscall read
2015-11-05T14:31:23.1846092Z ##[error]npm ERR! network read ECONNRESET
2015-11-05T14:31:23.1856086Z ##[error]npm ERR! network This is most likely not a problem with npm itself
2015-11-05T14:31:23.1856086Z ##[error]npm ERR! network and is related to network connectivity.
2015-11-05T14:31:23.1866093Z ##[error]npm ERR! network In most cases you are behind a proxy or have bad network settings.
2015-11-05T14:31:23.1886089Z ##[error]npm ERR! network
2015-11-05T14:31:23.1896094Z ##[error]npm ERR! network If you are behind a proxy, please make sure that the
2015-11-05T14:31:23.1906093Z ##[error]npm ERR! network 'proxy' config is set properly. See: 'npm help config'
2015-11-05T14:31:23.1916096Z > bufferutil@1.2.1 install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil
2015-11-05T14:31:23.1916096Z > node-gyp rebuild
2015-11-05T14:31:23.1956094Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:31:24.9346206Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:25.3976236Z bufferutil.cc
2015-11-05T14:31:26.6336320Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\bufferutil.vcxproj]
2015-11-05T14:31:26.8426386Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.exp
2015-11-05T14:31:26.8966331Z Generating code
2015-11-05T14:31:27.2668555Z Finished generating code
2015-11-05T14:31:27.7978584Z bufferutil.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.node
2015-11-05T14:31:27.8618591Z > utf-8-validate@1.2.1 install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate
2015-11-05T14:31:27.8618591Z > node-gyp rebuild
2015-11-05T14:31:27.8718598Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:31:29.7993872Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:30.1703955Z validation.cc
2015-11-05T14:31:31.5963992Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\validation.vcxproj]
2015-11-05T14:31:31.9144016Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.exp
2015-11-05T14:31:31.9204017Z Generating code
2015-11-05T14:31:32.2444029Z Finished generating code
2015-11-05T14:31:32.6984068Z validation.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.node
2015-11-05T14:31:32.7294064Z > bufferutil@1.2.1 install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil
2015-11-05T14:31:32.7304055Z > node-gyp rebuild
2015-11-05T14:31:32.7374057Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:31:34.6165673Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:34.9955704Z bufferutil.cc
2015-11-05T14:31:36.2335780Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\bufferutil.vcxproj]
2015-11-05T14:31:36.5954953Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.exp
2015-11-05T14:31:36.6954960Z Generating code
2015-11-05T14:31:37.0984982Z Finished generating code
2015-11-05T14:31:37.6648305Z bufferutil.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.node
2015-11-05T14:31:37.6988342Z > fsevents@1.0.5 install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents
2015-11-05T14:31:37.6998357Z > node-pre-gyp install --fallback-to-build
2015-11-05T14:31:39.0979884Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" clean ) else (node clean )
2015-11-05T14:31:39.4999910Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64 ) else (node configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64 )
2015-11-05T14:31:40.8039993Z ##[error]Traceback (most recent call last):
2015-11-05T14:31:40.8049977Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 18, in
2015-11-05T14:31:40.8059976Z ##[error] sys.exit(gyp.script_main())
2015-11-05T14:31:40.8059976Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp__init__.py", line 533, in script_main
2015-11-05T14:31:40.8069975Z ##[error] return main(sys.argv[1:])
2015-11-05T14:31:40.8069975Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp__init__.py", line 526, in main
2015-11-05T14:31:40.8079972Z ##[error] return gyp_main(args)
2015-11-05T14:31:40.8089982Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp__init__.py", line 511, in gyp_main
2015-11-05T14:31:40.8089982Z ##[error] generator.GenerateOutput(flat_list, targets, data, params)
2015-11-05T14:31:40.8099970Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2000, in GenerateOutput
2015-11-05T14:31:40.8129982Z ##[error] sln_projects, project_objects, flat=msvs_version.FlatSolution())
2015-11-05T14:31:40.8139982Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1759, in _GatherSolutionFolders
2015-11-05T14:31:40.8169981Z ##[error] return _DictsToFolders('', root, flat)
2015-11-05T14:31:40.8179980Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1712, in _DictsToFolders
2015-11-05T14:31:40.8209969Z ##[error] for folder, contents in bucket.iteritems():
2015-11-05T14:31:40.8219982Z ##[error]AttributeError: 'MSVSProject' object has no attribute 'iteritems'
2015-11-05T14:31:40.8369994Z ##[error]gyp ERR! configure error
2015-11-05T14:31:40.8379993Z ##[error]gyp ERR! stack Error: gyp failed with exit code: 1
2015-11-05T14:31:40.8379993Z ##[error]gyp ERR! stack at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:355:16)
2015-11-05T14:31:40.8389997Z ##[error]gyp ERR! stack at ChildProcess.emit (events.js:110:17)
2015-11-05T14:31:40.8399989Z ##[error]gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
2015-11-05T14:31:40.8399989Z ##[error]gyp ERR! System Windows_NT 6.3.9600
2015-11-05T14:31:40.8409992Z ##[error]gyp ERR! command "node" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "configure" "--fallback-to-build" "--module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node" "--module_name=fse" "--module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64"
2015-11-05T14:31:40.8419989Z ##[error]gyp ERR! cwd C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents
2015-11-05T14:31:40.8419989Z ##[error]gyp ERR! node -v v0.12.7
2015-11-05T14:31:40.8429985Z ##[error]gyp ERR! node-gyp -v v2.0.1
2015-11-05T14:31:40.8429985Z ##[error]gyp ERR! not ok
2015-11-05T14:31:40.8519991Z ##[error]node-pre-gyp ERR! build error
2015-11-05T14:31:40.8529995Z ##[error]node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64' (1)
2015-11-05T14:31:40.8539998Z ##[error]node-pre-gyp ERR! stack at ChildProcess. (C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
2015-11-05T14:31:40.8549993Z ##[error]node-pre-gyp ERR! stack at ChildProcess.emit (events.js:110:17)
2015-11-05T14:31:40.8549993Z ##[error]node-pre-gyp ERR! stack at maybeClose (child_process.js:1015:16)
2015-11-05T14:31:40.8559994Z ##[error]node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1087:5)
2015-11-05T14:31:40.8559994Z ##[error]node-pre-gyp ERR! System Windows_NT 6.3.9600
2015-11-05T14:31:40.8569988Z ##[error]node-pre-gyp ERR! command "node" "C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\node_modules\node-pre-gyp\bin\node-pre-gyp" "install" "--fallback-to-build"
2015-11-05T14:31:40.8579988Z ##[error]node-pre-gyp ERR! cwd C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents
2015-11-05T14:31:40.8579988Z ##[error]node-pre-gyp ERR! node -v v0.12.7
2015-11-05T14:31:40.8589998Z ##[error]node-pre-gyp ERR! node-pre-gyp -v v0.6.15
2015-11-05T14:31:40.8589998Z ##[error]node-pre-gyp ERR! not ok
2015-11-05T14:31:40.8600086Z Failed to execute 'node-gyp.cmd configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64' (1)
2015-11-05T14:31:40.9899996Z ##[error]npm ERR! Windows_NT 6.3.9600
2015-11-05T14:31:40.9920075Z ##[error]npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "--force"
2015-11-05T14:31:40.9929999Z ##[error]npm ERR! node v0.12.7
2015-11-05T14:31:40.9940000Z ##[error]npm ERR! npm v2.11.3
2015-11-05T14:31:40.9950005Z ##[error]npm ERR! shasum check failed for C:\Users\BUILDG~1\AppData\Local\Temp\npm-2456-06618d10\registry.npmjs.org\babel-core-\babel-core-5.8.33.tgz
2015-11-05T14:31:40.9959996Z ##[error]npm ERR! Expected: 77e63eacff2072c695a827297a6d91b43546b2f4
2015-11-05T14:31:40.9959996Z ##[error]npm ERR! Actual: 37a677cdeca5a6b04f3567c66a75451075fcb6cc
2015-11-05T14:31:40.9969997Z ##[error]npm ERR! From: https://registry.npmjs.org/babel-core/-/babel-core-5.8.33.tgz
2015-11-05T14:31:40.9980004Z ##[error]npm ERR!
2015-11-05T14:31:40.9980004Z ##[error]npm ERR! If you need help, you may report this error at:
2015-11-05T14:31:40.9990000Z ##[error]npm ERR! https://github.com/npm/npm/issues
2015-11-05T14:31:40.9990000Z ##[error]npm ERR! Please include the following file with any support request:
2015-11-05T14:31:41.0000000Z ##[error]npm ERR! C:\a\1\s\EP.WebClient\npm-debug.log
2015-11-05T14:31:41.0759993Z ##[error]Unexpected exit code 1 returned from tool npm.cmd

@tempworks

Funny, I was just looking at this thread again this morning and dismayed to see no updates since September. We tried the --force parameter as well and it seemed to work for a bit but in the last 2-3 weeks the NPM failures have become a daily occurrence again, actually several times a day on our CI builds. Our build set up is very simple:

image

The builds usually fail with this report:

Summary | Build
6255 error(s), 4 warning(s)
Build - 6255 error(s), 4 warning(s)
(): npm ERR! Windows_NT 6.3.9600
(): npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "--force"
(): npm ERR! node v0.12.7
(): npm ERR! npm v2.11.3
(): npm ERR! code ECONNRESET
(): npm ERR! errno ECONNRESET
(): npm ERR! syscall read
(): npm ERR! network read ECONNRESET
(): npm ERR! network This is most likely not a problem with npm itself
(): npm ERR! network and is related to network connectivity.

It know it was a suggestion early one to create your build server with high I/O throughput but I'd like to avoid another server to maintain for our simplistic build needs. The new vNext build system is really nice but these errors are starting to interrupt our production workflow. May I suggest that MS investigate creating a second tier of higher performing build servers for those who need it.

-Paul

@IDontEatSoap

Same problem here:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! syscall read
npm ERR! network read ECONNRESET
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly. See: 'npm help config'

Absolutely defeats using the unreliable hosted build environment...

@barretts

Though I am not using vso-agent-tasks this was the only place I've found a solid connection to the errors we've been running into doing npm install on AWS EC2 instances. EPERM, rename errors would come up more often than not but we could never reproduce it locally. When we logged into the boxes we couldn't reproduce it. Following @bryanmacfarlane's idea I bumped up the EC2 machines to be i2.xlarge which have better disk I/O performance and I haven't had anymore problems. I had tried M3 large, M4 large, C4, C3 before this. We are running Windows Server 2012 and 2008. I'll report back if this doesn't continue working. Thanks!

@bryanmacfarlane
Member

@barretts - note that you don't necessarily have to jump to the expensive xlarge etc... VMs. There's an option in azure called Premium Storage that you opt for on disks. We use for D2 skus.

@tempworks - we're investigating just giving Premium I/O to the hosted pool. But we're running tests now. Hopefully you can just get it but we'll see where that lands. We're doing it this sprint.

@barretts

@bryanmacfarlane thanks for the Azure tip! we've been evaluating them as well for the larger array of Windows versions and Edge browser testing so that will come in handy

I see some of you having ECONNRESET issues, we were able to resolve that by pointing NPM to the http registry after reading http://stackoverflow.com/a/18428563/604861

@bryanmacfarlane
Member

@barretts - thanks we'll investigate the ECONRESET (diff issue than this thread).

@tempworks

@bryanmacfarlane So to recap the recent discussions today.

Hosting your own vso-agents with high performing I/O will not experience this problem.

There is no promise of a delivery or timeline of high performing vso-agent tiers.

So for those experiencing this problem, it's Microsoft's opinion is to host your own vso-agents.

@sethreidnz

Hi I'm having the same problem:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! shasum check failed for C:\Users\BUILDG~1\AppData\Local\Temp\npm-2792-3de7b145\registry.npmjs.org\less\-\less-1.7.5.tgz
npm ERR! Expected: 4f220cf7288a27eaca739df6e4808a2d4c0d5756
npm ERR! Actual: 3acfbcebee77bd1ef7a044c90a48b97316af4b6c
npm ERR! From: https://registry.npmjs.org/less/-/less-1.7.5.tgz
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! <https://github.com/npm/npm/issues>
npm ERR! Please include the following file with any support request:
npm ERR! C:\a\1\s\dist\npm-debug.log
Unexpected exit code 1 returned from tool npm.cmd
******************************************************************************
Finishing task: NpmInstall
******************************************************************************
Task NpmInstall failed. This caused the job to fail. Look at the logs for the task for more details.
******************************************************************************
@bryanmacfarlane
Member

@justsayno That doesn't look like the same problem. This thread is about an EPERM race condition in npm. It's not a catch all for all errors that could come out of npm

@sethreidnz

Sorry about that I was on a call to a Microsoft person in New Zealand and we both mistakenly got the impression this issue was a similar thing because of the above comment by @brutaldev and @pescado that does include an error about :

 shasum check failed for .....

I'll look through issues and if I cannot find one for this issue then I will open one.

@bryanmacfarlane
Member

@justsayno - thanks. Very soon, I'm going to summarize this thread/issue into an npm doc/faq.

@barretts

@bryanmacfarlane I wish I had better news but even with those over powered machines I still got hit by the error. I've thrown up my hands and implemented an npm install retry batch file for my situation. I haven't seen it error twice in a row yet but I've got it setup to try 3 times before failing. Hope y'all can figure out what the actual issue is. Good luck!

Edit: overpowered being the i2.xlarge which was said might not be needed.

@bryanmacfarlane
Member

@barretts - Thanks. I don't know what over powered machines means.

The issue is a deep npm dependency graph with a high number of duplicates in that graph with race conditions over locking and updating files.

Here's the root issue: npm/npm#9696

@bryanmacfarlane
Member

@barretts
Here's the specific option for disks in azure to minimize: https://azure.microsoft.com/en-us/documentation/articles/storage-premium-storage-preview-portal

As I stated earlier, we are investigating enabling on hosted build machines this sprint.

@Bartmax
Bartmax commented Nov 18, 2015

Can't this be solved it instead trying to lower the ocurrence with SSD disks (expensive solution) ?

TravisCI works with no issue on the free tier without SSD.

@bryanmacfarlane
Member

@Bartmax - the npm bug is a race condition that repros on windows with low I/O. This isn't a CI issue - I sent npm a repro (way up in the thread) with a package.json and npm cmd line (no CI involved). We just run cmd line just like Travis. Travis also doesn't build on windows.

And yes, as I pointed out in the thread way above, there's an inexpensive workaround which is to pass --force as additional args to npm. That causes it to bypass the cache which avoids the race and deep graphs with multiple writes without locking to the cache dir ( npm/npm#9696 ). We are considering just doing that on hosted (with retries like other CIs do) since it's always a fresh machine and the cache doesn't help much.

@tghamm
tghamm commented Nov 19, 2015

We're getting this error using the NPM task (with --force) on our self-hosted instance consistently (we just set it up in the last couple of days):

npm ERR! syscall rename
npm ERR! Error: EPERM: operation not permitted, rename     'C:\Users\tfsbuild\AppData\Roaming\npm-    cache\minimist\1.2.0\package\package.json.414d5643225a5c0e9714b0cdc0eaf8b4' ->     'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM: operation not permitted, rename 'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json.414d5643225a5c0e9714b0cdc0eaf8b4' -> 'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! syscall: 'rename',
npm ERR! path: 'C:\\Users\\tfsbuild\\AppData\\Roaming\\npm-cache\\minimist\\1.2.0\\package\\package.json.414d5643225a5c0e9714b0cdc0eaf8b4',
npm ERR! dest: 'C:\\Users\\tfsbuild\\AppData\\Roaming\\npm-cache\\minimist\\1.2.0\\package\\package.json',
npm ERR! parent: 'gulp-util' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

Is there any other way to get this working?

@bryanmacfarlane
Member

@tghamm - npm is writing to the cache folder:

EPERM: operation not permitted, rename 'C:\Users\tfsbuild\AppData\Roaming\npm-cache ...

Which implies --force is not being passed (force fetches and bypassed the cache)

Can you:

  • gather your CI steps and inputs
  • set system.debug to true on the variables tab of the definition
  • gather task output
  • since you have your own vm, gather the npm log the output refers to
  • one more - your package.json (or at a minimum the dependencies section)
    Zip that up and send it bigbldt at microsoft dot com
@tghamm
tghamm commented Nov 20, 2015

@bryanmacfarlane will do. It's actually working intermittently now so I'm waiting for it to fail again.

@konrad-ha

Using the "--force" works for now but is far from an optimal solution. We're trying to roll out a rather large pipeline using npm right now and I'm afraid this might get a serious issue.

@bryanmacfarlane
Member

@konrad-ha - agreed. The only right fix is npm fixing the race condition. --force, retries, and faster I/O are all workarounds/mitigations.

@joaomoreno
Member

How about a focused hackathon by Microsoft dedicated to fix this for npm?

@ivanz
ivanz commented Nov 25, 2015

I have been running my own VSO agent on a VM with premium storage data disk for 10 days and even though it dramatically improves the situation (and build times) - I still see the occasional EPERM error (very rare though so far)

@Bartmax
Bartmax commented Nov 25, 2015

what about a central cache/disk for npm in the microsoft infrastructure. Instead of downloading everything, the vm can attach the cache drive and download there or use it as cache improving speed by a lot and prevent this kind of errors. (same can be done for nuget).

if the vm is run without cache (some option?), no disk will be attached and you can opt in for the current behavior (for those needing that stuff actually gets downloaded)

Personally, It will make me very happy to use hosted agent and don't waste download speed / bandwidth.

@AMoghrabi

Our team city agents also run into this issue. Using the --force argument does not work unfortunately. Here is a sample build log:

    [11:25:59][Step 1/1] Execute npm_install (13m:23s)
    [11:25:59][Execute npm_install] Running npm with parameters: ["install", "--force"]
    [11:26:03][Execute npm_install] npm WARN using --force I sure hope you know what you are doing.
    ...
    ...
    11:38:47][Execute npm_install] npm ERR! Windows_NT 6.1.7601
    [11:38:48][Execute npm_install] npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--force"
    [11:38:48][Execute npm_install] npm ERR! node v0.12.7
    [11:38:48][Execute npm_install] npm ERR! npm  v2.11.3
    [11:38:48][Execute npm_install] npm ERR! path C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json
    [11:38:48][Execute npm_install] npm ERR! code EPERM
    [11:38:48][Execute npm_install] npm ERR! errno -4048
    [11:38:48][Execute npm_install] 
    [11:38:48][Execute npm_install] npm ERR! Error: EPERM, rename 'C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json'
    [11:38:48][Execute npm_install] npm ERR!     at Error (native)
    [11:38:48][Execute npm_install] npm ERR!  { [Error: EPERM, rename 'C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json']
    [11:38:48][Execute npm_install] npm ERR!   errno: -4048,
    [11:38:48][Execute npm_install] npm ERR!   code: 'EPERM',
    [11:38:48][Execute npm_install] npm ERR!   path: 'C:\\Users\\teamcityagent\\AppData\\Roaming\\npm-cache\\inherits\\2.0.1\\package\\package.json',
    [11:38:48][Execute npm_install] npm ERR!   parent: 'npmconf' }
    [11:38:48][Execute npm_install] npm ERR! 
    [11:38:48][Execute npm_install] npm ERR! Please try running this command again as root/Administrator.
    [11:39:22][Execute npm_install] 

Our agents are running on Windows 7 and currently it is not feasible to upgrade the agents to use an SSD...

@bryanmacfarlane
Member

Lots of progress in finding the root issue from Microsoft open tech. The issue is in npm2. More here: npm/npm#9696

The good news is it's fixed in npm3 ([sudo] npm install npm@3). We're progressing to see if a fix can be pushed to npm2.

We are looking to move hosted build pool to npm3 and we're almost done with premium storage in the hosted pool (even if npm3 fixes, still much faster I/O).

So to sum up the workarounds:

  • own VM with premium storage, and/or
  • own VM with npm3
  • on hosted pool use --force
@joaomoreno
Member

Very cool!

@Maarten88

Based on this thread, I may have found a workaround: first install npm@3, then run that using a Command Line task to do the actual npm install:

(npm task) -> npm install npm@3
(Command Line task) -> $(Build.SourcesDirectory)\[your project]\node_modules\.bin\npm.cmd

My npm install tasks finished successfully without having to restart it multiple times for the first time in months!

@bryanmacfarlane
Member

Hosted build image is getting npm3 this sprint.

Note that if you have your own agent and you install npm3 (or node5.x) and it resolves in your path (confirm with npm -v), then you can use the npm task (no need use command line task but that's fine as well).

@chrisrpatterson
Member

We had to roll back npm3 because it broke Cordova so it will be a bit longer before we can move to it.

From: Bryan MacFarlane [mailto:notifications@github.com]
Sent: Friday, January 15, 2016 8:09 AM
To: Microsoft/vso-agent-tasks vso-agent-tasks@noreply.github.com
Cc: Chris Patterson Chris.Patterson@microsoft.com
Subject: Re: [vso-agent-tasks] npm install task (not msbuild) fails with EPERM on hosted build (#323)

Hosted build image is getting npm3 this sprint.

Note that if you have your own agent and you install npm3 (or node5.x) and it resolves in your path (confirm with npm -v), then you can use the npm task (no need use command line task but that's fine as well).


Reply to this email directly or view it on GitHubhttps://github.com/Microsoft/vso-agent-tasks/issues/323#issuecomment-171955379.

@San7hos
San7hos commented Jan 24, 2016

What is the progress on this?

Would it help to simply include node_modules into the repository and basically skip the npm install? Is it a good idea?

@ChBrain
ChBrain commented Jan 24, 2016

I would not do that. You kind of break the concept behind npm install. npm install npm@3 takes usually less than 2 minutes for us. npm install of course has to download quite some stuff (depending on project), but thats the point, to keep the dependencies up to date and to latest release.

@bryanmacfarlane
Member

Progress:

Microsoft open tech fixed the root problem in npm2.
npm/npm#9696
npm/npm@e982858

When that is released, we will update our hosted image to it. On your machines you can do the same or install npm@3

We are also in the process of rolling out premium storage disks to our hosted pool scale units at no cost to our customers. Early numbers show significant I/O gains and have cut the time in half for some of our sample builds. This will minimize the likelyhood of hitting the bug.

This upcoming sprint, we will offer a drop down in the npm task to specify which npm version. It will be leveraging a tools concept we will be adding: https://github.com/Microsoft/vso-agent-tasks/blob/master/docs/tools.md

@tempworks

Good news Bryan and thanks to MS open tech. Will there be an announcement on a blog somewhere when the upcoming sprint is deployed? Better yet would be a note here, it would be fantastic for those subscribed to this thread. -Paul

@San7hos
San7hos commented Jan 24, 2016

Thanks for the information. If I understand you correctly: when using the Hosted Agent, we need to wait until the fix is released. However, the upcoming sprint will bring some improvements sooner and the fix will be released a bit later.

@KaiTwiggy

There has been some time from the last comment so i was wondering has there been any progress on this issue ? The workaround that @Maarten88 posted works fine, but it would be much cleaner if VSO would provide different versions of npm out of the box.

I was wondering if the dropdown version selection will as well apply to node and other dependencies or is it only npm for now ?

@bryanmacfarlane
Member

Closing.

It was confirmed as an npm2 bug and the fix was made to npm2
It's not a bug in npm3.
We finished moving most pools to premium storage in the hosted pool (at our cost) which minimizes the chance if you hit it.

@medfreeman medfreeman added a commit to medfreeman/markdown-it-toc-and-anchor that referenced this issue May 5, 2016
@medfreeman medfreeman added force flag to appveyor npm install command, fixes npm/npm#9696 M… 9add02c
@medfreeman medfreeman added a commit to medfreeman/markdown-it-toc-and-anchor that referenced this issue May 5, 2016
@medfreeman medfreeman added npm3 installation to appveyor config, dfd5ec1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment