Problem with npm dependency resolution

[manuelbernhardt]

There seems to be an issue when attempting to resolve npm modules declared in package.json when they depend upon the node-gyp package. Steps to reproduce:

• add JSDOM as a dependency in the sbt-js-engine-tester project:

{
    "name": "somename",
    "version": "1.0.0",
    "dependencies": {
        "console-browserify": "0.1.6",
        "jsdom": "1.0.3"
    }
}

• attempt to run npm module resolution:

sbt web-assets:jseNpmNodeModules

when that one package gets installed by hand previously (npm install node-gyp) the command runs without a hitch.

[benmccann]

+1 it would be cool to figure out what's causing this

[aparkinson]

@manuelbernhardt Are you seeing this problem when running on Ubuntu?

I get this error on 4 different Ubuntu machines but it works fine on a Mac

[huntc]

I'm unsure whether I've previously asked this question, so apologies if so, but what engine is being used i.e. have we tried this using Node as the engine?

[AlexanderDaniel]

I reported the issue together with Manuel.

@aparkinson I am seeing this issue on the Mac. Essential for reproducing the issue to remove the node_modules folder before running sbt, i.e. rm -rf node_modules/ && sbt web-assets:jseNpmNodeModules

@huntc Node is being used as engine. It is not set explicitly in sbt-js-engine-tester but I have -Dsbt.jse.engineType=Node in my SBT_OPTS.

I can also reproduce the issue when using Trireme.

[huntc]

I will take a look soon. Thanks for the detailed explanations everyone.

[AlexanderDaniel]

I successfully ran the workaround

rm -rf node_modules/ && npm install node-gyp && sbt web-assets:jseNpmNodeModules

both with Linux (Ubuntu) and with Mac OS X.

Note: On Ubuntu Linux node-gyp requires the build-essential package and on Mac OS X the XCode Command Line Tools.

[benmccann]

I tried to figure out what was going on here today and didn't find anything useful, but I don't know this code well. I'm not sure whether it's an sbt-js-engine bug, but would be interested in what's going. The other issue you filed (playframework/playframework#3565) pretty clearly does look like a bug in sbt-js-engine or play though and I think is more frustrating too

[raphaelpanta]

I have a Play 2.3.8 Java project, which I generate the view Asssets with Yeoman (Angularjs seed) and I put grunt, bower, npm, package.json in same project. When I run ./activator test there're a message related to web-assets:jseNpmNodeModules task. I can ran tests in Eclipse.

I can run the app normally when I developing. I executed npm install node-gyp and rerun the test and it did not resolve my issue.

The error log is:

akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://sbt-web/user/$a#-1891362858]] after [120000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)
error akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://sbt-web/user/$a#-1891362858]] after [120000 ms]
[error] Total time: 120 s, completed Feb 23, 2015 8:07:42 PM

[naderghanbari]

For me the workaround works fine in Mac OS but in ubuntu I get the following error:

java.lang.RuntimeException: Problems with NPM resolution. Aborting build.
at scala.sys.package$.error(package.scala:27)
at com.typesafe.sbt.jse.SbtJsEngine$$anonfun$1$$anonfun$2.apply(SbtJsEngine.scala:94)
at com.typesafe.sbt.jse.SbtJsEngine$$anonfun$1$$anonfun$2.apply(SbtJsEngine.scala:71)
at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:235)
at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:235)
at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala
at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala
at sbt.Difference.apply(Tracked.scala:224)
at sbt.Difference.apply(Tracked.scala:206)
at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:245)
at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:244)
at sbt.Difference.apply(Tracked.scala:224)
at sbt.Difference.apply(Tracked.scala:200)
at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:244)
at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:242)
at com.typesafe.sbt.jse.SbtJsEngine$$anonfun$1.apply(SbtJsEngine.scala:102)
at com.typesafe.sbt.jse.SbtJsEngine$$anonfun$1.apply(SbtJsEngine.scala:66)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

[jroper]

It looks like we need to upgrade npm, this issue is unrelated to js-engine itself, since it can be reproduced with just node:

$ node project/target/node-modules/webjars/npm/lib/npm.js update
npm http GET https://registry.npmjs.org/jsdom/latest
[snip]
npm http GET https://registry.npmjs.org/entities

> contextify@0.1.14 install /Users/jroper/tmp/play-react-mocha/node_modules/jsdom/node_modules/contextify
> node-gyp rebuild

sh: /Users/jroper/tmp/play-react-mocha/project/target/node-modules/webjars/npm/bin/node-gyp-bin/node-gyp: Permission denied
npm ERR! EEXIST, mkdir '/Users/jroper/tmp/play-react-mocha/node_modules/jsdom/node_modules/cssstyle/lib/properties'
File exists: /Users/jroper/tmp/play-react-mocha/node_modules/jsdom/node_modules/cssstyle/lib/properties
Move it away, and try again. 

npm ERR! System Darwin 14.3.0
npm ERR! command "node" "/Users/jroper/tmp/play-react-mocha/project/target/node-modules/webjars/npm/lib/npm.js" "update"
npm ERR! cwd /Users/jroper/tmp/play-react-mocha
npm ERR! node -v v0.12.2
npm ERR! npm -v 1.3.26
npm ERR! path /Users/jroper/tmp/play-react-mocha/node_modules/jsdom/node_modules/cssstyle/lib/properties
npm ERR! fstream_path /Users/jroper/tmp/play-react-mocha/node_modules/jsdom/node_modules/cssstyle/lib/properties/textLineThrough.js
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code EEXIST
npm ERR! errno -17
npm ERR! fstream_stack /Users/jroper/tmp/play-react-mocha/project/target/node-modules/webjars/npm/node_modules/fstream/lib/writer.js:171:23
npm ERR! fstream_stack /Users/jroper/tmp/play-react-mocha/project/target/node-modules/webjars/npm/node_modules/mkdirp/index.js:37:53
npm ERR! fstream_stack FSReqWrap.oncomplete (fs.js:95:15)
npm ERR! contextify@0.1.14 install: `node-gyp rebuild`
npm ERR! Exit status 126
npm ERR! 
npm ERR! Failed at the contextify@0.1.14 install script.
npm ERR! This is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls contextify
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 14.3.0
npm ERR! command "node" "/Users/jroper/tmp/play-react-mocha/project/target/node-modules/webjars/npm/lib/npm.js" "update"
npm ERR! cwd /Users/jroper/tmp/play-react-mocha
npm ERR! node -v v0.12.2
npm ERR! npm -v 1.3.26
npm ERR! code ELIFECYCLE
npm http 304 https://registry.npmjs.org/caseless
[snip]
npm http 304 https://registry.npmjs.org/get-stdin
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/jroper/tmp/play-react-mocha/npm-debug.log
npm ERR! not ok code 0

[jroper]

Ok, I should have looked more carefully, the error is obvious, the npm webjar is not setting the executable bit on node-gyp. After a failed build, run:

chmod +x project/target/npm/bin/node-gyp-bin/node-gyp

Then run it again, and it works.

[jroper]

Solving this generally is going to be really hard. The npm webjar can be fixed to contain the right permissions, but the big problem is that the JDK zip support does not support file permissions, and so when sbt unzips it, which uses the JDK zip support, those permissions are not going to be there.

I think for now, the best thing to do is to hack it - specifically make that particular file executable.

[jroper]

Fixed in #35.

[benmccann]

Oh, I see. I thought it was only unzipping that's a problem, but it does appear that zipping it is done incorrectly as well. This can be verified by grabbing the jar.

I wonder if this is really the only module we'll have this problem with. It seems rather silly to take things that are in npm and repackage them into jars such that we lose all the file permissions in the process. Can't we just grab directly from npm instead?

This would work much better:

wget http://registry.npmjs.org/npm/-/npm-2.11.1.tgz
tar -zxvf npm-2.11.1.tgz

[benmccann]

This appears to be fixed now. Woohoo! Huge thanks to @jroper for this