Rename from tmp does not occur (Mac) #15

Closed
Krinkle opened this Issue Nov 23, 2012 · 32 comments

Projects

None yet

7 participants

@Krinkle

I just installed grunt-contrib-qunit, which depends on grunt-lib-phantomjs, which depens on node-phantomjs (this library).

Output:

$ npm install
...
npm http GET https://registry.npmjs.org/phantomjs
...
npm http 200 https://registry.npmjs.org/phantomjs
npm http GET https://registry.npmjs.org/phantomjs/-/phantomjs-0.2.2.tgz
npm http 200 https://registry.npmjs.org/phantomjs/-/phantomjs-0.2.2.tgz
...

> phantomjs@0.2.2 install .../node_modules/phantomjs
> node install.js

Requesting .../node_modules/phantomjs/tmp/phantomjs-1.7.0-macosx.zip
Receiving...
Recieved 783K...
Recieved 1565K...
Recieved 2361K...
Recieved 3145K...
Recieved 3927K...
Recieved 4709K...
Recieved 5491K...
Recieved 6274K...
Recieved 7055K...
Recieved 7837K...
Recieved 8619K...
Recieved 9401K...
Recieved 10182K...
Recieved 10182K total.
Extracting zip contents
grunt-contrib-qunit@0.1.0 node_modules/grunt-contrib-qunit
└── grunt-lib-phantomjs@0.1.0 (semver@1.0.14, eventemitter2@0.4.11, temporary@0.0.5, phantomjs@0.2.2)

$

It is supposed to move the files to the lib directory, right?

Screenshot of resulting tree

@Krinkle

I could reproduce this several times, but it looks like it is working now:

> phantomjs@0.2.2 install .../node_modules/phantomjs
> node install.js

Requesting .../node_modules/phantomjs/tmp/phantomjs-1.7.0-macosx.zip
Receiving...
Recieved 781K...
Recieved 1564K...
Recieved 2405K...
Recieved 3187K...
Recieved 3969K...
Recieved 4753K...
Recieved 5577K...
Recieved 6372K...
Recieved 7204K...
Recieved 8027K...
Recieved 8809K...
Recieved 9593K...
Recieved 10182K total.
Extracting zip contents
Renaming extracted folder phantomjs-1.7.0-macosx  -> phantom
Fixing file permissions
Done. Phantomjs binary available at .../phantomjs/lib/phantom/bin/phantomjs
jshint@0.9.1 node_modules/jshint
├── minimatch@0.0.5 (lru-cache@1.0.6)
└── cli@0.4.3 (glob@3.1.14)

grunt-contrib-qunit@0.1.0 node_modules/grunt-contrib-qunit
└── grunt-lib-phantomjs@0.1.0 (semver@1.0.14, eventemitter2@0.4.11, temporary@0.0.5, phantomjs@0.2.2)

No idea why it didn't work before.

@Krinkle Krinkle closed this Nov 23, 2012
@mattd

I can still recreate this sometimes. To make issues worse, I can't find a single, reliable set of conditions that determine whether the move happens or not. For me, it works 90% of the time. For a colleague of mine, the ratio is pretty much reversed - it works 10% of the time.

This smells like some kind of race condition, but that's backed up by 0 actual evidence.

@dpup
Medium member

O_o

Sorry you're having problems. We have had no issues, we're using it on OX 10.8.2 for development and on EC2 instances for automated testing.

Are you on latest node/npm?

@mattd

Yeah, 0.8.16, OSX 10.8.2. It's really baffling. I'm using this in conjunction with grunt-contrib-jasmine, not qunit, but the symptom is ultimately the same - the copy and permission fixes isn't happening.

@mattd mattd referenced this issue in gruntjs/grunt-lib-phantomjs Dec 18, 2012
Closed

grunt qunit failes due to PhantomJS Error #5

@dpup
Medium member
@mattd
@mattd

I should cc @cowboy on this, as he might have some insight into possible subprocess issues.

@dpup
Medium member

I'm not super familiar with the details -- will try to find out -- but we were having unrelated problems with our jenkins build due to an issue where the environment wasn't correctly passed onto the child process.

I was wondering if perhaps the child process doesn't have permission to do the copy and chmod the extracted files.

@dpup
Medium member

(but that wouldn't explain the intermittent failures :-/)

@GiantEnemyCrab

My output is like this, that phantomjs download is not happening like output from Krinkle's comment above. (I am 10.6):

Here is the output I am getting to support the theory of race condition, I will paste only the last half of it (because my full directory names shall not be seen)

Recieved 1564K...
npm http 200 https://registry.npmjs.org/abbrev/-/abbrev-1.0.3.tgz
Recieved 2345K...
npm http GET https://registry.npmjs.org/wordwrap
npm http 304 https://registry.npmjs.org/wordwrap
Recieved 3127K...

lodash@0.9.2 install /opx/rsm/client/node_modules/grunt/node_modules/lodash
node build/post-install

Recieved 3917K...
npm http GET https://registry.npmjs.org/argparse
Recieved 4698K...
npm http 200 https://registry.npmjs.org/argparse
npm http GET https://registry.npmjs.org/argparse/-/argparse-0.1.8.tgz
npm http 200 https://registry.npmjs.org/argparse/-/argparse-0.1.8.tgz
Recieved 5480K...
npm http GET https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/underscore.string
Recieved 6261K...
npm http 304 https://registry.npmjs.org/underscore.string
npm http GET https://registry.npmjs.org/underscore.string/-/underscore.string-2.1.1.tgz
npm http 200 https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/underscore/-/underscore-1.3.3.tgz
npm http 200 https://registry.npmjs.org/underscore.string/-/underscore.string-2.1.1.tgz
Recieved 7043K...
npm http 200 https://registry.npmjs.org/underscore/-/underscore-1.3.3.tgz
Recieved 7824K...
Recieved 8606K...
Recieved 9387K...
Recieved 10169K...
Recieved 10182K total.
Extracting zip contents

@dpup
Medium member

@A415Hz Are you also using grunt?

(for the record we're just using ye olde make)

@dpup
Medium member

So its sounding more and more likely to be a grunt issue, no?

@mattd

@dpup Yes, it is. :) Thanks for your help.

@cowboy

This issue that people are reporting occurs during npm install. It has nothing to do with grunt.

All grunt-lib-phantomjs does is:

If a problem exists, it's between this lib and npm. Will you consider re-opening this issue here?

@dpup
Medium member
@cowboy

@dpup grunt shouldn't be running npm, ever! Do you have some specific code you're talking about?

@cowboy

I'm pretty sure all the examples in this issue are people just running npm install on the command line.

@dpup
Medium member

Sorry, I don't know anything about grunt, was reading between the lines on this thread. The only people reporting problems seem to be using grunt and I have seen some issues (unrelated to phantom) with NPM being invoked as a child-process, so further up the thread I queried whether that might be an issue and it sounded like it could.

I can't reproduce any flakiness and we have half-a-dozen developers using the lib daily with no issues, on both dev macs and EC2 instances.

Are the exact repro steps simply:
npm install phantomjs
?

@dpup dpup reopened this Dec 19, 2012
@dpup
Medium member

Reopening so it's easier to track. Would be really nice to narrow down the exact cases where this is breaking.

dan@Cayos: /tmp $ npm install grunt-contrib-qunit
npm http GET https://registry.npmjs.org/grunt-contrib-qunit
npm http 304 https://registry.npmjs.org/grunt-contrib-qunit
npm http GET https://registry.npmjs.org/grunt-lib-phantomjs
npm http 304 https://registry.npmjs.org/grunt-lib-phantomjs
npm http GET https://registry.npmjs.org/eventemitter2
npm http GET https://registry.npmjs.org/semver
npm http GET https://registry.npmjs.org/temporary
npm http GET https://registry.npmjs.org/phantomjs
npm http 304 https://registry.npmjs.org/phantomjs
npm http 304 https://registry.npmjs.org/semver
npm http 304 https://registry.npmjs.org/temporary
npm http 304 https://registry.npmjs.org/eventemitter2
npm http GET https://registry.npmjs.org/rimraf
npm http GET https://registry.npmjs.org/unzip
npm http GET https://registry.npmjs.org/package
npm http 304 https://registry.npmjs.org/unzip
npm http 304 https://registry.npmjs.org/package
npm http 304 https://registry.npmjs.org/rimraf
npm http GET https://registry.npmjs.org/graceful-fs
npm http GET https://registry.npmjs.org/fstream
npm http GET https://registry.npmjs.org/pullstream/0.0.4
npm http GET https://registry.npmjs.org/binary
npm http 304 https://registry.npmjs.org/graceful-fs
npm http 304 https://registry.npmjs.org/fstream
npm http 304 https://registry.npmjs.org/pullstream/0.0.4
npm http 304 https://registry.npmjs.org/binary
npm http GET https://registry.npmjs.org/over
npm http GET https://registry.npmjs.org/stream-buffers
npm http GET https://registry.npmjs.org/graceful-fs
npm http GET https://registry.npmjs.org/mkdirp
npm http GET https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/buffers
npm http GET https://registry.npmjs.org/chainsaw
npm http 304 https://registry.npmjs.org/over
npm http 304 https://registry.npmjs.org/stream-buffers
npm http 304 https://registry.npmjs.org/buffers
npm http 304 https://registry.npmjs.org/chainsaw
npm http 304 https://registry.npmjs.org/graceful-fs
npm http 304 https://registry.npmjs.org/mkdirp
npm http 304 https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse

> phantomjs@0.2.3 install /private/tmp/node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs
> node install.js

Requesting /private/tmp/node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/tmp/phantomjs-1.7.0-macosx.zip
Receiving...
Recieved 781K...
Recieved 1573K...
Recieved 2356K...
Recieved 3138K...
Recieved 3953K...
Recieved 4735K...
Recieved 5517K...
Recieved 6298K...
Recieved 7080K...
Recieved 7862K...
Recieved 8643K...
Recieved 9425K...
Recieved 10182K total.
Extracting zip contents
Renaming extracted folder phantomjs-1.7.0-macosx  -> phantom
Fixing file permissions
Done. Phantomjs binary available at /private/tmp/node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/lib/phantom/bin/phantomjs
grunt-contrib-qunit@0.1.0 node_modules/grunt-contrib-qunit
└── grunt-lib-phantomjs@0.1.0 (semver@1.0.14, eventemitter2@0.4.11, temporary@0.0.5, phantomjs@0.2.3)
dan@Cayos: /tmp $ l node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/lib/phantom/bin/phantomjs 
-rwxr-xr-x  1 dan  staff    10M Dec 19 10:13 node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/lib/phantom/bin/phantomjs
dan@Cayos: /tmp $ node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/bin/phantomjs 
phantomjs>

@cowboy

Doesn't seem to be working here.

$ npm install phantomjs; node -pe 'var p = require("phantomjs").path; "\n" + p + "\n" + require("fs").existsSync(p)'
npm http GET https://registry.npmjs.org/phantomjs
npm http 304 https://registry.npmjs.org/phantomjs
npm http GET https://registry.npmjs.org/rimraf
npm http GET https://registry.npmjs.org/unzip
npm http 304 https://registry.npmjs.org/unzip
npm http 304 https://registry.npmjs.org/rimraf
npm http GET https://registry.npmjs.org/graceful-fs
npm http GET https://registry.npmjs.org/fstream
npm http GET https://registry.npmjs.org/pullstream/0.0.4
npm http GET https://registry.npmjs.org/binary
npm http 304 https://registry.npmjs.org/graceful-fs
npm http 304 https://registry.npmjs.org/pullstream/0.0.4
npm http 304 https://registry.npmjs.org/binary
npm http 304 https://registry.npmjs.org/fstream
npm http GET https://registry.npmjs.org/over
npm http GET https://registry.npmjs.org/stream-buffers
npm http GET https://registry.npmjs.org/mkdirp
npm http GET https://registry.npmjs.org/graceful-fs
npm http GET https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/chainsaw
npm http GET https://registry.npmjs.org/buffers
npm http 304 https://registry.npmjs.org/stream-buffers
npm http 304 https://registry.npmjs.org/inherits
npm http 304 https://registry.npmjs.org/over
npm http 304 https://registry.npmjs.org/mkdirp
npm http 304 https://registry.npmjs.org/graceful-fs
npm http 304 https://registry.npmjs.org/chainsaw
npm http 304 https://registry.npmjs.org/buffers
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse

> phantomjs@0.2.3 install /Users/cowboy/tmp2/node_modules/phantomjs
> node install.js

Requesting /Users/cowboy/tmp2/node_modules/phantomjs/tmp/phantomjs-1.7.0-macosx.zip
Receiving...
Recieved 782K...
Recieved 1564K...
Recieved 2346K...
Recieved 3129K...
Recieved 3911K...
Recieved 4694K...
Recieved 5476K...
Recieved 6258K...
Recieved 7041K...
Recieved 7823K...
Recieved 8606K...
Recieved 9388K...
Recieved 10170K...
Recieved 10182K total.
Extracting zip contents
phantomjs@0.2.3 node_modules/phantomjs
├── rimraf@2.0.3 (graceful-fs@1.1.14)
└── unzip@0.0.4 (pullstream@0.0.4, fstream@0.1.19, binary@0.3.0)

/Users/cowboy/tmp2/node_modules/phantomjs/lib/phantom/bin/phantomjs
false

Then:

$ tree -I node_modules /Users/cowboy/tmp2/node_modules/phantomjs/Users/cowboy/tmp2/node_modules/phantomjs
├── LICENSE.txt
├── README.md
├── bin
│   └── phantomjs
├── install.js
├── lib
│   └── phantomjs.js
├── package.json
├── test
│   ├── loadspeed.js
│   └── tests.js
└── tmp
    ├── phantomjs-1.7.0-macosx
    │   └── bin
    │       └── phantomjs
    └── phantomjs-1.7.0-macosx.zip

6 directories, 10 files
@cowboy

Why not, in your code:

var binPath = path.join(__dirname, '..')

(I haven't tested it in anything but this OS X machine)

@dpup
Medium member

Ok, I copied your setup completely and I can now reproduce. Previously I was in /tmp but if I create ~/tmp it fails. Bizarre. At least I have something to work from now, thanks.

@dpup
Medium member

But not consistently. wtf.

@dpup
Medium member

So it looks like when it fails the readstream that we pipe to the unzip lib isn't dispatching the end or error events.

@jackfranklin

I've absolutely no idea if this is even relevant, but when I tried installing with npm install phantomjs whilst within a tmux session, it failed. However, exiting out of the tmux session and trying again worked absolutely fine.

As I said, I don't know if it's even relevant in the slightest but thought I'd mention it on the slim chance it might help.

@dpup
Medium member

ok, can someone give 0.2.4 a try. I had it doing a npm cache clean; rm -r node_modules; npm install in a loop and it didn't fail once in 60 tries, previously it'd fail much faster.

@dpup
Medium member

Made a couple more tweaks and updated the unzip library. Latest version is 0.2.6

@dpup
Medium member

Pretty sure those changes fixed it. Please let me know if you continue to see problems.

@dpup dpup closed this Dec 19, 2012
@GiantEnemyCrab

Thank you for the wonderful fix.

@mattd

Yep, that seems to have fixed the issue! Thanks so much for your help, @dpup. And thanks to @cowboy for your input!

@TrejGun

I still can reproduce this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment