This repository has been archived by the owner. It is now read-only.

`npm rebuild` should not fail when an optional dependency fails to build #5095

Closed
jamesshore opened this Issue Apr 17, 2014 · 16 comments

Comments

Projects
None yet
@jamesshore
Copy link

jamesshore commented Apr 17, 2014

For people who check their node_modules into source control, it's possible for npm rebuild to fail when an optional dependency was installed on a different platform. This issue can be fixed by making sure npm rebuild works the same as npm install: it should not fail when an optional dependency fails to build. Instead, the optional dependency should be... removed, perhaps?

Example:

Chokidar installs fine on Mac OS

Install Chokidar on Mac OS: npm install chokidar@0.8.2.

Chokidar installs the Mac-only fsevents module and everything works fine.

Chokidar installs fine on Windows

Install Chokidar on Windows: npm install chokidar@0.8.2.

The fsevents module doesn't work on Windows and it isn't installed. That's okay, because fsevents is an optionalDependency. Everything works fine.

Chokidar fails if installed on Mac and rebuilt on Windows

Install Chokidar on the Mac and check it in:

npm install chokidar@0.8.2 --ignore-scripts
echo node_modules/chokidar/node_modules/fsevents/build >> .gitignore
git add . && git commit -m "Installed Chokidar 0.8.2"
npm rebuild

The fsevents module is installed and everything works fine.

Now clone the git repository on Windows and rebuild: git clone XXX && cd XXX && npm rebuild

The rebuild fails even though fsevents is an optionalDependency. It should not fail. I'm not sure what the correct behavior should be in the general case, but for this specific example, it would be appropriate to remove the fsevents module.

@jamesshore jamesshore changed the title `npm rebuild` should not fail when building `optionalDependencies` `npm rebuild` should not fail when an optional dependency fails to build Apr 17, 2014

@elwayman02

This comment has been minimized.

Copy link

elwayman02 commented May 19, 2014

+1.

@pshomov

This comment has been minimized.

Copy link

pshomov commented May 27, 2014

+1

@CMTegner

This comment has been minimized.

Copy link

CMTegner commented Jun 5, 2014

FYI: npm i --no-optional might be a good workaround for this situation. Another solution would be to add a flag to npm prune which would remove optional dependencies.

@othiym23 othiym23 added bug labels Jun 5, 2014

@quicksnap

This comment has been minimized.

Copy link

quicksnap commented Jun 10, 2014

This causes issues with our cross-platform automated builds. Will try --no-optional, but then I assume the environment would be missing said dependencies.

In chokidar's case, it would then lack fsevents for file watching. This would make setting up developer machines for better file watching a pain, tiptoeing around accidentally checking in the optional dependency.

@CMTegner

This comment has been minimized.

Copy link

CMTegner commented Jun 11, 2014

I'm assuming you are committing node_modules to avoid build issues if a package disappears/npm is unavailable, but the developers could run npm install after pulling to ensure they have every module their current environment supports.

@quicksnap

This comment has been minimized.

Copy link

quicksnap commented Jun 11, 2014

@CMTegner -- You're right; seems like a reasonable workaround. However, npm rebuild not failing would be an awesome improvement.

Thanks!

@jgrund

This comment has been minimized.

Copy link

jgrund commented Jul 31, 2014

I'm running into this case for offline building as well. I would be nice to have the full dependency structure represented but be able to recover from errors on optional deps. Perhaps a flag on npm rebuild?

What if npm looked at the package.json of the module being rebuilt? If rebuilding the module throws and it's an optionalDependency in the package.json then just continue on.

@BlueHotDog

This comment has been minimized.

Copy link

BlueHotDog commented Sep 7, 2014

having the same issue
+1

@andreaskern

This comment has been minimized.

Copy link

andreaskern commented Sep 8, 2014

+1

The package 'fsevents' is workaround for the node.js fs module. I think fsevents should be merged into node.js so that it is unnecessary to handle platform specific workaround packages.

@othiym23

This comment has been minimized.

Copy link
Contributor

othiym23 commented Sep 13, 2014

We believe this (incredibly irritating) issue in how optionalDependencies are rolled back has been addressed by the resolution of #6043. See that issue for details on how we solved the problem and how you can solve the fix. Thanks for your patience, and sorry for the wait!

@othiym23 othiym23 closed this Sep 13, 2014

@es128

This comment has been minimized.

Copy link
Contributor

es128 commented Nov 8, 2014

The #6043 solution didn't actually apply here, and npm rebuild of an optional dependency still fails in this situation. I just reproduced it on Ubuntu with both npm 1.4.23 and 2.1.7 using fsevents@0.3.0.

However, fsevents@0.3.1 was able to solve this by reconfiguring its gyp bindings to just build an empty bundle by default, and only do the real compilation conditionally by OS.

@dsebastien

This comment has been minimized.

Copy link

dsebastien commented Jun 30, 2015

I dont know if this is actually fixed but I wasted hours just trying to get node-gyp to work, realizing that I needed to install ~10GB of binaries just so that it could build an optional dependency that I do not care about at all. --no-optional helped me work around the issue but the package installation really shouldn't fail because of optional dependencies. Moreover the node-gyp thing is just horrendous on Windows (though that ain't npm's fault I guess :p);

@jgrund

This comment has been minimized.

Copy link

jgrund commented Nov 10, 2015

Seeing this exact problem. using node v5.0.0 and npm v3.3.12

I do an install with --ignore-scripts. This installs and optionalDependencies that failed to build are not trimmed out.

Then I do an npm rebuild. At this point it tries to build the dependencies. The issue is it now treats all deps as not optional, so if they don't build the whole process fails.

@othiym23

This comment has been minimized.

Copy link
Contributor

othiym23 commented Nov 10, 2015

So, this sounds like a new regression in npm@3 to me, and not the original issue from 2014. Can somebody open a new issue with as many details about their package.json, any shrinkwraps that may be getting pulled in, and the like, so we can get this fixed in npm@3?

@jgrund

This comment has been minimized.

Copy link

jgrund commented Nov 10, 2015

@othiym23 sure. Just created: #10335

@silkentrance

This comment has been minimized.

Copy link

silkentrance commented Nov 5, 2016

@othiym23 @jgrund this also applies to npm install and npm update as well.

N.B. previous npm versions prior to at least 3.10.9 did not expose that kind of behavior.

At least with version 3.5.2 it does not show

cklein@gandalf:/var/workspaces/coldrye/collab/openproject/frontend$ npm --version
3.5.2
cklein@gandalf:/var/workspaces/coldrye/collab/openproject/frontend$ npm install
npm WARN prefer global coffee-script@1.8.0 should be installed with -g

> fsevents@1.0.14 install /var/workspaces/coldrye/collab/openproject/frontend/node_modules/fsevents
> node-pre-gyp install --fallback-to-build

node-pre-gyp ERR! Tried to download: https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.0.14/fse-v1.0.14-node-v46-linux-x64.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for fsevents@1.0.14 and node@4.6.1 (node-v46 ABI) (falling back to source compile with node-gyp) 
make: Entering directory '/var/workspaces/coldrye/collab/openproject/frontend/node_modules/fsevents/build'
  SOLINK_MODULE(target) Release/obj.target/.node
  COPY Release/.node
make: Leaving directory '/var/workspaces/coldrye/collab/openproject/frontend/node_modules/fsevents/build'

> ws@0.4.32 install /var/workspaces/coldrye/collab/openproject/frontend/node_modules/ws
> (node-gyp rebuild 2> builderror.log) || (exit 0)

make: Entering directory '/var/workspaces/coldrye/collab/openproject/frontend/node_modules/ws/build'
  CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
bufferutil.target.mk:92: recipe for target 'Release/obj.target/bufferutil/src/bufferutil.o' failed
make: Leaving directory '/var/workspaces/coldrye/collab/openproject/frontend/node_modules/ws/build'

> phantomjs-prebuilt@2.1.13 install /var/workspaces/coldrye/collab/openproject/frontend/node_modules/phantomjs-prebuilt
> node install.js

PhantomJS not found on PATH
Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
Receiving...
  [=======================================-] 97%
Received 22866K total.
Extracting tar contents (via spawned process)
Removing /var/workspaces/coldrye/collab/openproject/frontend/node_modules/phantomjs-prebuilt/lib/phantom
Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1478384567199/phantomjs-2.1.1-linux-x86_64 -> /var/workspaces/coldrye/collab/openproject/frontend/node_modules/phantomjs-prebuilt/lib/phantom
Writing location.js file                                                                                                                                                                                                                     
Done. Phantomjs binary available at /var/workspaces/coldrye/collab/openproject/frontend/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs                                                                                            

> openproject-frontend@0.1.0 postinstall /var/workspaces/coldrye/collab/openproject/frontend                                                                                                                                                 
> bower install --config.interactive=false                                                                                                                                                                                                   

openproject-frontend@0.1.0 /var/workspaces/coldrye/collab/openproject/frontend
├── amdefine@1.0.0                                                                                                                                                                                                                           
├─┬ angular-elastic@2.5.1
[...]

And the install just worked without any error, including provisioning of the fsevents package under ./node_modules/fsevents.

It does show, however, on install with

(nodeenv)cklein@gandalf:/var/workspaces/coldrye/opp/openproject/frontend$ npm --version
3.10.9
(nodeenv)cklein@gandalf:/var/workspaces/coldrye/opp/openproject/frontend$ npm install
npm WARN openproject-frontend@0.1.0 No repository field.
npm WARN openproject-frontend@0.1.0 No license field.
npm ERR! Linux 4.7.0-1-amd64
npm ERR! argv "/var/workspaces/coldrye/opp/nodeenv/bin/node" "/var/workspaces/coldrye/opp/nodeenv/bin/npm" "install"
npm ERR! node v7.0.0
npm ERR! npm  v3.10.9
npm ERR! code EBADPLATFORM

npm ERR! notsup Unsupported platform for fsevents@1.0.14: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm ERR! notsup Valid OS:    darwin
npm ERR! notsup Valid Arch:  any
npm ERR! notsup Actual OS:   linux
npm ERR! notsup Actual Arch: x64
> echo $?
1

and also on update, however errors now become simple warnings instead

(nodeenv)cklein@gandalf:/var/workspaces/coldrye/opp/openproject/frontend$ npm update
openproject-frontend@0.1.0 /var/workspaces/coldrye/collab/openproject/frontend
├── amdefine@1.0.1 
└── typescript@2.0.7 

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.0.15: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN openproject-frontend@0.1.0 No repository field.
npm WARN openproject-frontend@0.1.0 No license field.
> echo $?
0
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.