This repository has been archived by the owner. It is now read-only.

Too many dependencies break the Windows file system #3697

Closed
mcartoixa opened this Issue Jul 25, 2013 · 66 comments

Comments

Projects
None yet
@mcartoixa

On Windows, the file system itself cannot seem to cope very well with a too deeply nested node_modules directory...

We just added bower to our dependencies, and now:

  • the node_modules directory cannot be deleted from a batch command: DEL /S /Q /F node_modules fails.
  • CLOC fails, even when told to ignore the node_modules directory completely.
  • Git clean fails C:\PROGRA~2\Git\bin\git.exe clean -d -f -x.
  • MSBuild item wildcards are not replaced anymore.

We have tried to run npm dedupe to flatten our hierarchy but this is not enough.

Effectively our Continuous Integration and delivery platform is broken for node.js/bower projects. Other people seem to have encountered the same kinds of problems on Windows.

Any heads up? A more efficient dedupe command, perhaps?

Using npm 1.3.2 on node.js 0.10.13. Reproduced on Windows 8 (x64) and Windows Server 2008 R2 (x64).

@dominictarr

This comment has been minimized.

Show comment
Hide comment
@dominictarr

dominictarr Aug 7, 2013

oh, I've been experimenting with a new npm client, https://github.com/dominictarr/npmd
you can do npmd install X --greedy and it installs everything directly into node_modules,
only going deeper if there is a version conflict.

This is all pretty experimental, and npmd is very young, so no promises, but it might fix your problem.

oh, I've been experimenting with a new npm client, https://github.com/dominictarr/npmd
you can do npmd install X --greedy and it installs everything directly into node_modules,
only going deeper if there is a version conflict.

This is all pretty experimental, and npmd is very young, so no promises, but it might fix your problem.

@ralt

This comment has been minimized.

Show comment
Hide comment
@ralt

ralt Aug 27, 2013

Contributor

This sounds like a huge issue. Is it a problem that npm cannot solve? Can we determine exactly what corrupts the filesystem? Is there no workaround? (Like switching an option on or off in ntfs.)

The issue seems serious enough that time should be spent on it. Having no windows, I can't help, but from an external point of view, this looks serious.

Contributor

ralt commented Aug 27, 2013

This sounds like a huge issue. Is it a problem that npm cannot solve? Can we determine exactly what corrupts the filesystem? Is there no workaround? (Like switching an option on or off in ntfs.)

The issue seems serious enough that time should be spent on it. Having no windows, I can't help, but from an external point of view, this looks serious.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Aug 27, 2013

Member

The file system is not corrupted. You just need to manipulate the files in question using the GUI or using UNC paths instead of the command line and traditional paths.

Member

domenic commented Aug 27, 2013

The file system is not corrupted. You just need to manipulate the files in question using the GUI or using UNC paths instead of the command line and traditional paths.

@mcartoixa

This comment has been minimized.

Show comment
Hide comment
@mcartoixa

mcartoixa Aug 27, 2013

Maybe my wording is wrong, as could be my understanding of the problem.
But the symptoms remain:

  • the GUI is not a very practical solution in continuous integration scenarii.
  • node.js tools like npm or grunt can fail too on our systems due to the mere presence of these folders (even on unrelated projects in different directories).
  • I am not sure I could rewrite all our scripts to use UNC paths instead of traditional paths.

@domenic Do you have more insight (maybe links) as to why UNC paths would work? I have once had to manually rename a whole hierarchy of folders to single letter names so that they could be deleted, even using the GUI...

Maybe my wording is wrong, as could be my understanding of the problem.
But the symptoms remain:

  • the GUI is not a very practical solution in continuous integration scenarii.
  • node.js tools like npm or grunt can fail too on our systems due to the mere presence of these folders (even on unrelated projects in different directories).
  • I am not sure I could rewrite all our scripts to use UNC paths instead of traditional paths.

@domenic Do you have more insight (maybe links) as to why UNC paths would work? I have once had to manually rename a whole hierarchy of folders to single letter names so that they could be deleted, even using the GUI...

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Aug 27, 2013

Member

@mcartoixa http://msdn.microsoft.com/en-us/library/Aa365247

My impression was that libuv does this prefixing of "\\?\" transparently for you and so node programs should never run in to this problem. I have not verified however.

Member

domenic commented Aug 27, 2013

@mcartoixa http://msdn.microsoft.com/en-us/library/Aa365247

My impression was that libuv does this prefixing of "\\?\" transparently for you and so node programs should never run in to this problem. I have not verified however.

@mcartoixa

This comment has been minimized.

Show comment
Hide comment
@mcartoixa

mcartoixa Aug 28, 2013

Here's an excerpt of a failing npm install output, due to filesystem related issues:

unbuild isogeo-id@0.0.1
error : error rolling back Error: ENOTEMPTY, rmdir 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob'
error : error rolling back express-hbs@0.2.0 { [Error: ENOTEMPTY, rmdir 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob']
error : error rolling back errno: 53,
error : error rolling back code: 'ENOTEMPTY',
error : error rolling back path: 'D:\\ccnet\\projects\\Isogeo.App\\ContinuousBuild\\work\\node_modules\\isogeo-id\\node_modules\\express-hbs\\node_modules\\glob' }
npm http GET https://registry.npmjs.org/i/-/i-0.3.2.tgz
error : Error: ENOENT, lstat 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md'
error : If you need help, you may report this log at:
error : <http://github.com/isaacs/npm/issues>
error : or email it to:
error : <npm-@googlegroups.com>
error : System Windows_NT 6.1.7601
error : command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
error : cwd D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\build
error : node -v v0.10.13
error : npm -v 1.3.2
error : path D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md
error : fstream_path D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md
error : fstream_type File
error : fstream_class FileWriter
error : code ENOENT
error : errno 34
error : fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fstream\lib\writer.js:284:26
error : fstream_stack Object.oncomplete (fs.js:107:15)
npm http GET https://registry.npmjs.org/esprima/-/esprima-1.0.3.tgz
npm http 304 https://registry.npmjs.org/argparse

The same command would work on other OSs (this is Windows Server 2008 x64). A build on a Windows 7 x64 machine would fail at the grunt stage. Note that the folders described in the output are unlikely to be the cause of the trouble, but at this stage other deeply nested dependencies have already been fetched (based on bower on our case) that trigger all the kinds of strange behaviour described above.

Here's an excerpt of a failing npm install output, due to filesystem related issues:

unbuild isogeo-id@0.0.1
error : error rolling back Error: ENOTEMPTY, rmdir 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob'
error : error rolling back express-hbs@0.2.0 { [Error: ENOTEMPTY, rmdir 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob']
error : error rolling back errno: 53,
error : error rolling back code: 'ENOTEMPTY',
error : error rolling back path: 'D:\\ccnet\\projects\\Isogeo.App\\ContinuousBuild\\work\\node_modules\\isogeo-id\\node_modules\\express-hbs\\node_modules\\glob' }
npm http GET https://registry.npmjs.org/i/-/i-0.3.2.tgz
error : Error: ENOENT, lstat 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md'
error : If you need help, you may report this log at:
error : <http://github.com/isaacs/npm/issues>
error : or email it to:
error : <npm-@googlegroups.com>
error : System Windows_NT 6.1.7601
error : command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
error : cwd D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\build
error : node -v v0.10.13
error : npm -v 1.3.2
error : path D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md
error : fstream_path D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md
error : fstream_type File
error : fstream_class FileWriter
error : code ENOENT
error : errno 34
error : fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fstream\lib\writer.js:284:26
error : fstream_stack Object.oncomplete (fs.js:107:15)
npm http GET https://registry.npmjs.org/esprima/-/esprima-1.0.3.tgz
npm http 304 https://registry.npmjs.org/argparse

The same command would work on other OSs (this is Windows Server 2008 x64). A build on a Windows 7 x64 machine would fail at the grunt stage. Note that the folders described in the output are unlikely to be the cause of the trouble, but at this stage other deeply nested dependencies have already been fetched (based on bower on our case) that trigger all the kinds of strange behaviour described above.

@JoseOneTime

This comment has been minimized.

Show comment
Hide comment
@JoseOneTime

JoseOneTime Dec 31, 2013

Is there a resolution for this problem yet? I'm running into it on Windows 7.

Is there a resolution for this problem yet? I'm running into it on Windows 7.

@ghost23

This comment has been minimized.

Show comment
Hide comment
@ghost23

ghost23 Feb 5, 2014

I am running into this, too. I can install almost no packages, always get errors like shown above. If i browse through the folder structure, i see huge nested 'node_modules' structures.

ghost23 commented Feb 5, 2014

I am running into this, too. I can install almost no packages, always get errors like shown above. If i browse through the folder structure, i see huge nested 'node_modules' structures.

@clca

This comment has been minimized.

Show comment
Hide comment
@clca

clca Feb 6, 2014

to repro the problem I did the following:
create a temp directory
ran npm install bower
cd node_modules/bower
npm install
(at that point I could see the huge node_modules structure)

I ran rmdir /s /q node_modules (from the temp directory) to delete the whole directory structure. It worked just fine on my Win8 machine.

clca commented Feb 6, 2014

to repro the problem I did the following:
create a temp directory
ran npm install bower
cd node_modules/bower
npm install
(at that point I could see the huge node_modules structure)

I ran rmdir /s /q node_modules (from the temp directory) to delete the whole directory structure. It worked just fine on my Win8 machine.

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Feb 7, 2014

Member

It looks like this is related to a bug that's been fixed in later versions of rimraf and npm. Please update, and it should be fixed.

While the Windows cmd program may have issues with long folder names, Node (and thus npm) always uses the UNC paths internally for fs operations, so it should work fine in npm.

If you want to remove a folder, you can do npm install -g rimraf and then use the rimraf bin to do this. rimraf path\to\folder

Member

isaacs commented Feb 7, 2014

It looks like this is related to a bug that's been fixed in later versions of rimraf and npm. Please update, and it should be fixed.

While the Windows cmd program may have issues with long folder names, Node (and thus npm) always uses the UNC paths internally for fs operations, so it should work fine in npm.

If you want to remove a folder, you can do npm install -g rimraf and then use the rimraf bin to do this. rimraf path\to\folder

@isaacs isaacs closed this Feb 7, 2014

@calbertts

This comment has been minimized.

Show comment
Hide comment
@calbertts

calbertts Feb 11, 2014

rimraf command works for me, thanks.

Shit Windows!.

rimraf command works for me, thanks.

Shit Windows!.

@mcartoixa

This comment has been minimized.

Show comment
Hide comment
@mcartoixa

mcartoixa Feb 11, 2014

But what if I want to use CLOC or MSBuild item wildcards while the node_modules folder is still around?

But what if I want to use CLOC or MSBuild item wildcards while the node_modules folder is still around?

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Feb 11, 2014

Member

I suppose you can only hope those programs are as well designed as node and npm are, and can cope with paths longer than 256 characters!

Member

domenic commented Feb 11, 2014

I suppose you can only hope those programs are as well designed as node and npm are, and can cope with paths longer than 256 characters!

@jamesplease jamesplease referenced this issue in gruntjs/grunt-contrib-imagemin May 1, 2014

Closed

Can't delete because folder path is too long. #209

@neenhouse

This comment has been minimized.

Show comment
Hide comment
@neenhouse

neenhouse May 4, 2014

I just wrote a node module "npm-flatten" that can do this. Check it out here: https://www.npmjs.org/package/npm-flatten. Source is here: https://github.com/neenhouse/npm-flatten

I just wrote a node module "npm-flatten" that can do this. Check it out here: https://www.npmjs.org/package/npm-flatten. Source is here: https://github.com/neenhouse/npm-flatten

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jun 23, 2014

An easy solution would be have an option to symlink dependencies directly into the top level node_modules folder. Say you had something like bower. The top level node_modules directory would have a folder structure looking something like:

node_modules
├───abbrev
│   └───1.0.5
├───amdefine
│   └───0.1.0
├───ansi-styles
│   └───1.0.0
├───ansicolors
│   └───0.2.1
├───archy
│   └───0.0.2
├───argparse
│   └───0.1.15
├───array-filter
│   └───0.0.1
├───array-map
│   └───0.0.0
├───array-reduce
│   └───0.0.0
├───asn1
│   └───0.1.11
├───assert-plus
│   └───0.1.2
├───async
│   ├───0.2.10
│   └───0.9.0
├───aws-sign
│   └───0.3.0
├───aws-sign2
│   └───0.5.0
├───binary
│   └───0.3.0
├───block-stream
│   └───0.0.7
├───boom
│   └───0.4.2
├───bower
│   └───1.3.4
├───bower-config
│   └───0.5.1
├───bower-endpoint-parser
│   └───0.2.1
├───bower-json
│   └───0.4.0
├───bower-logger
│   └───0.2.2
├───bower-registry-client
│   └───0.2.1
├───buffers
│   └───0.1.1
├───cardinal
│   └───0.4.4
├───chainsaw
│   └───0.1.0
├───chalk
│   └───0.4.0
├───chmodr
│   └───0.1.0
├───cli-color
│   └───0.2.3
├───configstore
│   ├───0.2.3
│   └───0.3.1
├───cookie-jar
│   └───0.3.0
├───core-util-is
│   └───1.0.1
├───cryptiles
│   └───0.2.2
├───ctype
│   └───0.5.2
├───decompress-zip
│   └───0.0.8
├───deep-extend
│   └───0.2.10
├───es5-ext
│   └───0.9.2
├───esprima
│   └───1.0.4
├───event-emitter
│   └───0.2.2
├───forever-agent
│   └───0.5.2
├───form-data
│   └───0.1.3
├───fstream
│   └───0.1.27
├───fstream-ignore
│   └───0.0.8
├───glob
│   └───3.2.11
├───graceful-fs
│   ├───2.0.3
│   └───3.0.2
├───handlebars
│   └───1.3.0
├───has-color
│   └───0.1.7
├───hawk
│   └───1.0.0
├───hoek
│   └───0.9.1
├───http-signature
│   └───0.10.0
├───inherits
│   └───2.0.1
├───inquirer
│   └───0.4.1
├───insight
│   └───0.3.1
├───intersect
│   └───0.0.3
├───is-root
│   └───0.1.0
├───isarray
│   └───0.0.1
├───js-yaml
│   └───3.0.2
├───json-stringify-safe
│   └───5.0.0
├───jsonify
│   └───0.0.0
├───junk
│   └───0.3.0
├───lockfile
│   └───0.4.2
├───lodash
│   └───2.4.1
├───lodash.debounce
│   └───2.4.1
├───lodash.isfunction
│   └───2.4.1
├───lodash.isnative
│   └───2.4.1
├───lodash.isobject
│   └───2.4.1
├───lodash.now
│   └───2.4.1
├───lodash.objecttypes
│   └───2.4.1
├───lru-cache
│   ├───2.3.1
│   └───2.5.0
├───memoizee
│   └───0.2.6
├───minimatch
│   └───0.3.0
├───minimist
│   ├───0.0.10
│   └───0.0.8
├───mkdirp
│   ├───0.3.5
│   └───0.5.0
├───mkpath
│   └───0.1.0
├───mout
│   └───0.9.1
├───mute-stream
│   └───0.0.4
├───next-tick
│   └───0.1.0
├───nopt
│   ├───1.0.10
│   └───2.2.1
├───oauth-sign
│   └───0.3.0
├───object-assign
│   ├───0.1.2
│   └───0.3.1
├───opn
│   └───0.1.2
├───optimist
│   ├───0.3.7
│   └───0.6.1
├───osenv
│   ├───0.0.3
│   └───0.1.0
├───p-throttler
│   └───0.0.1
├───promptly
│   └───0.2.0
├───punycode
│   └───1.2.4
├───q
│   ├───0.9.7
│   └───1.0.1
├───qs
│   └───0.6.6
├───read
│   └───1.0.5
├───readable-stream
│   └───1.1.13-1
├───readline2
│   └───0.1.0
├───redeyed
│   └───0.4.4
├───request
│   ├───2.27.0
│   ├───2.34.0
│   └───2.36.0
├───request-progress
│   └───0.3.1
├───request-replay
│   └───0.2.0
├───retry
│   └───0.6.0
├───semver
│   ├───2.2.1
│   └───2.3.1
├───shell-quote
│   └───1.4.1
├───sigmund
│   └───1.0.0
├───sntp
│   └───0.2.4
├───source-map
│   └───0.1.34
├───stringdecoder
│   └───0.10.25-1
├───stringify-object
│   └───0.2.1
├───strip-ansi
│   └───0.1.1
├───tar
│   └───0.1.19
├───throttleit
│   └───0.0.2
├───through
│   └───2.3.4
├───tmp
│   └───0.0.23
├───touch
│   └───0.0.2
├───tough-cookie
│   └───0.12.1
├───traverse
│   └───0.3.9
├───tunnel-agent
│   ├───0.3.0
│   └───0.4.0
├───uglify-js
│   └───2.3.6
├───underscore
│   └───1.4.4
├───underscore.string
│   └───2.3.3
├───update-notifier
│   └───0.1.10
├───uuid
│   └───1.4.1
├───which
│   └───1.0.5
└───wordwrap
    └───0.0.2

Then each of the dependencies under node_modules for bower would be symlinked to these versions. Same would happen for dependencies of dependencies. At most you would only ever have a folder depth of 3 under node_modules (node_modules/bower/1.3.4/node_modules/<symlink>)

├─┬ bower
│ ├── abbrev (Symlink to ../../../abbrev/1.0.5)
│ ├── archy (Symlink to ../../../archy/0.0.2)
│ ├── bower-config (Symlink to ../../../bower-config/0.5.1)
│ ├── bower-endpoint-parser (Symlink to ../../../bower-endpoint-parser/0.2.1)
│ ├── bower-json (Symlink to ../../../bower-json/0.4.0)
│ ├── bower-logger (Symlink to ../../../bower-logger/0.2.2)
│ ├── bower-registry-client (Symlink to ../../../bower-registry-client/0.2.1)
│ ├── cardinal (Symlink to ../../../cardinal/0.4.4)
│ ├── chalk (Symlink to ../../../chalk/0.4.0)
│ ├── chmodr (Symlink to ../../../chmodr/0.1.0)
│ ├── decompress-zip (Symlink to ../../../decompress-zip/0.0.8)
│ ├── fstream (Symlink to ../../../fstream/0.1.27)
│ ├── fstream-ignore (Symlink to ../../../fstream-ignore/0.0.8)
│ ├── glob (Symlink to ../../../glob/3.2.11)
│ ├── graceful-fs (Symlink to ../../../graceful-fs/2.0.3)
│ ├── handlebars (Symlink to ../../../handlebars/1.3.0)
│ ├── inquirer (Symlink to ../../../inquirer/0.4.1)
│ ├── insight (Symlink to ../../../insight/0.3.1)
│ ├── is-root (Symlink to ../../../is-root/0.1.0)
│ ├── junk (Symlink to ../../../junk/0.3.0)
│ ├── lockfile (Symlink to ../../../lockfile/0.4.2)
│ ├── lru-cache (Symlink to ../../../lru-cache/2.5.0)
│ ├── mkdirp (Symlink to ../../../mkdirp/0.3.5)
│ ├── mout (Symlink to ../../../mout/0.9.1)
│ ├── nopt (Symlink to ../../../nopt/2.2.1)
│ ├── opn (Symlink to ../../../opn/0.1.2)
│ ├── osenv (Symlink to ../../../osenv/0.0.3)
│ ├── p-throttler (Symlink to ../../../p-throttler/0.0.1)
│ ├── promptly (Symlink to ../../../promptly/0.2.0)
│ ├── q (Symlink to ../../../q/1.0.1)
│ ├── request (Symlink to ../../../request/2.34.0)
│ ├── request-progress (Symlink to ../../../request-progress/0.3.1)
│ ├── retry (Symlink to ../../../retry/0.6.0)
│ ├── semver (Symlink to ../../../semver/2.2.1)
│ ├── shell-quote (Symlink to ../../../shell-quote/1.4.1)
│ ├── stringify-object (Symlink to ../../../stringify-object/0.2.1)
│ ├── tar (Symlink to ../../../tar/0.1.19)
│ ├── tmp (Symlink to ../../../tmp/0.0.23)
│ ├── update-notifier (Symlink to ../../../update-notifier/0.1.10)
│ └── which (Symlink to ../../../which/1.0.5)

This would also help with deduplicating because you could easily just repoint the symlink to a module.

This, from the programming point of view, wouldn't be difficult to implement. I'd happily submit a pull request, however it would be a question of whether or not it was considered the correct direction for the project manager as a whole.

ghost commented Jun 23, 2014

An easy solution would be have an option to symlink dependencies directly into the top level node_modules folder. Say you had something like bower. The top level node_modules directory would have a folder structure looking something like:

node_modules
├───abbrev
│   └───1.0.5
├───amdefine
│   └───0.1.0
├───ansi-styles
│   └───1.0.0
├───ansicolors
│   └───0.2.1
├───archy
│   └───0.0.2
├───argparse
│   └───0.1.15
├───array-filter
│   └───0.0.1
├───array-map
│   └───0.0.0
├───array-reduce
│   └───0.0.0
├───asn1
│   └───0.1.11
├───assert-plus
│   └───0.1.2
├───async
│   ├───0.2.10
│   └───0.9.0
├───aws-sign
│   └───0.3.0
├───aws-sign2
│   └───0.5.0
├───binary
│   └───0.3.0
├───block-stream
│   └───0.0.7
├───boom
│   └───0.4.2
├───bower
│   └───1.3.4
├───bower-config
│   └───0.5.1
├───bower-endpoint-parser
│   └───0.2.1
├───bower-json
│   └───0.4.0
├───bower-logger
│   └───0.2.2
├───bower-registry-client
│   └───0.2.1
├───buffers
│   └───0.1.1
├───cardinal
│   └───0.4.4
├───chainsaw
│   └───0.1.0
├───chalk
│   └───0.4.0
├───chmodr
│   └───0.1.0
├───cli-color
│   └───0.2.3
├───configstore
│   ├───0.2.3
│   └───0.3.1
├───cookie-jar
│   └───0.3.0
├───core-util-is
│   └───1.0.1
├───cryptiles
│   └───0.2.2
├───ctype
│   └───0.5.2
├───decompress-zip
│   └───0.0.8
├───deep-extend
│   └───0.2.10
├───es5-ext
│   └───0.9.2
├───esprima
│   └───1.0.4
├───event-emitter
│   └───0.2.2
├───forever-agent
│   └───0.5.2
├───form-data
│   └───0.1.3
├───fstream
│   └───0.1.27
├───fstream-ignore
│   └───0.0.8
├───glob
│   └───3.2.11
├───graceful-fs
│   ├───2.0.3
│   └───3.0.2
├───handlebars
│   └───1.3.0
├───has-color
│   └───0.1.7
├───hawk
│   └───1.0.0
├───hoek
│   └───0.9.1
├───http-signature
│   └───0.10.0
├───inherits
│   └───2.0.1
├───inquirer
│   └───0.4.1
├───insight
│   └───0.3.1
├───intersect
│   └───0.0.3
├───is-root
│   └───0.1.0
├───isarray
│   └───0.0.1
├───js-yaml
│   └───3.0.2
├───json-stringify-safe
│   └───5.0.0
├───jsonify
│   └───0.0.0
├───junk
│   └───0.3.0
├───lockfile
│   └───0.4.2
├───lodash
│   └───2.4.1
├───lodash.debounce
│   └───2.4.1
├───lodash.isfunction
│   └───2.4.1
├───lodash.isnative
│   └───2.4.1
├───lodash.isobject
│   └───2.4.1
├───lodash.now
│   └───2.4.1
├───lodash.objecttypes
│   └───2.4.1
├───lru-cache
│   ├───2.3.1
│   └───2.5.0
├───memoizee
│   └───0.2.6
├───minimatch
│   └───0.3.0
├───minimist
│   ├───0.0.10
│   └───0.0.8
├───mkdirp
│   ├───0.3.5
│   └───0.5.0
├───mkpath
│   └───0.1.0
├───mout
│   └───0.9.1
├───mute-stream
│   └───0.0.4
├───next-tick
│   └───0.1.0
├───nopt
│   ├───1.0.10
│   └───2.2.1
├───oauth-sign
│   └───0.3.0
├───object-assign
│   ├───0.1.2
│   └───0.3.1
├───opn
│   └───0.1.2
├───optimist
│   ├───0.3.7
│   └───0.6.1
├───osenv
│   ├───0.0.3
│   └───0.1.0
├───p-throttler
│   └───0.0.1
├───promptly
│   └───0.2.0
├───punycode
│   └───1.2.4
├───q
│   ├───0.9.7
│   └───1.0.1
├───qs
│   └───0.6.6
├───read
│   └───1.0.5
├───readable-stream
│   └───1.1.13-1
├───readline2
│   └───0.1.0
├───redeyed
│   └───0.4.4
├───request
│   ├───2.27.0
│   ├───2.34.0
│   └───2.36.0
├───request-progress
│   └───0.3.1
├───request-replay
│   └───0.2.0
├───retry
│   └───0.6.0
├───semver
│   ├───2.2.1
│   └───2.3.1
├───shell-quote
│   └───1.4.1
├───sigmund
│   └───1.0.0
├───sntp
│   └───0.2.4
├───source-map
│   └───0.1.34
├───stringdecoder
│   └───0.10.25-1
├───stringify-object
│   └───0.2.1
├───strip-ansi
│   └───0.1.1
├───tar
│   └───0.1.19
├───throttleit
│   └───0.0.2
├───through
│   └───2.3.4
├───tmp
│   └───0.0.23
├───touch
│   └───0.0.2
├───tough-cookie
│   └───0.12.1
├───traverse
│   └───0.3.9
├───tunnel-agent
│   ├───0.3.0
│   └───0.4.0
├───uglify-js
│   └───2.3.6
├───underscore
│   └───1.4.4
├───underscore.string
│   └───2.3.3
├───update-notifier
│   └───0.1.10
├───uuid
│   └───1.4.1
├───which
│   └───1.0.5
└───wordwrap
    └───0.0.2

Then each of the dependencies under node_modules for bower would be symlinked to these versions. Same would happen for dependencies of dependencies. At most you would only ever have a folder depth of 3 under node_modules (node_modules/bower/1.3.4/node_modules/<symlink>)

├─┬ bower
│ ├── abbrev (Symlink to ../../../abbrev/1.0.5)
│ ├── archy (Symlink to ../../../archy/0.0.2)
│ ├── bower-config (Symlink to ../../../bower-config/0.5.1)
│ ├── bower-endpoint-parser (Symlink to ../../../bower-endpoint-parser/0.2.1)
│ ├── bower-json (Symlink to ../../../bower-json/0.4.0)
│ ├── bower-logger (Symlink to ../../../bower-logger/0.2.2)
│ ├── bower-registry-client (Symlink to ../../../bower-registry-client/0.2.1)
│ ├── cardinal (Symlink to ../../../cardinal/0.4.4)
│ ├── chalk (Symlink to ../../../chalk/0.4.0)
│ ├── chmodr (Symlink to ../../../chmodr/0.1.0)
│ ├── decompress-zip (Symlink to ../../../decompress-zip/0.0.8)
│ ├── fstream (Symlink to ../../../fstream/0.1.27)
│ ├── fstream-ignore (Symlink to ../../../fstream-ignore/0.0.8)
│ ├── glob (Symlink to ../../../glob/3.2.11)
│ ├── graceful-fs (Symlink to ../../../graceful-fs/2.0.3)
│ ├── handlebars (Symlink to ../../../handlebars/1.3.0)
│ ├── inquirer (Symlink to ../../../inquirer/0.4.1)
│ ├── insight (Symlink to ../../../insight/0.3.1)
│ ├── is-root (Symlink to ../../../is-root/0.1.0)
│ ├── junk (Symlink to ../../../junk/0.3.0)
│ ├── lockfile (Symlink to ../../../lockfile/0.4.2)
│ ├── lru-cache (Symlink to ../../../lru-cache/2.5.0)
│ ├── mkdirp (Symlink to ../../../mkdirp/0.3.5)
│ ├── mout (Symlink to ../../../mout/0.9.1)
│ ├── nopt (Symlink to ../../../nopt/2.2.1)
│ ├── opn (Symlink to ../../../opn/0.1.2)
│ ├── osenv (Symlink to ../../../osenv/0.0.3)
│ ├── p-throttler (Symlink to ../../../p-throttler/0.0.1)
│ ├── promptly (Symlink to ../../../promptly/0.2.0)
│ ├── q (Symlink to ../../../q/1.0.1)
│ ├── request (Symlink to ../../../request/2.34.0)
│ ├── request-progress (Symlink to ../../../request-progress/0.3.1)
│ ├── retry (Symlink to ../../../retry/0.6.0)
│ ├── semver (Symlink to ../../../semver/2.2.1)
│ ├── shell-quote (Symlink to ../../../shell-quote/1.4.1)
│ ├── stringify-object (Symlink to ../../../stringify-object/0.2.1)
│ ├── tar (Symlink to ../../../tar/0.1.19)
│ ├── tmp (Symlink to ../../../tmp/0.0.23)
│ ├── update-notifier (Symlink to ../../../update-notifier/0.1.10)
│ └── which (Symlink to ../../../which/1.0.5)

This would also help with deduplicating because you could easily just repoint the symlink to a module.

This, from the programming point of view, wouldn't be difficult to implement. I'd happily submit a pull request, however it would be a question of whether or not it was considered the correct direction for the project manager as a whole.

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Jun 23, 2014

Member

So, like i said in my comment on joyent/node, the issue isn't that we haven't invented any flat reification strategies. There are plenty of those, and a lot of them work, even using symlinks. (One single global folder where everything lives, with symlinks all over the damn place, and swapping out the require.paths value, is not an ideal strategy, but there are many others.)

We could, for example, have node_modules/.store/<pkgname>/<version> much like @Xeoncore is proposing. A problem you quickly run into is people doing require('.store/abbrev/1.2.3'), which, if you think people won't do, you have a short memory ;)

We can avoid that by having the name be non-predictable, but then that kinda sucks for a lot of other reasons.

One of the really nice things about nesting node_modules is that it will literally break half the time if you rely on it being in ./node_modules, so people don't do that for long

So, let's say you have:

node_modules
+-- .store
|   +-- a
|   |   +-- 1.2.3
|   |   +-- 2.5.4
|   |
|   +-- b
|       +-- 1.2.3
|       +-- 5.4.3
|
+-- a -> .store/a/1.2.3
    +-- node_modules
        +-- b -> ../.store/b/5.4.3

I guess you'd have to put the a/node_modules/b in .store/a/node_modules/b, and it's a symlink to a dir, so that's how it'd work anyway.

But now you end up with a rats' nest pretty fast, because actually it's this:

node_modules
+-- .store
|   +-- a
|   |   +-- 1.2.3
|   |   |   +-- node_modules
|   |   |       +-- b -> ../../../.store/b/5.4.3
|   |   +-- 2.5.4
|   |
|   +-- b
|       +-- 1.2.3
|       +-- 5.4.3
|
+-- a -> .store/a/1.2.3

Let's say b->c, and c->a@2.5.4, and a@2.5.4 depends on the other version of b.

Updating a dep, then, is not a simple matter of replacing a single dir. you still have to build the tree, analyze the tree, reify the tree. We still have to figure out who needs what, and dedupe it as much as possible, and then analyze the changes, or else we're just trading deep trees for a giant .store dir.

What's more, we MUST maintain the symlinks in each .store/*/node_modules dir, because otherwise they'll fall up to the top level node_modules folder, and potentially get an incompatible version!

My point here is not to shoot down @Xeoncore's proposal, but merely to point out that coming up with the reification strategy is only a small part of the solution (albeit the creative fun part.) The other parts are much less fun:

  1. Figuring out the ways that people will abuse it, and how to make sure that they feel the pain of doing so as early as possible, when it is easy to correct.
  2. Working out all the ways that the strategy will fall over when there are deeply nested dep trees with lots of conflict management.
  3. Implementing the code that predictably guarantees dep version satisfaction in the craziest desert-island scenarios that we can imagine.
Member

isaacs commented Jun 23, 2014

So, like i said in my comment on joyent/node, the issue isn't that we haven't invented any flat reification strategies. There are plenty of those, and a lot of them work, even using symlinks. (One single global folder where everything lives, with symlinks all over the damn place, and swapping out the require.paths value, is not an ideal strategy, but there are many others.)

We could, for example, have node_modules/.store/<pkgname>/<version> much like @Xeoncore is proposing. A problem you quickly run into is people doing require('.store/abbrev/1.2.3'), which, if you think people won't do, you have a short memory ;)

We can avoid that by having the name be non-predictable, but then that kinda sucks for a lot of other reasons.

One of the really nice things about nesting node_modules is that it will literally break half the time if you rely on it being in ./node_modules, so people don't do that for long

So, let's say you have:

node_modules
+-- .store
|   +-- a
|   |   +-- 1.2.3
|   |   +-- 2.5.4
|   |
|   +-- b
|       +-- 1.2.3
|       +-- 5.4.3
|
+-- a -> .store/a/1.2.3
    +-- node_modules
        +-- b -> ../.store/b/5.4.3

I guess you'd have to put the a/node_modules/b in .store/a/node_modules/b, and it's a symlink to a dir, so that's how it'd work anyway.

But now you end up with a rats' nest pretty fast, because actually it's this:

node_modules
+-- .store
|   +-- a
|   |   +-- 1.2.3
|   |   |   +-- node_modules
|   |   |       +-- b -> ../../../.store/b/5.4.3
|   |   +-- 2.5.4
|   |
|   +-- b
|       +-- 1.2.3
|       +-- 5.4.3
|
+-- a -> .store/a/1.2.3

Let's say b->c, and c->a@2.5.4, and a@2.5.4 depends on the other version of b.

Updating a dep, then, is not a simple matter of replacing a single dir. you still have to build the tree, analyze the tree, reify the tree. We still have to figure out who needs what, and dedupe it as much as possible, and then analyze the changes, or else we're just trading deep trees for a giant .store dir.

What's more, we MUST maintain the symlinks in each .store/*/node_modules dir, because otherwise they'll fall up to the top level node_modules folder, and potentially get an incompatible version!

My point here is not to shoot down @Xeoncore's proposal, but merely to point out that coming up with the reification strategy is only a small part of the solution (albeit the creative fun part.) The other parts are much less fun:

  1. Figuring out the ways that people will abuse it, and how to make sure that they feel the pain of doing so as early as possible, when it is easy to correct.
  2. Working out all the ways that the strategy will fall over when there are deeply nested dep trees with lots of conflict management.
  3. Implementing the code that predictably guarantees dep version satisfaction in the craziest desert-island scenarios that we can imagine.
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jun 24, 2014

@isaacs You make a good point with the user abuse situation. I wasn't suggesting a solution, more so trying to encourage discussion and development because in my opinion this is a legitimate and prevalent issue.

To solve the issue with people directly using modules from the .store directory, is prohibit requiring of modules from it. That wouldn't be hard to do would it?

ghost commented Jun 24, 2014

@isaacs You make a good point with the user abuse situation. I wasn't suggesting a solution, more so trying to encourage discussion and development because in my opinion this is a legitimate and prevalent issue.

To solve the issue with people directly using modules from the .store directory, is prohibit requiring of modules from it. That wouldn't be hard to do would it?

@pushpan999

This comment has been minimized.

Show comment
Hide comment
@pushpan999

pushpan999 Jun 25, 2014

@isaacs Thanks a lot. I tried every possible option that I found in google, none of them worked. Finally rimraf worked :)

@isaacs Thanks a lot. I tried every possible option that I found in google, none of them worked. Finally rimraf worked :)

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jul 3, 2014

Fixing this in NPM isn't enough, because normal tools don't do things like this, so when you try to use node with other things, like the Windows Git package, these paths break them by proxy.

Please stop pursuing the existing bad install strategy.

Fixing this in NPM isn't enough, because normal tools don't do things like this, so when you try to use node with other things, like the Windows Git package, these paths break them by proxy.

Please stop pursuing the existing bad install strategy.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jul 3, 2014

@StoneCypher You shouldn't be committing your node_modules folder anyway.

ghost commented Jul 3, 2014

@StoneCypher You shouldn't be committing your node_modules folder anyway.

@justrhysism

This comment has been minimized.

Show comment
Hide comment
@justrhysism

justrhysism Jul 14, 2014

@Xeoncore You're right, we "shouldn't". But unfortunately, in the real-world, we need to. I've been bitten one too many times relying on package versions to handle this for me, and when a client calls for an update to an old project with old dependencies - and for whichever reason these dependencies aren't available, the project cannot be built and many, many hours are wasted trying to get a project to build just so a simply update can be done.

@Xeoncore You're right, we "shouldn't". But unfortunately, in the real-world, we need to. I've been bitten one too many times relying on package versions to handle this for me, and when a client calls for an update to an old project with old dependencies - and for whichever reason these dependencies aren't available, the project cannot be built and many, many hours are wasted trying to get a project to build just so a simply update can be done.

@ryanmargheriti

This comment has been minimized.

Show comment
Hide comment

@Xeoncore Irrelevant.

@graphnode

This comment has been minimized.

Show comment
Hide comment
@graphnode

graphnode Sep 1, 2014

Maybe I'm just showing my ignorance but couldn't we have a simple structure like:

node_modules
+-- a@1.2.3
|   +-- a.js
|   +-- package.json
+-- a@2.5.4
|   +-- a.js
|   +-- package.json
+-- b@1.2.3
|   +-- b.js
|   +-- package.json
+-- b@5.4.3
|   +-- b.js
|   +-- package.json
main.js
package.json

Where, when a module requires another module, node.js tries to:

  • Get the module from the local node_module folder; if not found:
  • Get the module from the main node_module where the application started (like above); if not found:
  • Get the module from the global node_module folder.

Note: All of this with the specific version requested (from package.json) in mind.

Any attempt to directly require a versioned module should throw a error (ex: require('a@2.5.4') or require('node_modules/a@2.5.4'): the versions used are in the package.json.

No symbolic links are used and you can still use deep nesting if you wish (for module development?).

What do you guys think?

Maybe I'm just showing my ignorance but couldn't we have a simple structure like:

node_modules
+-- a@1.2.3
|   +-- a.js
|   +-- package.json
+-- a@2.5.4
|   +-- a.js
|   +-- package.json
+-- b@1.2.3
|   +-- b.js
|   +-- package.json
+-- b@5.4.3
|   +-- b.js
|   +-- package.json
main.js
package.json

Where, when a module requires another module, node.js tries to:

  • Get the module from the local node_module folder; if not found:
  • Get the module from the main node_module where the application started (like above); if not found:
  • Get the module from the global node_module folder.

Note: All of this with the specific version requested (from package.json) in mind.

Any attempt to directly require a versioned module should throw a error (ex: require('a@2.5.4') or require('node_modules/a@2.5.4'): the versions used are in the package.json.

No symbolic links are used and you can still use deep nesting if you wish (for module development?).

What do you guys think?

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Sep 1, 2014

Should work.

Should work.

@bradleyflood

This comment has been minimized.

Show comment
Hide comment

Sounds good @graphnode

@johangu

This comment has been minimized.

Show comment
Hide comment
@johangu

johangu Sep 12, 2014

Over the last two weeks I've had to develop on a Windows machine and I've had the exact same thoughts as @graphnode since with such a structure there's absolutely no need for nested stuff with symlinks as @isaacs describe.

johangu commented Sep 12, 2014

Over the last two weeks I've had to develop on a Windows machine and I've had the exact same thoughts as @graphnode since with such a structure there's absolutely no need for nested stuff with symlinks as @isaacs describe.

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Sep 12, 2014

There has been a fix consensus for more than a year.

There has been a fix consensus for more than a year.

@droppedoncaprica

This comment has been minimized.

Show comment
Hide comment
@droppedoncaprica

droppedoncaprica Sep 12, 2014

The last time this was discussed, someone got so heated the entire thread had to be locked....

In any case, that thread outlines how npm is going to work moving forward in the future.

The last time this was discussed, someone got so heated the entire thread had to be locked....

In any case, that thread outlines how npm is going to work moving forward in the future.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 15, 2014

The only issue with @graphnode's idea is the use of the @ symbol. While only a minor detail, not every operating system likes handling them, especially in a Linux terminal!

ghost commented Sep 15, 2014

The only issue with @graphnode's idea is the use of the @ symbol. While only a minor detail, not every operating system likes handling them, especially in a Linux terminal!

@graphnode

This comment has been minimized.

Show comment
Hide comment
@graphnode

graphnode Sep 16, 2014

@Xeoncore

I noticed that NuGet, which uses a very similar approach, uses dot as the separator between package id and version.

I feel that is kind of weird since the name can also include numbers and dots in npm then you end with example.14.0.1. Is that package example.14 with version 0.1 or package example with version 14.0.1?

For the dot to work we would need to define that version must have 3 numbers or that the name must not end with dot/number... or use another symbol.

@Xeoncore

I noticed that NuGet, which uses a very similar approach, uses dot as the separator between package id and version.

I feel that is kind of weird since the name can also include numbers and dots in npm then you end with example.14.0.1. Is that package example.14 with version 0.1 or package example with version 14.0.1?

For the dot to work we would need to define that version must have 3 numbers or that the name must not end with dot/number... or use another symbol.

@othiym23 othiym23 added enterprise and removed documentation labels Nov 14, 2014

@am11 am11 referenced this issue in sass/node-sass Nov 14, 2014

Closed

Dependencies: Mitigate deep nesting #539

@am11 am11 referenced this issue in madskristensen/WebEssentials2013 Dec 28, 2014

Merged

Fix some bugs with less #1690

@robertjd robertjd referenced this issue in stormpath/stormpath-restify Jan 5, 2015

Open

npm uninstall fails on Windows #1

@hopeseekr

This comment has been minimized.

Show comment
Hide comment
@hopeseekr

hopeseekr Jan 25, 2015

I had to do a very distasteful workaround that may help others:

  1. Install cygwin and rsync.
  2. Load the source code on a Linux OS (I used a Ubuntu VM).
  3. Run npm install on the Ubuntu VM.
  4. Run rsync -aP vmhost:path/node_modules /cygdrive/c/path/to/site.

Cygwin's rsync can handle the long paths. Your only problem is that you can never run npm on windows, ever, or you will have issues. And you have to block many code utilities from ever looking inside the node_modules directory.

I had to do a very distasteful workaround that may help others:

  1. Install cygwin and rsync.
  2. Load the source code on a Linux OS (I used a Ubuntu VM).
  3. Run npm install on the Ubuntu VM.
  4. Run rsync -aP vmhost:path/node_modules /cygdrive/c/path/to/site.

Cygwin's rsync can handle the long paths. Your only problem is that you can never run npm on windows, ever, or you will have issues. And you have to block many code utilities from ever looking inside the node_modules directory.

@justrhysism

This comment has been minimized.

Show comment
Hide comment
@justrhysism

justrhysism Jan 25, 2015

Your only problem is that you can never run npm on windows, ever, or you will have issues.

Seriously @hopeseekr? That's not a solution.

Your only problem is that you can never run npm on windows, ever, or you will have issues.

Seriously @hopeseekr? That's not a solution.

@hopeseekr

This comment has been minimized.

Show comment
Hide comment
@hopeseekr

hopeseekr Jan 25, 2015

@justrhysism That's why the keyword in my statement is _“workaround”_. [I even copied and pasted the smart quotes to add emphasis ;-]

@justrhysism That's why the keyword in my statement is _“workaround”_. [I even copied and pasted the smart quotes to add emphasis ;-]

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jan 25, 2015

@justrhysism The big problem is that the node people blame the npm people, the npm people blame the node people, and if you point out that it's unacceptable for it to be solved years later, they call you a troll and close any thread you try to talk about it on, no matter how many hundreds of people are involved.

@hopeseekr @justrhysism There's an easier workaround that involves a largely forgotten dos command subst, which allows you to mount a given directory as a drive. Mount the halfway point in your directory tree as z:, then delete sensibly from there, then unmount and remove the rest.

@justrhysism The big problem is that the node people blame the npm people, the npm people blame the node people, and if you point out that it's unacceptable for it to be solved years later, they call you a troll and close any thread you try to talk about it on, no matter how many hundreds of people are involved.

@hopeseekr @justrhysism There's an easier workaround that involves a largely forgotten dos command subst, which allows you to mount a given directory as a drive. Mount the halfway point in your directory tree as z:, then delete sensibly from there, then unmount and remove the rest.

@hopeseekr

This comment has been minimized.

Show comment
Hide comment
@hopeseekr

hopeseekr Jan 25, 2015

What's worse is all of the OSX fanboys at the offices that use grunt or gulp that go around exclaiming, "You're running Windows! What do you expect?!" and then this node internested-to-infinity problem is all put on me as if it's my fault for being most productive in the most widely used OS by far.

What's worse is all of the OSX fanboys at the offices that use grunt or gulp that go around exclaiming, "You're running Windows! What do you expect?!" and then this node internested-to-infinity problem is all put on me as if it's my fault for being most productive in the most widely used OS by far.

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jan 25, 2015

@hopeseekr well, all other operating systems should conform to their expectations, you see, and if they've found an api that allows abuse, that means that all software globally should update to the thing they expect, and damn the consequences. this is especially easy since all software worldwide is actively being maintained, and nobody anywhere needs unsupported legacy software for anything.

also what's a red book cd

clearly that's much more straightforward than just having a sensible module deploy system in a single product that claims first class windows support, and didn't bother to learn how the filesystem worked

@hopeseekr well, all other operating systems should conform to their expectations, you see, and if they've found an api that allows abuse, that means that all software globally should update to the thing they expect, and damn the consequences. this is especially easy since all software worldwide is actively being maintained, and nobody anywhere needs unsupported legacy software for anything.

also what's a red book cd

clearly that's much more straightforward than just having a sensible module deploy system in a single product that claims first class windows support, and didn't bother to learn how the filesystem worked

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jan 25, 2015

anything that doesn't work the way they expect is emphasis-bold broken, and should immediately conform to them

anything that doesn't work the way they expect is emphasis-bold broken, and should immediately conform to them

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Jan 25, 2015

Contributor

@StoneCypher I know you're passionate about this, but I'd just like to point out that I've been doing all of my non-release process work on npm so far this year running Windows 8.1 as my sole OS (i.e. not in a vm), and I've already addressed several issues with Windows, as well as working with the io.js team to improve the Windows experience. In addition, npm@3 will be beta soon, and it builds a flatter dependency tree by default. I don't think the npm team can make any more plain that we do care about Windows and are committed to improving the experience of npm on it.

Contributor

othiym23 commented Jan 25, 2015

@StoneCypher I know you're passionate about this, but I'd just like to point out that I've been doing all of my non-release process work on npm so far this year running Windows 8.1 as my sole OS (i.e. not in a vm), and I've already addressed several issues with Windows, as well as working with the io.js team to improve the Windows experience. In addition, npm@3 will be beta soon, and it builds a flatter dependency tree by default. I don't think the npm team can make any more plain that we do care about Windows and are committed to improving the experience of npm on it.

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jan 25, 2015

The commentary mostly regards community positions and tone.

As before, I'm engaging in someone else's pre-existing commentary theretowards.

I like node. A lot. I continue to use it. I am slowly becoming productive in it.

However, every time Windows people point out where Node is doing severely broken things unnecessarily, like abusing the internationalization api's widechar support to stuff bytes where the API explicitly states they're not permitted to be, we are met with claims that because we do things differently, we suck and are broken, and that 40 years of other software's stability should be violated so that a marginally more sophisticated layout algorithm need not be deployed.

We're fine with "this is pretty hard to do because it turns out that it involves rewriting a bunch of the core." ljharb told me about that the other day. I didn't get a clear context on the problem, and to be frank it seems super weird, but if that's the case, then fine.

But that's not what we got. We got mocking and taunting and sophistry that didn't make sense.

This community has an admirable focus on choice of wording to prevent subtle hostility from excluding people. The gender pronoun debate, though I think handled poorly, was on the right topic.

You do realize that more than ninety percent of developers are Windows goons, right?

Have you ever wondered why you have so few of them?

I am glad that you choose to stamp out the exclusory behaviors that you can see, so as to make a more inclusive environment where you can get the greatest participation.

I just wish that you saw more of them. People hang lanterns on them all the time.

Hopeseeker's a fun dude. I met him in the PHP community. Don't ignore him just because you see my name.

The commentary mostly regards community positions and tone.

As before, I'm engaging in someone else's pre-existing commentary theretowards.

I like node. A lot. I continue to use it. I am slowly becoming productive in it.

However, every time Windows people point out where Node is doing severely broken things unnecessarily, like abusing the internationalization api's widechar support to stuff bytes where the API explicitly states they're not permitted to be, we are met with claims that because we do things differently, we suck and are broken, and that 40 years of other software's stability should be violated so that a marginally more sophisticated layout algorithm need not be deployed.

We're fine with "this is pretty hard to do because it turns out that it involves rewriting a bunch of the core." ljharb told me about that the other day. I didn't get a clear context on the problem, and to be frank it seems super weird, but if that's the case, then fine.

But that's not what we got. We got mocking and taunting and sophistry that didn't make sense.

This community has an admirable focus on choice of wording to prevent subtle hostility from excluding people. The gender pronoun debate, though I think handled poorly, was on the right topic.

You do realize that more than ninety percent of developers are Windows goons, right?

Have you ever wondered why you have so few of them?

I am glad that you choose to stamp out the exclusory behaviors that you can see, so as to make a more inclusive environment where you can get the greatest participation.

I just wish that you saw more of them. People hang lanterns on them all the time.

Hopeseeker's a fun dude. I met him in the PHP community. Don't ignore him just because you see my name.

@am11

This comment has been minimized.

Show comment
Hide comment
@am11

am11 Jan 25, 2015

Guys what I have learned from IRC (freenode -> nodejs@freenode channel) is that this issue will be fixed in npm v3 as @othiym23 mentioned. Also @iarna indicated above that she was working on it. TBVH, this is no straightforward workaround for this 256 char limitation of NTFS (not very new after all). Not sure what is the situation with ReFS. Nonetheless, I also sent a feedback for this on Windows 10 preview, _hopefully others will do the same_ in order to make them realize that this limitation needs to be revoked and/or reworked. IMO, as a whole npm team has done much better job compared to other package managers (Nuget Packages, Ruby Gems etc.).

Well the reluctance to Windows in any non-MSFT technology community is very common. Although, some people are quite transparent in delivering their concern (for instance the guy who wrote dietlibc (which I was using to make node-sass binary cross-Linux-distro compatible) has to say something in his no windows passage), but on that stance they forget that others (MacWhack) are doing similar stuff (making revenue -- limiting freedom). Since WindOS being non-pure-Unix-like, it will keep getting this special treatment which is not something really new / surprising. :)

On the flipside, see the work from neovim and clang folks, how hard they had to struggle to make their software compatible with Windows, when VC team was not making it easier for masses up until very recently (C99, C11, C++11, C++14 STL and lang features and since so forth).

am11 commented Jan 25, 2015

Guys what I have learned from IRC (freenode -> nodejs@freenode channel) is that this issue will be fixed in npm v3 as @othiym23 mentioned. Also @iarna indicated above that she was working on it. TBVH, this is no straightforward workaround for this 256 char limitation of NTFS (not very new after all). Not sure what is the situation with ReFS. Nonetheless, I also sent a feedback for this on Windows 10 preview, _hopefully others will do the same_ in order to make them realize that this limitation needs to be revoked and/or reworked. IMO, as a whole npm team has done much better job compared to other package managers (Nuget Packages, Ruby Gems etc.).

Well the reluctance to Windows in any non-MSFT technology community is very common. Although, some people are quite transparent in delivering their concern (for instance the guy who wrote dietlibc (which I was using to make node-sass binary cross-Linux-distro compatible) has to say something in his no windows passage), but on that stance they forget that others (MacWhack) are doing similar stuff (making revenue -- limiting freedom). Since WindOS being non-pure-Unix-like, it will keep getting this special treatment which is not something really new / surprising. :)

On the flipside, see the work from neovim and clang folks, how hard they had to struggle to make their software compatible with Windows, when VC team was not making it easier for masses up until very recently (C99, C11, C++11, C++14 STL and lang features and since so forth).

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jan 25, 2015

Member

@StoneCypher When you complain here about "you all" not taking windows seriously, and "here" is an issue on npm, it's hard to not take that as "npm is not taking windows issues seriously." We're aware that we've got some work to build back trust for Windows users, but I'm hopeful that 2015 will be a year that will change that perception. As far as fixing issues with nodejs, you might try raising the issues again over on the iojs issue board (if you haven't already), as it's possible they'll be more receptive.

@am11 That's right: npm@3's dependency resolution algorithm makes the node_modules folder as flat as possible (that is, if there are no conflicts, it puts all requirements at the top level. This results in a implicitly deduped and very shallow tree. Outside of pathological cases, I anticipate that this will solve the "256 char limit" problem by never producing folders nested deeply enough to hit it.

Member

iarna commented Jan 25, 2015

@StoneCypher When you complain here about "you all" not taking windows seriously, and "here" is an issue on npm, it's hard to not take that as "npm is not taking windows issues seriously." We're aware that we've got some work to build back trust for Windows users, but I'm hopeful that 2015 will be a year that will change that perception. As far as fixing issues with nodejs, you might try raising the issues again over on the iojs issue board (if you haven't already), as it's possible they'll be more receptive.

@am11 That's right: npm@3's dependency resolution algorithm makes the node_modules folder as flat as possible (that is, if there are no conflicts, it puts all requirements at the top level. This results in a implicitly deduped and very shallow tree. Outside of pathological cases, I anticipate that this will solve the "256 char limit" problem by never producing folders nested deeply enough to hit it.

@hopeseekr

This comment has been minimized.

Show comment
Hide comment
@hopeseekr

hopeseekr Jan 26, 2015

@iarna That's great news! And it's a complete reversal from the first comments about this issue, that went something like, "Haha! You run Windows? You Plebe! Suck it!"

@iarna That's great news! And it's a complete reversal from the first comments about this issue, that went something like, "Haha! You run Windows? You Plebe! Suck it!"

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jan 26, 2015

@iarna The io.js people have been great; I'm a semi-regular in their channel.

@iarna The io.js people have been great; I'm a semi-regular in their channel.

@droppedoncaprica

This comment has been minimized.

Show comment
Hide comment
@droppedoncaprica

droppedoncaprica Jan 26, 2015

@StoneCypher As a non-npm or node/io developer just lurking the issue trackers, you've always seemed very combative on the subject of Windows integration, which is why I assume most people aren't taking you seriously on it.

Not trying to poke the bear here; I understand you're frustrated. But I do think the developres are moving towards a solution here.

@StoneCypher As a non-npm or node/io developer just lurking the issue trackers, you've always seemed very combative on the subject of Windows integration, which is why I assume most people aren't taking you seriously on it.

Not trying to poke the bear here; I understand you're frustrated. But I do think the developres are moving towards a solution here.

@StoneCypher

This comment has been minimized.

Show comment
Hide comment
@StoneCypher

StoneCypher Jan 26, 2015

@droppedoncaprica I understand how telling people how it feels to be treated this way must seem combative, yes.

@droppedoncaprica I understand how telling people how it feels to be treated this way must seem combative, yes.

@johnnyreilly

This comment has been minimized.

Show comment
Hide comment
@johnnyreilly

johnnyreilly Jan 28, 2015

@iarna - great to hear that npm@3 is likely to improve life in Windows. Is there any thought as to when that might ship?

@iarna - great to hear that npm@3 is likely to improve life in Windows. Is there any thought as to when that might ship?

@nurbek-ab

This comment has been minimized.

Show comment
Hide comment
@nurbek-ab

nurbek-ab Mar 25, 2015

@StoneCypher thanks a lot! subst solution provided by you works like a charm. All other solutions suggested here didn't work for me but by simply mounting node_modules folder to Z drive I was able to remove the whole folder structure.

@StoneCypher thanks a lot! subst solution provided by you works like a charm. All other solutions suggested here didn't work for me but by simply mounting node_modules folder to Z drive I was able to remove the whole folder structure.

@DullReferenceException

This comment has been minimized.

Show comment
Hide comment
@DullReferenceException

DullReferenceException Apr 30, 2015

Is this still being tracked? I see it's been closed but not resolved. My perception when a non-resolved issue is closed is that it's being ignored, which I really hope is not the case.

Is this still being tracked? I see it's been closed but not resolved. My perception when a non-resolved issue is closed is that it's being ignored, which I really hope is not the case.

@kevva

This comment has been minimized.

Show comment
Hide comment
@kevva

kevva Apr 30, 2015

It's being fixed in npm@3.

kevva commented Apr 30, 2015

It's being fixed in npm@3.

@deostroll

This comment has been minimized.

Show comment
Hide comment
@deostroll

deostroll May 4, 2015

You can uninstall node modules in that folder serially to solve the problem. http://stackoverflow.com/a/29915329/145682

You can uninstall node modules in that folder serially to solve the problem. http://stackoverflow.com/a/29915329/145682

@am11 am11 referenced this issue in sass/sass May 16, 2015

Closed

Windows cache problem #1563

@chrispappas

This comment has been minimized.

Show comment
Hide comment
@chrispappas

chrispappas May 28, 2015

Found this suggestion for using robocopy with an empty directory and node_modules, using the purge switch:
http://stackoverflow.com/a/3320317

Paraphrased TL;DR:
robocopy empty_dir node_modules /purge

I wasn't able to install rimraf because of a borked up npm directory... hoping I can get everything back working again. Hope this helps someone!

Found this suggestion for using robocopy with an empty directory and node_modules, using the purge switch:
http://stackoverflow.com/a/3320317

Paraphrased TL;DR:
robocopy empty_dir node_modules /purge

I wasn't able to install rimraf because of a borked up npm directory... hoping I can get everything back working again. Hope this helps someone!

@crertel

This comment has been minimized.

Show comment
Hide comment
@crertel

crertel Jun 2, 2015

So, what is the solution in npm@3 going to be? And how far away is that, exactly?

I'm rather blocked on this at work, and honestly I'm really getting a bit sick of dealing with the Node ecosystem's quirks. I don't want to clutter up this thread with continued kvetching, but I'm really a bit puzzled that a problem well-solved by, say, Bundler and Hex is somehow still causing grief.

(And unfortunately, yes, some of us are stuck with Windows deployments--and no, we'd rather not be, but them's the breaks.)

crertel commented Jun 2, 2015

So, what is the solution in npm@3 going to be? And how far away is that, exactly?

I'm rather blocked on this at work, and honestly I'm really getting a bit sick of dealing with the Node ecosystem's quirks. I don't want to clutter up this thread with continued kvetching, but I'm really a bit puzzled that a problem well-solved by, say, Bundler and Hex is somehow still causing grief.

(And unfortunately, yes, some of us are stuck with Windows deployments--and no, we'd rather not be, but them's the breaks.)

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jun 23, 2015

Member

npm@3 makes the install "maximally flat" by hoisting everything it can to the top level node_modules. This means nesting only occurs on conflicts and as such, trees should never get very deep. As such, the windows path length limitation shouldn't be run into.

Timeline-wise, we're looking to do the first public release this week.

Member

iarna commented Jun 23, 2015

npm@3 makes the install "maximally flat" by hoisting everything it can to the top level node_modules. This means nesting only occurs on conflicts and as such, trees should never get very deep. As such, the windows path length limitation shouldn't be run into.

Timeline-wise, we're looking to do the first public release this week.

@hopeseekr

This comment has been minimized.

Show comment
Hide comment
@hopeseekr

hopeseekr Jun 24, 2015

Thank God[dess]!! After the release, NodeJS can announce it once again supports Windows operating systems! I've been battling this issue for... get this!! 2 years and 4 months! For such a critical issue for so many of my clients, I am very happy something is finally being done. Because, frankly, I've had to create a pure PHP equivalent of npm, Gulp and Grunt that doesn't have the damn nested problem. And it works a lot better, too.

Thank God[dess]!! After the release, NodeJS can announce it once again supports Windows operating systems! I've been battling this issue for... get this!! 2 years and 4 months! For such a critical issue for so many of my clients, I am very happy something is finally being done. Because, frankly, I've had to create a pure PHP equivalent of npm, Gulp and Grunt that doesn't have the damn nested problem. And it works a lot better, too.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jun 24, 2015

Member

I'm locking comments here as they're no longer constructive. Anything new issues around the things discussed here should be opened as new issues.

Member

iarna commented Jun 24, 2015

I'm locking comments here as they're no longer constructive. Anything new issues around the things discussed here should be opened as new issues.

@npm npm locked and limited conversation to collaborators Jun 24, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.