Node's nested node_modules approach is basically incompatible with Windows #6960

Closed
jonrimmer opened this Issue Jan 24, 2014 · 175 comments

Projects

None yet
@jonrimmer

Node needs an alternative approach to endless, recursively nested node_modules folders on Windows. Most Windows tools, utilities and shells cannot handle file and folder paths longer than 260 characters at most. This limit is easily exceeded, and once it is, install scripts start breaking and node_modules folders can no longer be deleted using conventional methods.

This will only get worse as Node packages get more complicated, with bigger and deeper dependency hierarchies. There should be a way to install and run packages that does not use file system recursion to represent the dependency hierarchy.

@Mithgol
Mithgol commented Jan 27, 2014

Suggestion:   to mitigate the above mentioned problem, make Node.js version 1.0 (and the corresponding npm) use n_m (3 characters) instead of node_modules (12 characters) by default. This change is expected to double the possible hierarchy depth for modules with names containing nine letters or less.

If a backwards compatibility is necessary, node_modules could still be supported (though not as default). For example,

  • require('modulename') could look in node_modules if the module is not found in n_m,
  • npm update modulename could kill node_modules/modulename (if it exists) and install n_m/modulename and dependencies.
@OrangeDog

That doesn't solve the problem, just gives you slightly more breathing room.

The only real solution is going to have all modules at a single level, requiring each other, instead of all having private copies of their dependencies. This would either have to be versioned, and/or module maintainers need to be more careful about breaking changes.
Most package managers work this way, and seem to be getting along nicely.

@indutny
Member
indutny commented Jan 27, 2014

As an idea: node_modules/module@version

@Mithgol
Mithgol commented Mar 4, 2014

The module@version solution eliminates the problem much better than n_m, but not without introducing a bunch of other problems.

Example: if somemodulename@1.2.3 and somemodulename@4.5.6 are both present, what should npm update somemodulename do?

In the same example, what should npm uninstall somemodulename do?

@randunel
randunel commented Mar 4, 2014

@Mithgol it should probably install somemodule@la.te.st, alongside the others. But npm purge or npm clean should probably be added.

@OrangeDog

@Mithgol if the latest version is 4.5.6 do nothing, otherwise add the latest version. If you can be clever and determine that a previous version is no longer needed then remove it, otherwise don't worry about it.

@randunel somemodule@latest would be far more sensible if there's an actual need for it, though for backwards-compatibility, I'd guess somemodule would symlink to the latest anyway.

@jdalton
jdalton commented Mar 20, 2014

👍, devs are running into this with individual lodash packages too.
See lodash/lodash#501, gulpjs/gulp-util#23, & twitter/theporchrat.

There looks to be the start of addressing the issue here, but it may need to be used in other methods.
Related to #6891.

@Mithgol
Mithgol commented Mar 21, 2014

I guess Node's developers would have to rewrite both require() and npm to use \\?\… paths. If it's possible, 256-character limitation becomes void — even without resorting to n_m or @version.

(Maybe not exactly require() and npm, but rather the entire underlying fs module, also process.cwd(), etc.)

@jonrimmer

Even if all of Node and NPM are rewritten to use long paths, so long as NPM supports scripts there is a danger of them randomly blowing up when run on Windows due to finding themselves inside a very deep hierarchy then trying to run utilities that don't support long paths.

@Mithgol
Mithgol commented Mar 22, 2014

@JonRimmer Many third-party scripts for NPM (such as Grunt, or Mocha, or node-pre-gyp) are Node.js scripts and thus they are to become automagically fixed when Node.js core modules (such as fs and path) and path-related methods (such as process.cwd()) all start using \\?\… unilaterally.

The other utilities would indeed break and thus the community would have to replace them or demand upgrades. (A similar ruckus once happened when Node.js started supporting Windows and many UN*X-only tools and CLI scripts experienced a lot of incompatibility-related problems. There was much suffering; however, the community eventually fixed the problems or developed some workarounds, sometimes as simple as using path.join instead of a former + '/' +. Or using os.EOL instead of a former \n.)

@langri-sha

Hello everyone! I was unpleasantly surprised when I found out our module was hitting an unexpected wall in Windows host environments. If anyone needs a quick remedy:

  • You can delete the Windows paths by slicing out a branch of the Node dependency tree and moving it to a shorter path in the host environment.
  • If you are developing on a Windows host, but using a Unix environment in production, you can use Vagrant to create a virtual machine, and run you application in the guest environment.
    If you would like to keep your sources and Node modules synced with the Windows host, you can install lodash in the guest environment and use npm link to resolve it as a module dependency for your application.
@domenic
Member
domenic commented Apr 19, 2014

FYI Node (and thus npm) always uses UNC paths internally, before actually calling into the filesystem. It is only third-party tools that have a problem. (Unfortunately one of those third-party tools is Windows Explorer, but, that's Microsoft's bug...)

@jdalton
jdalton commented Apr 20, 2014

FYI Node (and thus npm) always uses UNC paths internally

I do see the _makeLong() helper used a lot in fs though is there a chance something was missed?
Is it handled in require?

@jonrimmer

@domenic Microsoft have said many times that MAX_PATH limitations are not considered a bug, just a feature of Windows. The file system supports long paths, the OS does not, there is a difference.

If Node's intention is to support NTFS, then long paths are fine. If the intention is to support Windows, then there needs to be an alternative to long paths.

@domenic
Member
domenic commented Apr 20, 2014

@JonRimmer The OS does support long paths, or at least all API methods that libuv uses support long paths. So the distinction you are trying to make does not apply.

@jdalton require delegates to fs.

@jdalton
jdalton commented Apr 20, 2014

@domenic Cool.

I'm experimenting with this on my Windows 8.1 machine (I haven't hit this issue on my own yet):

  • I created several nested long name directories and navigated via the command line as far as I could before receiving The filename or extension is too long. errors.
  • I then did npm install lodash.template. While I couldn't create further folders in Explorer or navigate via the command line the npm install succeeded and the folders were created.
  • Once they were created by npm I could navigate to them with Explorer
    (still not with the command line though).
  • In Node, from the directory I performed npm install in, I did require('lodash.template') and it loaded the module without problems.
  • I even did require('lodash.template/node_modules/lodash.defaults/node_modules/lodash._objecttypes') and it loaded the module w/o issue.
  • I noticed in Explorer when I try to copy the path it gives me a truncated form:
    C:\Users\jdalton\Desktop\A12345~1\A12345~1\A12345~1\A12345~1\NODE_M~1\LODASH~1.TEM\NODE_M~1\LODASH~1.DEF\node_modules\lodash._objecttypes.
  • Also if I cd <that truncated path> I can navigate to the path in the command line as well. So now I'm in the directory in the command line and in Explorer (I can also create more folders/files) and the module is loading in Node.
@jonrimmer

@domenic Microsoft: "In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters." [1]

Some (but not all) Windows APIs support, as an alternative, unicode paths up to 32,767 chars, but this is an alternative - not the default. It is not the option used in many Windows utilities including core parts of the OS. Claiming therefore, that Windows supports long paths, is like claiming old NT was Unix compatible because there is a POSIX subsystem, or claiming Linux is compatible with the win32 APIs because you can install Wine. It is a entirely disingenuous attempt to misrepresent the actual reality. Windows as an OS does not support long paths. That it is possible to write Windows compatible software that supports long paths does not change that fact.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

@domenic
Member
domenic commented Apr 21, 2014

I don't really care about whatever word games you want to play around the word "support." Sure, if you wish, all I'm saying is that Node and npm are Windows compatible software that support long paths.

@jonrimmer

You are the one playing games - calling core parts of Windows like Explorer "3rd party tools", and suggesting that not supporting long paths is a bug. Microsoft have made it clear repeatedly that non-support for long paths is not a bug, and not something that will change.

A package manager creating paths that do not work with the majority of the software written for an OS, then claiming compatibility with the OS, is playing games, at your users expense.

@edef1c
edef1c commented May 1, 2014

I guess we'll have to stick with just being compatible with node and npm for Windows then.
To argue that the node module system should change due to deficiencies in parts of Windows is a dead end.

@matt-bernhardt

Well, I guess that writes off my ability to use node (or at least some node projects) under a Windows environment, then? I've been attempting to merge a repository elsewhere on Github that uses node, and the merge command fails due to the length of the pathnames.

Maybe there's a workaround I can use (seriously, I'm open to suggestions) - or maybe you can argue that the problem lies with how the Git command line deals with directory paths - but the end result is the same: I can't work with this node project under Windows.

@OrangeDog

Nested dependenices shouldn't be in git repositories. Each dependency has its own repository, and npm assembles them.
You could always open a ticket with git to add support for long paths, just like node does.

@dougwilson

Or you can check out the git repo directly under C: ?

@matt-bernhardt

@OrangeDog - I'm looking into ways that I can work with this repository specifically - I'm not sure how it came to be this way, but it is unworkable currently. My completely-off-the-cuff speculation would be trying to .gitignore the node_modules directory, trusting the node system to build the deep directories correctly.

@dougwilson - that was the first thing I tried, but no dice - even putting the repo at c:/g/ the paths are still too long:

c:/g/node_modules/grunt-contrib-imagemin/node_modules/image-min/node_modules/gifsicle/node_modules/bin-wrapper/node_modules/download/node_modules/request/node_modules/form-data/node_modules/combined-stream/node_modules/delayed-stream/test/integration

@bmeck
bmeck commented May 1, 2014

Just a comment. This appears to be problems with tools that do not support long paths in Windows; is this truly a Node issue or just a common problem with Windows applications?

@Mithgol
Mithgol commented May 2, 2014

So, Node and npm both work with long paths. That's good. But… what if a module contains a C/C++ addon? Can Python and Visual Studio actually build it if a path is long?

@isochronous

The most annoying part of this bug to me is that I can't move or delete folders with long path names inside of them. To do so, I either have to crawl the directory tree manually, slicing out portions and pasting them into a temp folder, or resort to a cygwin shell, which will actually move/delete even very long path names.

One of the things that struck me was that people seem to rely strongly on all of these miniscule lodash mini-packages. Is lodash truly so large, and your requirements truly so strict, that you can't just require all of lodash? If you require a certain lodash mini-package, that package will require other lodash-mini packages, which in turn require their own lodash mini-packages, to the point where you've basically included half of lodash, and your startup times are going to be far higher than they would have been with the full lodash package just from having to load so many individual modules.

The gulp-jshint module is a good example of this. Install it and then trace down the node_modules tree, and see how ridiculously far down the rabbit hole goes.

@jdalton
jdalton commented May 2, 2014

One of the things that struck me was that people seem to rely strongly on all of these miniscule lodash mini-packages. Is lodash truly so large, and your requirements truly so strict, that you can't just require all of lodash?

Modules are great and large dep graphs can and do happen. Trying to poo-poo a dev for using small reusable modules is wonky. Things should "just work".

If you require a certain lodash mini-package, that package will require other lodash-mini packages, which in turn require their own lodash mini-packages, to the point where you've basically included half of lodash, and your startup times are going to be far higher than they would have been with the full lodash package just from having to load so many individual modules.

There are pros & cons to monolithic vs individual module vs bundles of modules.

The gulp-jshint module is a good example of this. Install it and then trace down the node_modules tree, and see how ridiculously far down the rabbit hole goes.

To avoid this issue in the next release Lo-Dash core is inlining ~60 functions into various modules which greatly reduces the dep graphs (many to 0 deps) at the cost of duplication. It's a bummer of a band-aid but it's the state of things at the moment.

@othiym23
othiym23 commented May 2, 2014

To avoid this issue in the next release Lo-Dash core is inlining ~60 functions into various modules which greatly reduces the dep graphs (many to 0 deps) at the cost of duplication. It's a bummer of a band-aid but it's the state of things at the moment.

From a maintainability and quality perspective this kind of sucks, but from a performance perspective this is a win, because those functions will be defined in the same V8 context and can thus be inlined when the various pieces of lodash get hot, which can't be done when helpers are defined in separate modules.

@jdalton
jdalton commented May 2, 2014

From a maintainability and quality perspective this kind of sucks,

Generally yes, but for Lo-Dash maintainability and quality aren't a problem because it's all generated from the same monolithic reference source and all modules are run through the same unit tests. So for me it was just adding a list of always-inlined modules and adding a heuristic to inline functions with only one dependent. Even though functions are inlined the individual modules for each are still created so devs can still use every bit and bob.

but from a performance perspective this is a win, because those functions will be defined in the same V8 context

Good to know.

@jdalton
jdalton commented May 2, 2014

Moving the thread back on track. About two weeks ago I did some experiments to narrow down the issue. I couldn't come up with a case where Node/npm failed however I did find gotchas in the Windows UI/command-line.

For devs running into these issues is there a recommended workaround or best practice?
If so, is it documented?

A bigger question would be is there another dependency structure that Node could use that would avoid the nested issue?

@phated
phated commented May 2, 2014

Haven't read all of this thread, but couldn't this be solved by NPM doing a dedupe on postinstall or at least allowing the npm dedupe command to have a --save option to save the deduped tree?

@mstade
mstade commented May 3, 2014

Aside from being a substantial piece of work, are there any real no-can-do issues with changing the structure to be flat, like the previously suggested node_modules/module@version?

The issues mentioned aren't things that will fundamentally break node, just things that need some thinking to come up with a decent solution. Keeping on with the nested folders approach however is causing real problems, regardless of what one might think of windows explorer or other applications that don't play nice with long paths.

@vkurchatkin
Member

@mstade what require('module-name') is supposed to do if there is node_modules/module-name@version1 and node_modules/module-name@version2?

@mstade
mstade commented May 4, 2014

@vkurchatkin node would need a new look-up algorithm of course, presumably making use of package.json in order to figure out version information and such.

I doubt that it's impossible to come up with a new algorithm that works with a flat folder structure. The current algorithm is simple and intuitive, but the assumption that deep folder nesting is A-OK doesn't hold up on a major OS, so it might be worth spending a few brain cells figuring out a more fitting solution.

I don't think this is rocket surgery; but I also don't doubt it'd be a fair bit of work to implement, particularly in making sure existing projects don't get shafted.

@bmeck
bmeck commented May 4, 2014

@mstade There are several issues at hand with a naive flatten of all modules with the same version to a single folder. The following is not a complete list, but are real world problems that would need to be argued against as it could/would damage experiences on other OS.

  1. Shared state expectations in modules is broken. See the following as a simple example:

    var uid = 0;
    module.exports = function next() {
     return ++uid;
    }

    If this is originally 2 submodules and is then converted into a single one (name@version); the next() function can no longer be guaranteed to be incrementing by 1 since it may (very dangerous word) be consumed by other modules.

  2. This does not fit with the current require's algorithm, which is very stable and changing it would have dramatic consequences. In particular, you will need to be sure to tag to a specific version of submodules rather than having possible collisions.

  3. This does not work with how some plugin/nested modules work, where they require('../../..') to interact with a parent module since the parent module has been flattened.

  4. Collisions become much more of an issue since you can have multiple possible matches in a node_modules directory with different versions. Imagine require('project_assets/api.js'). Having this resolve to multiple things can introduce much complexity to a simple system.

@mstade
mstade commented May 4, 2014
  1. will increment by one regardless, but you couldn't make the assumption that next won't be called from another module that shares the same dependency. Arguably, this code is not particularly good, and the current algorithm enables it, but I realize that's not a particularly good argument for flattening the dependencies – breaking existing code never helped anyone.

  2. is again bad code making too many assumptions, just as 1). Of course, the current algorithm enables it, so that's unfortunate. Again, I realize pointing fingers at bad code and suggesting they suit themselves isn't a viable option, just stating some obvious things.

I'll assume 2) is a continuation of 1) and you're absolute right – if this is a widespread pattern then it would have pretty dramatic consequences. I didn't know (and still don't, really) this was a common pattern. I don't use and haven't seen either 1) or 2) before; please forgive my ignorance.

I think 4) has a straight forward solution in using the package.json metadata.

The examples you bring up with code making too many assumptions (enabled by the current algorithm) are a good argument against flattening, since it would indeed introduce some serious complexity. My gut tells me this is a problem that can be overcome, but off hand I have no reasonable suggestions.

Thanks for the insights @bmeck, I learned something.

@CaleyD CaleyD pushed a commit to CaleyD/rcfinder that referenced this issue May 6, 2014
Caley Allow inclusion within project on Windows platform. Nesting depth of …
…node_modules with lodash.clonedeep and its dependencies create file paths that are too long for some file operations on the Windows platform.

See nodejs/node-v0.x-archive#6960 for details.
966b07e
@briandipalma

As an idea: node_modules/module@version

I like this idea. It would be nice, if such a large change were ever to occur, to also rename node_modules to packages, it would nicely compliment package.json and remove the word node which makes certain developers believe npm is a tool for node development and nothing else.

@trevnorris

I might be stating the obvious, but isn't this a discussion to be had on https://github.com/npm/npm ?

@Mithgol
Mithgol commented May 8, 2014

Implementing the above discussed changes would require some Node.js changes outside of npm as well; for example, require. (Might also affect node-gyp, I dunno. My comment #6960 (comment) is still unanswered.)

@dariuszp

Example: if somemodulename@1.2.3 and somemodulename@4.5.6 are both present, what should npm update somemodulename do?

Nothing. My exdi module as example:

npm install exdi -> install latest exdi in "exdi" directory, npm update will install "exdi"
npm install exdi@1.1 -> install exdi 1.1 in "exdi@1.1" directory and leave it alone

require is a problem but we could follow exact same convention so if You require "exdi" it will look for "exdi" and then for "exdi@*" and if we require "exdi@1.1" it will look for "exdi@1.1" directory

If there is a package.json with "exdi: 1.1" then require('exdi') will always require version 1.1.

In the same example, what should npm uninstall somemodulename do?

Again, same thing as above. This way we can have multiple versions of one package without need to have nested structure. When parent directory is node_modules, require could look into current directory for deps and not into subdirectory "node_modules".

@khaledh
khaledh commented May 18, 2014

I'm new to node development on Windows, and this is basically a showstopper. The lodash packages are particularly problematic, and cause some installs to hang silently. The problem is exacerbated when trying to clean up the failed installation manually, which doesn't work because of the long path problem.

This is, IMO, a very bad experience for people starting to develop with node on Windows, which most likely will drive them away. If the node/npm teams think this is a problem in Windows itself, and they're not willing to make it work, fine, but please don't provide Windows installers and don't claim Windows support.

@Mithgol
Mithgol commented May 18, 2014

This all-or-nothing approach is no good. Of course it is essential to continue providing Windows installers for Node.js.

@bmeck
bmeck commented May 18, 2014

@khaledh node/npm itself should work fine, what part of the install hangs and can you provide and logs npm-debug or other around this. The only thing which should fail is other windows programs which do not support long paths, such as cmd.exe

@sindresorhus sindresorhus referenced this issue in gruntjs/grunt-contrib-imagemin May 25, 2014
Closed

Long directory causes issues in Windows #221

@hyrmedia

I develop on Windows, and this is a huge problem for me right now.

mstade said:

I doubt that it's impossible to come up with a new algorithm that works with a flat folder structure. The current algorithm is simple and intuitive, but the assumption that deep folder nesting is A-OK doesn't hold up on a major OS, so it might be worth spending a few brain cells figuring out a more fitting solution.

I don't think this is rocket surgery; but I also don't doubt it'd be a fair bit of work to implement, particularly in making sure existing projects don't get shafted.

I couldn't agree more. I can understand concerns about taking the time to design & implement a solution, but from both an architecture and a usability perspective, this seems like it should be a top priority. For the end-developer, maintaining endlessly-nested levels of "node_modules" directories is vastly more difficult to support and maintain, on Windows or otherwise, than a simple flat list of "module@version" entries -- but the blatant brokenness of the situation on Windows is beyond frustrating.

Btw -- I know there are several issues to address, but specifically regarding the question about require() and which module version it should actually be requiring -- isn't that easily determined from the version referenced in each module's own packages.json manifest?

@edef1c
edef1c commented May 30, 2014

@hyrmedia
The only field in package.json that node looks at is main. node's require algorithm is entirely unaware of module versions. it's also a frozen part of node core.
As far as I can tell, nobody is particularly enthusiastic about changing the behaviour of npm or node entirely for Windows alone — a platform few of us use, even fewer of us like, and only a couple of us care about. node has done the correct thing by handling long paths correctly, and at this point it feels like we're being asked to compensate for all the software around us on Windows being crap.

@isochronous

Could you please not use phrases like "a platform few of us use" ? The last two jobs I've had have been exclusively windows shops, and both of them have used node extensively. Don't make generalizations based only on your personal experience. I do like windows, and care a lot about node compatibility on windows, and I know many others aside from the multiple people represented in this thread that do too. I'm not sure exactly what you mean by "node has done the correct thing by handling long paths correctly," other than the fact that node supports long paths, which is not so much a node feature as it is a file system feature of the OS you're on. Since windows continues to hold roughly 90% of the OS market, it seems like increasing node's compatibility with the platform can only benefit node in the long run. In addition, I feel like the module@version approach would have at least two major benefits on any OS compared to the current approach - eliminating duplication and reducing complexity.

@notac
notac commented May 30, 2014

The "Software around us on Windows" that rightly or not that are referred to as crap are core utilities. Developers on Windows would have to do a lot of extra work to avoid getting jammed up on this.

Yes, in the ideal world we wouldn't have to deal with this. But developers work in an environment that is dirty and messy, and they just need to get stuff done. Refusing to recognize reality is not a viable position. We are being asked to compensate for the good of the project, and we should.

I don't use windows, but I know a lot of developers who do, and who would like to use node. Please, let's just fix this (or compensate/whatever). Unless we really just want to cut out what is still a majority of the potential market for node.

@trevnorris

I'm still missing why this discussion is happening here and not on npm's
issue tracker.

@isochronous

I think it's a valid discussion for either location. Node has to understand
how to locate dependencies, while npm needs to understand how & where to
install them.

On Fri, May 30, 2014 at 4:14 PM, Trevor Norris notifications@github.com
wrote:

I'm still missing why this discussion is happening here and not on npm's
issue tracker.


Reply to this email directly or view it on GitHub
#6960 (comment).

@bmeck
bmeck commented May 31, 2014

Another aside on why this may be more pertinent on npm is that flat file structures can emulate nested structures using symlinks (available in Windows Vista and up). Basic concept can be shown by running: https://github.com/bmeck/flatter to see the output (not finished but concept is sound). I will fix it when I get back from JSConf.

Basically:

root/node_modules/a
root/node_modules/a/node_modules/b
root/node_modules/a/node_modules/c
root/node_modules/a/node_modules/c/node_modules/d

becomes:

root/node_modules/f~0 # previously a
root/node_modules/f~1 # previously b
root/node_modules/f~2 # previously c
root/node_modules/f~3 # previously d

# root modules need to link to their flattened counterpart
root/node_modules/a -> root/node_modules/f~0

# flattened modules need to provide their module scope via symlink

# a's scope
root/node_modules/f~0/node_modules/a -> root/node_modules/f~0
root/node_modules/f~0/node_modules/b -> root/node_modules/f~1
root/node_modules/f~0/node_modules/c -> root/node_modules/f~2

# d's scope
root/node_modules/f~3/node_modules/a -> root/node_modules/f~0
root/node_modules/f~3/node_modules/b -> root/node_modules/f~1
root/node_modules/f~3/node_modules/c -> root/node_modules/f~2
root/node_modules/f~3/node_modules/d -> root/node_modules/f~3

npm supporting an install structure such as this can be done without breaking node.

caveats are limited to modules that are not self contained:

  • accessing files outside of your module will cause problems (ie. require(path-to-current-modules+'/../../'))
  • relying on realpath-ing to have the nested directory structure will break.
@notac
notac commented May 31, 2014

I am emphatically against this.

If we're going to modify anything here, it needs to be a clean break from current design. Symlinking to a flat structure is just avoiding addressing the problem, and is introducing complexity we don't need.

A clean break to a directory structure that looks like this is what I have in mind:

node_modules/foo@1.0.0
node_modules/bar@0.0.9
node_modules/foo@1.0.1
node_modules/foo@latest #symlink to 1.0.1
node_modules/bar@latest #symlink to 0.0.9

npm update foo would install the latest version of foo to foo@version, and update the symlink foo@latest.

Tell me I'm crazy.

@isochronous

No offense, but that seems like way too much unnecessary effort, and like it would just lend itself to confusing users.

On May 31, 2014, at 9:25 AM, Bradley Meck notifications@github.com wrote:

Another aside on why this may be more pertinent on npm is that flat file structures can emulate nested structures using symlinks (available in Windows Vista and up). Basic concept can be shown by running: https://github.com/bmeck/flatter to see the output (not finished but concept is sound). I will fix it when I get back from JSConf.

Basically:

root/node_modules/a
root/node_modules/a/node_modules/b
root/node_modules/a/node_modules/c
root/node_modules/a/node_modules/c/node_modules/d
becomes:

root/node_modules/f0 # previously a
root/node_modules/f
1 # previously b
root/node_modules/f2 # previously c
root/node_modules/f
3 # previously c

root modules need to link to their flattened counterpart

root/node_modules/a -> root/node_modules/f~0

flattened modules need to provide their module scope via symlink

a's scope

root/node_modules/f0/node_modules/a -> root/node_modules/f0
root/node_modules/f0/node_modules/b -> root/node_modules/f1
root/node_modules/f0/node_modules/c -> root/node_modules/f2

d's scope

root/node_modules/f3/node_modules/a -> root/node_modules/f0
root/node_modules/f3/node_modules/b -> root/node_modules/f1
root/node_modules/f3/node_modules/c -> root/node_modules/f2
root/node_modules/f3/node_modules/d -> root/node_modules/f3
npm supporting an install structure such as this can be done without breaking node.

caveats are limited to modules that are not self contained:

accessing files outside of your module will cause problems (ie. require(path-to-current-modules+'/../../'))
relying on realpath-ing to have the nested directory structure will break.

Reply to this email directly or view it on GitHub.

@olimortimer

I'm fairly new to Node, so forgive me if this is not at all possible...

Could all modules just reside alongside each other in node_modules/ ? This would do away with infinite levels of directories. You could then have a node_modules.json file which could show which modules depend on others

This would also mean that if multiple modules used the same module, there would only be one instance of it, rather than an instance in each module that requires it.

@briandipalma

You can use npm dedupe if you don't want multiple versions of the same package.

I think the important question to be answered in this thread is are the node and npm team willing to consider a change to how npm installs its packages.

If the answer is a firm "never" then this issue will not be fixed and will never go anywhere.

@dariuszp

We could always go for:

node_modules/my_module/1.0
node_modules/my_module/1.1
node_modules/my_module/current (reference to latest)

We just look into package.json for package version. If not set, current is used. Packages could be either always global or local but at application level. This should not break anything.

Or we could just ignore Windows users and go one with our lives without need to change.

@tlyng
tlyng commented May 31, 2014

I've just encountered this 'bug' also. I'm not developing on Windows, but are looking at setting up a development environment for javascript (amongst others) for a colleague of mine. I installed Windows into Virtualbox and encountered this as soon as I tried to build the generated code from yeoman/generator-angular - I'm glad I'm not using that operative system outside of Virtualbox.

I'm no node expert by any means, but in my point of view the flatten module@version structure with linked nested tree sounds like a possible solution. Won't we hit that 260 char path limit on linked structures?

@bmeck
bmeck commented Jun 1, 2014

Reminder: Windows (the OS and file system) supports long file names. Most applications included with Windows do not. The fix for said applications is to move to the unicode version or currently recommended function of the Windows APIs ( http://msdn.microsoft.com/en-us/library/windows/desktop/aa364232(v=vs.85).aspx & see "?" stuff in http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx and link).

I will try to reiterate that moving to a naive module@version can cause issues when modules expect shared state but are present multiple times at the same version in a dependency tree (uncommon but possible and present in various logging / config modules). You can see a couple of comments from various people above about this.

As for the problem with symlinks vs 260 char limit, this may cause issues depending on how the application consuming the file is set up (once again this is all application bugs not Windows the OS). If a realpath occurs prior to access, it will not be a problem; if access is merely absolute it can cause issues. This does let people do things like cd node_modules/really; cd node_modules/deep; cd node_modules/nesting which would have not worked before, and a number of broken apps should start working (not all, once again they should be using long name file access convention).

However, most (all to my knowledge) of these problems revolve around applications using old or limited APIs on purpose (or in ignorance) on Windows. There are several solutions, but I feel that full flat structure being the norm would require a very convincing reason to alter:

  1. The very stable module system that has build tools and infrastructure around it
    1. Shared state problems akin to dedupe problems are always present
    2. Multiple version pruning would require walking all dependencies
  2. We are working with an OS that supports long paths already... (not talking about applications)
    1. Seriously, this will never be fixed in Windows bundled applications as everyone decides to harm their structures to appease the 260 char bug in other Windows applications. Technical debt is our bff?

I do understand the issue, but a divide is clear where people using applications supporting long paths vs those without (not OS/FS, cause all modern OS/FS support long enough paths). Some agreement needs to be made on both sides; as flat mapping files causes some issues for long path based workflows (disallows various shared state modules), and long path based workflows do not work with outdated/bundled Windows applications.

I would suggest that this come around to npm if we can use flat structures to emulate nested modules as this would not require changing node. If emulating nested structures does not work sufficiently (basic dev tools need to stop complaining [this would most likely not tailor to various specific programs]), people may want to support a secondary module loading convention that would require changes to both npm and node. However, having 2 different ways to load modules is harmful (multiple possibilities == infinite complexity).

@kblcuk
kblcuk commented Jun 1, 2014

Just a side note, this npm issue says that

Node (and thus npm) always uses the UNC paths internally for fs operations

It seems (as noticed here already by others) that the problem described here is problem of some Windows tools rather than Node's/npm's.

@Qard
Member
Qard commented Jun 4, 2014

Why not add a field to each package.json file at install time that specifies a relative path to a node_modules directory and defaults to ./node_modules? Where necessary, you could generate a short uid to move the package back up to the top level.

Given an expected path like this;

app/node_modules/a/node_modules/b/node_modules/c/node_modules/d

On *nix, you can do;

app/package.json (packages_path = './node_modules')
app/node_modules/a/package.json (packages_path = './node_modules')
app/node_modules/a/node_modules/b/package.json (packages_path = './node_modules')
app/node_modules/a/node_modules/b/node_modules/c/package.json (packages_path = './node_modules')
app/node_modules/a/node_modules/b/node_modules/c/node_modules/d/package.json

And on Windows, you can do;

app/package.json (packages_path = './node_modules/{packages}')
app/node_modules/{packages}/a/package.json (packages_path = '../../{a_packages}')
app/node_modules/{a_packages}/b/package.json (packages_path = '../../{b_packages}')
app/node_modules/{b_packages}/c/package.json (packages_path = '../../{c_packages}')
app/node_modules/{c_packages}/d/package.json

Also, I agree with the idea of changing the node_modules folder name to packages. It feels more generic that way, and more unified with the naming convention of package.json. This design would make that easy too, or even customizable by any user.

@briandipalma

I have asked about renaming node_modules to packages and I think that's also a no go. I think looking at JSPM for a browser package manager might be helpful as it does have flat directory structures. Along with inter-op with npm and resource loading support.

@demoneaux demoneaux referenced this issue in gulpjs/gulp Jun 10, 2014
Closed

Filename too long #518

@LoveAndHappiness

I was developing on Windows and am suddenly not able to push on github anymore, because I installed gulp with npm. It is kind of frustrating, because I have to work around and recreate my developing area on ubuntu, which just consumes time. I wish that windows users should be warned before the installation, because Node really isn't supporting Windows right now. And it doesn't matter if this is the fault of the OS, an API or any other application. All that matters is that it isn't working on Windows right now and there doesn't seem to be a reasonable solution to this problem. So get your shit together and make some decisions, because it is very user unfriendl for Windowsuser.

@notac
notac commented Jun 10, 2014

This is ridiculous. This is hindering developer's ability to use node. They
are core windows utilites. Please stop referring to them like they're not -
ignoring reality doesn't change it.

If we don't want to fix it, fine. But let's stop lying to ourselves.

Daniel
On Jun 9, 2014 8:16 PM, "Bradley Meck" notifications@github.com wrote:

To be clear Node/NPM fully support windows, various Windows applications
do not. We cannot say it does not support Windows long paths, only that
other applications that are useful do not fully support Windows long paths.


Reply to this email directly or view it on GitHub
#6960 (comment).

@TooTallNate

See, here's what I don't get:

We're talking about the node_modules directory here. It's machine-created (via the npm client), and for all practical purposes should never be needed to be traversed. I've developed tons of modules and test them on Windows and have quite honestly never needed to dive into that directory because npm provides commands for everything you should ever need to do to that directory, and it supports long paths just fine.

So I clearly must be missing some part of your guys' app development flow because as a module author it's never been a problem for me.

@notac
notac commented Jun 10, 2014

I think an example was just provided of that. Just because it doesn't
disrupt your flow, or break any of your cases doesn't excuse core utilities
being broken. There needs to be at minimum a workaround devised for this.
It's ridiculous to not want to correct this. Unless we really just don't
want to support windows.
On Jun 9, 2014 8:33 PM, "Nathan Rajlich" notifications@github.com wrote:

See, here's what I don't get:

We're talking about the node_modules directory here. It's machine-created
(via the npm client), and for all practical purposes should never be
needed to be traversed. I've developed tons of modules and test them on
Windows and have quite honestly never needed to dive into that directory
because npm provides commands for everything you should ever need to do to
that directory, and it supports long paths just fine.

So I clearly must be missing some part of your guys' app development flow
because as a module author it's never been a problem for me.


Reply to this email directly or view it on GitHub
#6960 (comment).

@notac
notac commented Jun 10, 2014

None of us here (I hope) would deliver a system to a paying customer that
exhibited this behavior and claim that it supported Windows.
On Jun 9, 2014 8:40 PM, "Daniel Taylor" dantaylor08@gmail.com wrote:

I think an example was just provided of that. Just because it doesn't
disrupt your flow, or break any of your cases doesn't excuse core utilities
being broken. There needs to be at minimum a workaround devised for this.
It's ridiculous to not want to correct this. Unless we really just don't
want to support windows.
On Jun 9, 2014 8:33 PM, "Nathan Rajlich" notifications@github.com wrote:

See, here's what I don't get:

We're talking about the node_modules directory here. It's
machine-created (via the npm client), and for all practical purposes
should never be needed to be traversed. I've developed tons of modules
and test them on Windows and have quite honestly never needed to dive into
that directory because npm provides commands for everything you should ever
need to do to that directory, and it supports long paths just fine.

So I clearly must be missing some part of your guys' app development flow
because as a module author it's never been a problem for me.


Reply to this email directly or view it on GitHub
#6960 (comment).

@Mithgol
Mithgol commented Jun 10, 2014

@TooTallNate Try npm install mocha --dev on Windows. Its weird behaviour was once a reason behind my pull request mapbox/node-sqlite3#171, and I have always thought (ever since) that JavaScript modules are fine, but only unless they contain C/C++ addons that are harder to build in a deep directory.

@olimortimer

@TooTallNate it isn't necessarily down to traversing the directories, I found this issue when working with NPM and Git - (from memory) I commited node_modules into Git on my Ubuntu server, but when I pulled it down to my Windows dev environment, it couldn't create a directory due to how long the path was.

The Grunt plugin in question was grunt-contrib-imagemin, which had created the following path;

/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/imagemin-gifsicle/node_modules/gifsicle/node_modules/bin-wrapper/node_modules/download/node_modules/decompress/node_modules/adm-zip/test/assets/attributes_test
@LoveAndHappiness

I dislike that the problem seems to exist since January and that there isn't yet any solution to this, but only tricks that circumvent or postpone the problem.
It would be nice, if at least someone took a poll to find out, if the community is going to address this problem and if yes, to what priority.

Windows isn't goint to change their max char for filepaths. And the community has to decide, if they want to have Windwos Users or not.

If yes, npm has to change the way it creates directories. If not at least don't say that node is Windows compatibel, because it is not. Of course some will say that it "works", but what is it worth, if I can't push my work to my favourite collaboration plattform, because Windows doesn't let me touch those files?

Please consider changing the way npm works, so that everybody can benefit from Node and npm, and not only Non-Windows Users. (And yes, I dislke Windows a lot for this, but changing the development environment for a plugin, that is installed via npm, seems like a total overkill and it makes me to not want to implement this technology in my current projects.)

@dougwilson

you should be sure to enable core.longpaths in your git config to tell it to use long paths

note you need git 1.9+

@darsain
darsain commented Jun 10, 2014

Got hit by this when gyp failed to compile one of the native modules due to long paths.

Even if you don't care about windows, the endless abyss of node modules nesting always baffled me. It's just so inefficient, wasteful and detrimental to load speeds...

Here is my 2¢ on how I think npm install should work:

  1. Install every package to root node_modules by default

  2. When 2 semver incompatible versions of the same module are required, install the latest one to the root, and the lower one to the topmost node_modules without a conflict.

    You can look at it as playing Tetris with node modules where the tree grows up, and the highest versions should fall to the bottom.

This is simple, greatly increases the efficiency, eliminates 99% of nesting hell, and you don't have to touch require() at all. It'd also make npm dedupe* unneeded and obsolete.

If there would still be a module with big nesting problem (quite unlikely, but possible), it could be solved by module's maintainer updating its dependencies.


* Doing npm_dedupe would not work in my case, as there is nothing to dedupe because native module compilation failed and took the whole module with it. I'd have to find out which module caused it, and install it manually to the root, hoping the dependency abyss isn't too deep and it'll be enough. What fun.

@briandipalma

I have created an npm issue if you want to comment on this npm/npm#5458

@JeroMiya

I would like to throw out a suggestion that hasn't been stated yet. Use the module_name@semver pattern. On top of this, implement a virtual file system in node that allows virtual paths to be mapped to physical paths. Then, just as node already internally redirects file system calls to use UNC paths, file system calls from anywhere to a path within a virtual file system (e.g. a very long file path under node_modules/) would in turn be redirected to the corresponding physical directory, which can now be in a flat structure.

The VFS could also map multiple virtual paths to a single physical path, so that for example multiple copies of the same package in the current deep node_modules tree could be mapped to a single directory in the flat node_modules/module_name@semver physical directory. To make things simpler for npm, the VFS api could allow npm to map virtual files to physical files on the fly. So, npm would create a VFS for all paths under 'node_modules/' and then register file system overrides (to map virtual paths to physical paths and other operations like listing files/directories in a virtual directory). Node would also need to provide the capability to explicitly opt-out of VFS mapping, so that NPM could still manipulate the physical directories under node_modules.

This would solve the path-too-long problem on windows (paths can now be as large as a JS string can contain), eliminate duplication of packages for everyone (always only one copy of a library per version), but would not break existing packages or change any existing external node or npm behavior (e.g. npm update would work the same as before).

Additionally, by implementing the change with a virtual file system, you would limit the changes that would need to be made to fix this windows path issue to just 1) implementation of the VFS functionality, and 2) changes to npm to construct and maintain a VFS for paths under node_modules/. Anything else that uses the file system primitives through node would not even need to be aware of the VFS functionality.

@isaacs
isaacs commented Jun 10, 2014

Fyi, if you want to remove very long paths in Windows, you can npm i -g rimraf and then use the rimraf command line bin it creates. Works just like rm -rf on Unix.

@isaacs
isaacs commented Jun 10, 2014

@darsain Basically, we need to dedupe up-front at install time. Working on it. It's not trivial.

@TooTallNate

@isaacs I think @darsain's proposal was dedupe up-front as well as installing dependencies of dependencies "de-duped" (installed at the top level, so long as semver of the dependencies allows for it).

It would lose the portability of being able to move around a node module directory and being confident that its dependencies were moved along with it, but it sounds like it would make Windows dev life easier.

@bmeck
bmeck commented Jun 11, 2014

@isaacs does that include using a tmp dir with small pathname for running the install scripts per module? so gyp does not fail in long paths even when the nesting is a problem.

@teltploek

@nathan7 said:

As far as I can tell, nobody is particularly enthusiastic about changing the behaviour of npm or node entirely for Windows alone — a platform few of us use, even fewer of us like, and only a couple of us care about. node has done the correct thing by handling long paths correctly, and at this point it feels like we're being asked to compensate for all the software around us on Windows being crap.

Who are these people that aren't "particularly enthusiastic about changing the behavior of npm or node"? I were a bit chocked when I read your reply, and was wondering if this is solely your own personal opinion, or if its an opinion shared amongst the rest of the npm-team. If it's the latter I must say that my otherwise strong enthusiasm towards the npm as a project has taken somewhat of a hit.

@janniknilsson

@nathan7
I can only support what teltploek said. In my 15 years developing websites, I've only worked in companies that developed on Windows platforms, and I don't think that is such a rare occurrence.

Speaking from experience, the major web development and communication agencies in my country, use Windows machines to work on.

Be careful of making sweeping generalizations based, it seems, solely on your own preferences and workplace experiences.

That aside, since it seems that there is a real problem with too long path names, wouldn't it be more productive looking at a solution to the problem rather than just saying that it works as intended.

@llambda
llambda commented Jun 12, 2014

The way NPM handles dependencies is brilliant and a hallmark feature of npm.

If you move all modules to top level node_modules it's too hard to examine the node_modules used by a particular module. Making it more difficult to read/edit/experiment with submodules which is common for developers to do. I would be much less likely to read/edit source for submodules if they were all in one giant directory. It's very common to look in node_modules and see what is in there. This kind of tinkering is critical to the npm ecosystem.

@othiym23

@teltploek @janniknilsson As a developer at npm, Inc., I want npm to work well for you regardless of the platform you're developing on. Windows is a supported platform for Node and npm, and I believe that Node has been a success story on and for Windows. I want to see this continue, and if there are bugs and pain points in npm for Windows, I want to see them ironed out.

At the same time, the way npm and node themselves interact with Windows paths is correct, and was originally designed with input from MS developers. The problem is not with node or npm but with the tooling around them. The problem is painful enough that it's worth having this discussion, but the basic way that node's module system works is not a cowpath so much as an 18-lane paved freeway. Maybe more aggressive flattening a la npmd or npm dedupe will do the job, but moving entirely to a flat directory structure (or doing anything that changes how node's module loader looks up dependencies) is painful enough to be a nonstarter.

And, to a certain extent @nathan7 is right – I think @domenic is the only npm contributor who uses Windows as his development environment, and with the significant exception of people using Azure Web Server, the footprint of Windows as a production deployment environment for Node is minimal. I say this not to diminish or attempt to invalidate your frustrations, but to point out that any changes made to npm or Node to deal with environmental issues on Windows could cause pain for a much larger number of people than those currently affected.

@ArtemGovorov

@othiym23 said:

the footprint of Windows as a production deployment environment for Node is minimal

Depends on the definition of the "production deployment environment". Node is not only used as a web app platform: there are command line tools, nodewebkit applications, etc. My guess is that there are significantly more installations of grunt, gulp, less and other utilities written in node.js than production deployed websites written in node.js. Probably even the number of installations of grunt.js alone on windows is higher than the total number of deployed node.js web apps. And for a web developer using grunt for his front-end work, node.js is just a part of the grunt "product" and the fact that the product (or one of its plugins) doesn't work with his established workflow on windows is sad to say at least.

@LoveAndHappiness

Thank you for understanding and acknowledging our pain. I support your
suggestion to aggressively flatten the directory structure a la npmd or npm
depude into the native npm installment, so that we could at least postpone
the problem until later.

If other people think, that this makes sense for the beginning, then let's
implement it to the next update.

Horyzon - Online Marketing

+49 (0)178 714 2007
georg@horyzon.de
http://s.wisestamp.com/links?url=mailto%3Ageorg%40horyzon.de

www.horyzon.de
http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.horyzon.de

[image: Google Plus Page]
http://s.wisestamp.com/links?url=https%3A%2F%2Fplus.google.com%2Fu%2F0%2Fb%2F113367440655979214836%2F113367440655979214836%2Fposts
[image:
Facebook]
http://s.wisestamp.com/links?url=https%3A%2F%2Fwww.facebook.com%2Fgeorg.geladaris
[image:
YouTube]
http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.youtube.com%2Fchannel%2FUCO_PPRfmqqWsBqAiELkkRNQ
[image:
Twitter]
http://s.wisestamp.com/links?url=https%3A%2F%2Ftwitter.com%2FGeorgGeladaris

2014-06-13 4:13 GMT+02:00 Forrest L Norvell notifications@github.com:

@teltploek https://github.com/teltploek @janniknilsson
https://github.com/janniknilsson As a developer at npm, Inc., I want
npm to work well for you regardless of the platform you're developing on.
Windows is a supported platform for Node and npm, and I believe that
Node has been a success story on and for Windows. I want to see this
continue, and if there are bugs and pain points in npm for Windows, I want
to see them ironed out.

At the same time, the way npm and node themselves interact with Windows
paths is correct, and was originally designed with input from MS
developers. The problem is not with node or npm but with the tooling
around them. The problem is painful enough that it's worth having this
discussion, but the basic way that node's module system works is not a
cowpath so much as an 18-lane paved freeway. Maybe more aggressive
flattening a la npmd or npm dedupe will do the job, but moving entirely
to a flat directory structure (or doing anything that changes how node's
module loader looks up dependencies) is painful enough to be a nonstarter.

And, to a certain extent @nathan7 https://github.com/nathan7 is right –
I think @domenic https://github.com/domenic is the only npm contributor
who uses Windows as his development environment, and with the significant
exception of people using Azure Web Server, the footprint of Windows as a
production deployment environment for Node is minimal. I say this not to
diminish or attempt to invalidate your frustrations, but to point out that
any changes made to npm or Node to deal with environmental issues on
Windows could cause pain for a much larger number of people than those
currently affected.


Reply to this email directly or view it on GitHub
#6960 (comment).

@teltploek

@ArtemGovorov said:

Depends on the definition of the "production deployment environment". Node is not only used as a web app platform: there are command line tools, nodewebkit applications, etc. My guess is that there are significantly more installations of grunt, gulp, less and other utilities written in node.js than production deployed websites written in node.js. Probably even the number of installations of grunt.js alone on windows is higher than the total number of deployed node.js web apps. And for a web developer using grunt for his front-end work, node.js is just a part of the grunt "product" and the fact that the product (or one of its plugins) doesn't work with his established workflow on windows is sad to say at least.

I couldn't agree more. Although I was happy to read a more nuanced reply from @othiym23, I still feel that you're, somewhere in there, neglecting the fact that node and the utilities at hand are used in lot of different setups. We, for instance, have a Windows continuous integration server that, among other things, handles and processes our frontend assets using gulp etc., in what has become a somewhat ambitious setup. The long path problem is very much an issue in this setup, although I understand that it isn't directly a problem with node or npm.

@darsain
darsain commented Jun 13, 2014

@othiym23 as mentioned above, node is not just for web servers. I've personally written more node based desktop apps than I've written servers. And they need to run on windows. The cross-platform nature and the ease of development on node is the reason I chose it for this task in the first place.

And even if it would be only for servers, as I said above, if you don't care about windows, than look at the flattening by default approach described in my post as: Making your node apps more efficient by removing the dependency abyss and countless module duplicates. That in turn results into a smaller memory footprint, and way less fs ios which drastically speeds up loading times.

The inefficient way how node handles dependencies always bother me more than occasional nw-gyp errors anyway. And if fixing that also helps with long paths? Than one more fly swatted by a single stroke, and everyone benefits :)


Off topic
The other issue I have with npm is that publish only needed files is not enforced, just suggested by the existence of .npmignore and package.json:files property, neither of which is really used properly in the wild.

Building apps for desktop than means going through dependencies and removing bloat, which is in many cases 80% of the node_modules directory, if not more. And inlining the whole readme file in package.json by npm also doesn't help.

Just imagine npm install downloading 10MB of 100% needed code instead of 100MB of mostly bloat. Oh the wishes of the lieges.

Wanted to write a module for cleaning all that bloat out, but never got, nor looking forward to it :/

The way how dependencies work right now is great when compared to weird or none at all solutions of other environments, but I'd definitely not call it brilliant and a hallmark feature of npm (quoting @llambda above). At least not yet :)

@Qard
Member
Qard commented Jun 13, 2014

It's a bit telling that, of easily 50k+ node developers, less than 10 are actively complaining about this. The other 99% must either not be effected or not care.

That's not to say it's not worth thinking of a fix. It absolutely is. Just that the vocal minority should acknowledge that they are in fact the minority here, and will need to respect that any fix made would need to have no negative impact on the experience of the majority users.

If you think it's so important, submit a pull request, or suggest alternatives that don't require breaking changes. Just understand there is more to it than simply adding such functionality. It also needs to be maintained, and both node and npm are sorely lacking in maintainers that use Windows.

Don't be offended that all these suggestions are getting turned down or start making accusations that the devs don't care about Windows. There's only a handful of maintainers and they have limited bandwidth.

@nicjansma

It's a bit telling that, of easily 50k+ node developers, less than 10 are actively complaining about this. The other 99% must either not be effected or not care

Or not paying attention to this thread, or complaining elsewhere, or are shy, or affected slightly but not enough to complain, or don't have time, or are watching, or are hoping someone else can fix it, or...

My experience with my own consumer products, is that only an extremely small minority complain about issues that affect them. The rest either don't (for the reasons above), or stop using the product. My assumption is that when a single user is complaining about an issue affecting them, there are likely 100 others that are also affected but not complaining.

The fact that there are so many folks adding on to this enormously long thread is likely an indicator that there are at least thousands of devs affected by this.

@nicjansma

For what it's worth, I would love to contribute my time to helping fix this problem on Windows. From what I've read on this thread, there doesn't seem to be a consensus (from the wild or core Node developers) on the solution.

If this is a Windows-dev manpower thing, I'd be happy to donate my time, as I'm sure many others on the thread would be.

It's coming to an agreeable, long-term solution first that appears to be the challenge.

@robmuh
robmuh commented Jun 13, 2014

The base architecture of node_modules is ill-conceived to the point of rendering node unsustainable and useless without a serious architectural change. Downloading 30 lo_dash modules mostly with the same version level is not a sustainable direction even if it seems to be the only way to ensure the exact version in any npm bundle is used.

The core flaw in the default architecture — while it can be worked around — is enough to almost ensure that the current version of node will not receive wide-spread enterprise-level use for large applications for some time. Walmart and the gang have done amazing things working around this, but I get the impression it is because they've developed their own plugin architecture hapi instead of using the default node_modules and npm architectures.

I observed architectures come and go for the last 20 years (as Nike's Intranet webmaster and with IBM). I too want to help with this, but in our small private school I've gone back to emphasizing Python, Java, and C# for students focused on large applications as much as I love Node and want to see it succeed. There is no doubt it is the future, just not in it's current form.

@matt-bernhardt

While I enjoy hearing everyone's perspectives about the best way to handle package dependencies in abstract terms, I find myself trying to piece together a repeatable problem report around long paths in Windows. Does anyone have such a set of steps?

Without such a replicable situation, it seems that we are unable to move past conflicting apocryphal claims of functionality/brokenness made from personal experience - which I fear will doom this issue to an endless back-and-forth.

After an hour or two this morning, this is the closest thing I can come up with:

  1. Starting from a Node folder in my Documents
    The base path is c:\Users\username\Documents\Node

  2. install grunt-contrib-imagemin
    This then calls in nine levels of dependencies, which install correctly:

grunt-contrib-imagemin
1- imagemin
1-2- imagemin-gifsicle
1-2-3- gifsicle
1-2-3-4-bin-wrapper
1-2-3-4-5- download
1-2-3-4-5-6- decompress
1-2-3-4-5-6-7- tar
1-2-3-4-5-6-7-8- fstream
1-2-3-4-5-6-7-8-9- graceful-fs

  1. I'm able to browse to the bottom of this directory via the Windows Explorer UI (Windows 7)
    Toward the bottom the path switches to DOS-style names that are limited to eight characters. The full path is:

C:\Users\mjbernha\DOCUME1\Node\NODE_M1\GRUNT-3\NODE_M1\imagemin\NODE_M1\IMAGEM4\NODE_M1\gifsicle\NODE_M1\BIN-WR1\NODE_M1\download\NODE_M1\DECOMP1\NODE_M1\tar\NODE_M1\fstream\node_modules\graceful-fs

  1. However, when I try to traverse down to the bottom using a command prompt, I am unable to get past fstream - the path at that point is 252 characters long. Switching to DOS-style directory names can get me to the bottom, however.

Caveat:
I haven't tried to actually use anything in this setup yet - that's next, but I need to set this aside for a bit. My hope is that one of the other contributors to this thread can list out a chain of steps to break something under Windows more significantly than I outline above. I'm very much a beginner with Node, trying to learn as much as I can - but I'm happy to contribute what I can.

@randunel

Since we all agree that this would take some serious dev work and issue tackling, a decision must be taken by the project's maintainers regarding the changes, if any.
Once a decision is taken, I'm sure that the affected non-maintainers would be happy to contribute. Starting to work on this without a prior decision would be under a rejection risk, for bad architecture/design/implementation.
Just make up your mind, make a decision and mark it as low priority. When it's done, all you need to do is review the code.

@robmuh
robmuh commented Jun 13, 2014

Has there been any discussion of a node_modules local (and I hate the word) registry? Perhaps just something like packages.json? npm and company could use it to keep the n_m level to 3 tiers.

I think it is important to emphasis that this is not just a Windows problem. It addresses issues that the npm itself has already shown with the hundreds of redundant downloads it has to make.

@Mithgol
Mithgol commented Jun 13, 2014

@matt-bernhardt

I find myself trying to piece together a repeatable problem report around long paths in Windows. Does anyone have such a set of steps?

One step seems to be enough:

  1. npm install mocha --dev
@isochronous

Or just try deleting the root node_modules directory via DOS or windows explorer. Or moving it. Or trying to create a git repo with it.

On Jun 13, 2014, at 11:28 AM, Mithgol notifications@github.com wrote:

@matt-bernhardt

I find myself trying to piece together a repeatable problem report around long paths in Windows. Does anyone have such a set of steps?

Actually one step is enough:

npm install mocha --dev

Reply to this email directly or view it on GitHub.

@droppedoncaprica

@robmuh If you read above, @isaacs is working on default deduping on install, which should help the Windows depth issue.

@briandipalma

@robmuh If you read above, @isaacs is working on default deduping on install, which should help the Windows depth issue.

As long as you have dependencies that can be deduped, otherwise the issue remains.
Default deduping is great but from my understanding of the issue it's not a permanent fix.

@droppedoncaprica

It's not a perfect fix for all use-cases for Windows users, no. But it is a start that would help a lot of Windows users, if I understand the issue correctly.

@briandipalma

It's not a perfect fix for all use-cases for Windows users, no. But it is a start that would help a lot of Windows users, if I understand the issue correctly.

It's not a complete fix for the bug though and without one I think I'd prefer http://jspm.io/ over npm for a Windows developed enterprise web app. npm can still provide all of it's benefits without having a deep folder structure. It's just such a huge change that I can see it being an impossible ask. Maybe a flat structure can be used for the new namespaced packages?

@othiym23

Maybe a flat structure can be used for the new namespaced packages?

The only difference between scoped packages and global packages is in how the npm client resolves them back to a registry. There's no overlap of concerns between how they're implemented and this issue.

@edef1c
edef1c commented Jun 14, 2014

@teltploek
Perhaps I should've been more clear. You're free to consider my statements sweeping generalisations. I'm not the boss of anything.
Modifying node's module resolution algorithm is definitely out for reasons I've enumerated before.
But upfront dedupe of npm dep trees is something I've wanted for a long time, and not something I consider a change in npm's semantics.
For me, building native modules in a temporary directory with a fixed, short path is also pretty high up on the list too, even if it is purely for the sake of Windows.
There are things we could do for tree-flattening with symlinks and such, and I think those are cool, though I'd rather not deal with them on my systems for the sake of Windows. (symlinks in windows are also in a position rather similar to that of long paths, which is a little worrying. I think they're a pretty correct fix for long paths though, and it eliminates the practical problem)
On a side note, there is a sweeping generalisation being made about who uses node: apparently only web developers? I've served HTML from node maybe twice at most.

@binkydog

I switch between Linux and Windows development and I too was hit by this issue today on my Windows machine (try “yo angular” from the Windows command prompt and it will crap out on long paths…) While I can say “oh well, back to Linux…” I’m sure there are plenty of Windows developers that would prefer not to or may not be able to given their work environments.

Regardless if one thinks it’s a Node/npm issue or a Windows issue, it either needs to be addressed by the node community OR Node/npm support for Windows should be dropped altogether. Fish or cut bait! Personally, I think the latter would be unfortunate and short-sighted, but I kind of get the “let’s not take this on for a small minority” argument. Now that Microsoft seems to be getting on the Node bandwagon perhaps some of their engineers may be willing to pitch in… (?)

@Mithgol
Mithgol commented Jun 15, 2014

I’m sure there are plenty of Windows developers that would prefer not to or may not be able to given their work environments.

I am one of these developers and I find that “all or nothing” approach (i.e. “either needs to be addressed by the node community OR Node/npm support for Windows should be dropped altogether”) disturbing.

If the only alternative to the current (not ideal) behaviour is the total lack of Windows support, then let the current behaviour go on.

@isochronous

Agreed - working with semi-broken tooling is far preferable to not working at all. Of course, both of those options are inferior to fully working, and adoption is going to be affected on Windows until the tooling problems are fixed.

-----Original Message-----
From: "Mithgol" notifications@github.com
Sent: ‎6/‎15/‎2014 2:21 PM
To: "joyent/node" node@noreply.github.com
Cc: "Jeremy McLeod" isochronous@gmail.com
Subject: Re: [node] Node's nested node_modules approach is basicallyincompatible with Windows (#6960)

I’m sure there are plenty of Windows developers that would prefer not to or may not be able to given their work environments.
I am one of these developers and I find that “all or nothing” approach (i.e. “either needs to be addressed by the node community OR Node/npm support for Windows should be dropped altogether”) disturbing.
If the only alternative to the current (not ideal) behavour is the total lack of Windows support, then let the current behaviour go on.

Reply to this email directly or view it on GitHub.

@binkydog

Fair enough. Personally, I'd rather not waste my time with inadequate support, but I get where you're coming from. Bottom line, adoption WILL be affected on Windows without a fix, especially for Windows devs attempting to use it without much forewarning of the issues - or straight-forward workarounds. Maybe the "99%" don't warrant it fix-worthy, but if that's the case, it should at least be made clear up front what the limitations are on Windows.

@dsschnau

+1 We're trying to use node at my company and this is basically a dealbreaker.

@tlyng
tlyng commented Jun 17, 2014

OK, I see no progress on this issue so I just basically made up my mind and say that node is not compatible with Windows. It's probably possible to make it work somehow with some hacks here and there, but then default settings on it's package management tool is clearly broken - I have to concider the entire environment broken. Luckily I don't use Windows daily, but now I won't try to have other colleagues work with node applications on Windows - it will only give me more pain.

Until this issue is fixed I suggest removing the official "windows supported" status, documentation to many many third party packages (such as yeoman f.ex) is clearly broken. You may be able to run something under Windows, but this is clearly not 'cross platform'. Removing the official windows supported status would reflect the truth a bit more than it is today.

@OrangeDog

You don't need hacks, you just need to use tools that are good. Nobody would ever dream of doing serious development in Notepad and all their build scripts are only .bat files. Likewise if default Windows tools don't support long paths, then replace them with those that do.

Only Explorer, CMD and git have been mentioned (that I recall). The latter you just need to turn on long path support (though why you're checking in managed dependencies I don't know), and the others have easy to find replacements for a serious dev environment. Alternatives to npm with different structures have also been mentioned.

@tlyng
tlyng commented Jun 17, 2014

I've tried different terminals (powershell, cygwin, console2 and cmd) none of them work correctly. As of tooling I don't use any additional - only npm, bower, grunt and similar. Which texteditors people use may vary, I suppose most of them support long paths (sublime, eclipse and similar), but I'm still unable to run 'yo angular myapp' to generate boilerplate code. WebStorm probably support creating this boilerplate code, haven't tried - and won't as it's not a free alternative.

@jhibbard-et

same problem - large path depth in windows just makes everything go boom.

@llambda
llambda commented Jun 17, 2014

So is it true that node, npm and git handle long file names just fine; it's the other programs that do not?

@jhibbard-et

they handle them i believe. windows just gets as confused as a schizophrenic

@maxgrass

https://nodejstools.codeplex.com/, a great upcoming tool (~20k downloads) for nodejs web-developers on windows is vivid example how in some way imperfect architecture of product A could imcact the products, based on the product A. I'm sure folks @Mircosoft will try to do their best to check whether they use UNCs and why everything gets broken after installing gulp, but this points in the direction of changes needed to be done, until that moment this powerful tool is rendered useless for Node.js+Npm development on windows as well.

@binkydog

@OrangeDog - you say "use better tools" but where do these magical tools exist? I'm talking about a console app that supports long file paths in Windows. I mean Microsoft hasn't even been able to get this going with the Windows cmd or powershell (nor does it appear anyone else has for that matter). Are you aware of such tool? Is anyone? (I'm not being facetious - really wondering...)

@dz0ny
dz0ny commented Jun 19, 2014

The story of WNPM tool. Seriously if you need to "fix" the bug just fork npm(and nodejs require subsystem) name it however you like it and implement flatten modules. Bug is not about WINOS or nodejs, they both support long path names. It's about default tools shipped with Windows that don't support long path names, namely Notepad, cmd, Explorer, powershell...

@davedx
davedx commented Jun 19, 2014

So, another Windows dev here.

npm install mocha --dev is pretty hilarious when you run it on Windows.

Trying to clean up the mess afterwards is also a barrel of laughs:

$ rm -rf test
rm: cannot remove `test/node_modules/mocha/node_modules/debug/node_modules/browserify/node_modules/backbone/node_modules/phantomjs/node_modu
les/unzip/node_modules/pullstream/node_modules/stream-buffers/node_modules/vows/lib/vows/coverage/fragments/coverage-foot.html': File or pat
h name too long

Having said that: I use Windows 8 most of the time at work, and usually (running in the git bash shell) I don't run into path issues like this. I think in the cases where I have, I have usually decided to just not use that node module. Not reinventing the wheel is something of a sacred cow in software engineering, but sometimes pragmatism wins out.

The test runner we use is this tiny thing that I wrote in about 3 hours one day: https://github.com/davedx/drone

For the reasons why, see above. Why does a test runner need to compile native stuff? Because dependencies? Phantomjs's approach is better; download a damn tarball.

I can live with how this all works on Windows, personally. Of course, things go smoother on my Mac.

@jeremymurray

This path limit affects so many things, including MSBuild for Visual Studio. I recently had to create a zip container to hold build results because the built-in copy command for VS projects hit the limit even though the compiler and linker were fine to produce the deep files in the first place.

Unless a flatter directory structure is enforced, we're pretty much guaranteed to get inconsistent failures with the current recursive dependency tree setup. That wouldn't be node's fault, but could come from any tool used in a module that calls into the win32 APIs. This includes several common shell binaries like copy and common info operations like finding the last modified time of a file.

Unfortunately, the stateful nature of some node modules needs more than just a module@version structure. I don't have a good solution for that.

Long discussion of .NET, the path limits, and why it isn't an easy fix for MS either is here: http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx

@matejkramny

Perhaps its not a node bug, but a windows bug.

@brunolanevik

Just use vagrant when developing anything that does not require a window environment.

@matejkramny

Or dont use windows at all. problem solved.

Matej Kramny

On Thursday, 19 June 2014 at 09:09, Bruno wrote:

Just use vagrant when developing anything that does not require a window environment.


Reply to this email directly or view it on GitHub (#6960 (comment)).

@lorenzogatti

I'm using Node only on Windows, exclusively to run the Handlebars and Node-Sass command-line compilers for an ASP.NET web application that will run only on Windows servers with Microsoft IIS and will only be accessed from Windows clients.
Switching away from Windows isn't going to be an option until I change job, and even in that case it's very unlikely.

I can ignore the arrogant morons who expect Windows users to stop using Windows, but this discussion has a more worrying problem: there is a lot of interest for redesigning the cumbersome scheme of duplicated dependencies in deeply nested folders, but nobody seems to care about fixing broken tools, which should be relatively few and easy to deal with.
There are vague mentions of git (not a part of normal package installation and loading, and certainly not a concern for Node and npm bug fixing), cmd (not a part of Node or npm either) and other even less relevant software; the only real error is ENOENT from npm install and nobody tries to figure out how and why it happens.

@stefanpenner

This is obviously just window's legacy baggage. Bug or not, node/npm must just assume this is required for a good experience. The inability to explore (or even delete) aspects of the node_modules tree with the most common tools just isn't acceptable.

Flattening the graph does increase cognitive overhead slightly (1 additional step for viewing)

before:

cd package/node_modules/dep_package_name

after:

cat package.json | grep dep_package_name
cd package/node_modules/dep_package_name@<version from page.json>

Anyways, the proposal:

flatten and key packages by version to allow npm's duplicate versions.

node_modules/package@version
node_modules/lodash@1.0.0

to handle the npm link correctly, merely adjust the resolution code to also give priority to a package without a version identifier. This also ensures hackability

node_modules/package@version
node_modules/lodash@1.0.0
node_modules/lodash  // takes priority for the parent module

example when a nested dependency has a linked dependency

node_modules/package@version/node_modules/lodash // takes priority for this package
node_modules/lodash@1.0.0

with this strategy:

  • mitigates the windows problem
  • maintains hackability
  • maintains npm link
  • dedupes by default
  • dedupes in cases the current npm dedupe does not
@maxrev17

This has caused issues on many projects, and Microsoft most definitely don't want to fitx it. I think the ultimate solution is to get on their case and show how all the effort they have put into node for visual studio etc. is wasted if they can't sort out this major issue with path lengths. As an aside I think node could do with better laying out its package dependencies and this thread has loads of good ideas!

@jhibbard-et

Just throwing this out there: i'm having the same problem as the original poster, and I only exclusively use git-bash... don't think i've used cmd, power shell, or whatever else it is windows has for a native terminal in years?

@matejkramny

It may be the filesystem limitation, not which terminal is being used..

Matej Kramny

On Thursday, 19 June 2014 at 14:09, jhibbard-et wrote:

Just throwing this out there: i'm having the same problem as the original poster, and I only exclusively use git-bash... don't think i've used cmd, power shell, or whatever else it is windows has for a native terminal in years?


Reply to this email directly or view it on GitHub (#6960 (comment)).

@jhibbard-et

I agree with you, Matej, completely.

@Mithgol
Mithgol commented Jun 19, 2014

@matt-bernhardt I haven't heard anything from you here in the last ≈6 days. Have you tried my npm install mocha --dev receipt and was it enough to reproduce some manifestation of the discussed problem?

@davedx
davedx commented Jun 19, 2014

"Don't use Windows" is a useless comment when Node claims to support Windows.

@Fishrock123
Member

@stefanpenner's proposal seems reasonable to me, what is blocking something like that?

@JeroMiya

@stefanpenner's proposal would be acceptable for my use case. I wonder if making it an option, on by default on windows only, would make it a more palatable change? Or default it to off everywhere, but add a windows-specific warning message on npm install that some long-path issues may occur and pointing them to the option?

@stnkris
stnkris commented Jun 19, 2014

Just say Node doesn't work with Windows and move on. Let's not PHP this...

@matejkramny

👍 Well said.

Matej Kramny

On Thursday, 19 June 2014 at 23:26, stnkris wrote:

Just say Node doesn't work with Windows and move on. Let's not PHP this...


Reply to this email directly or view it on GitHub (#6960 (comment)).

@zladuric

Any of the proposed changes to the require() would break a lot of existing apps and modules built. I'm just here to express my opinion add to the noise that Node and npm are not at fault - when you use them, they work. If you develop them with consideration to the OS you're developing on or deploying on, you should not have a problem. But if you install a third-party module and it breaks your installation, then you should probably talk to the third-party app developer. At the very least, Npm maintainers, not node - node itself provides a require() which will work out a deep structure on Windows.

If npm install mocha --dev breaks on Windows, has @Mithgol tried to ask Mocha devs? (I know it's TJ from Joyent, I'm trying to make a point here.)

@cowboy
cowboy commented Jun 20, 2014

The much, much larger issue here is that Node.js claims to support Windows, but it doesn't, unless it's overwhelmingly convenient to do so. I've seen Windows-related issues languish without comments or any kind of acknowledgment for YEARS.

It took nearly 2 years to get stdio to work correctly in Windows. There are a TON of path-related Windows installer issues still open, some of which are 2 years old. There's this path length issue, which displays a fundamental misunderstanding of the limitations inherent in Windows filesystems. I'm sure there are PLENTY of other unresolved Windows-specific issues.

I don't want Node.js to drop support for Windows, but the project REALLY needs to set users' expectations better. If it's going to support Windows, make it a priority. If not, TELL PEOPLE.

As it stands, for the last few years, Node.js Windows support has been unreliable at best.

EDIT: I just want to say that I've only put this comment here to counter other comments saying things like "well don't use Windows" or "Node and npm work fine" which distract from the core issue: Node.js makes claims that it doesn't really back up. I, personally, REALLY want Node.js to work on Windows.

@zladuric

Well, I can't say that I agree. Does that mean that Java or C++ also do not support Windows? Because people can do things with it which will not work?

@TooTallNate

If npm install mocha --dev breaks on Windows, has @Mithgol tried to ask Mocha devs? (I know it's TJ from Joyent, I'm trying to make a point here.)

You're confusing https://github.com/tjfontaine with https://github.com/visionmedia ;)

@isochronous

That's a false analogy. Java and C++ don't have dependency structures that
break the native tooling in windows. Node does.

I also have to disagree with your assertion that "if you develop them with
consideration to the OS you're developing or deploying on, you should not
have a problem," because for some cases, the only way to do that is to NOT
use a module that has a dependency tree that breaks the 255 character
limit. Even if you do that, and none of your dependencies or their
dependencies break that limit, there's nothing stopping ANYTHING in your
dependency tree from adding a new dependency in an updated version that
could break your carefully engineered project. Like I said before, as a
Windows developer, my experience is that Node works fine on Windows, right
up until it doesn't, and there's no way to know when it will work and when
it won't without actually installing a module. And as I just said, there's
no guarantee that a module that works fine at version 0.1.0 will continue
working at version 0.2.0, as it only takes a single added dependency that
itself has a deep dependency tree to break things.

On Thu, Jun 19, 2014 at 8:18 PM, Zlatko notifications@github.com wrote:

Well, I can't say that I agree. Does that mean that Java or C++ also do
not support Windows? Because people can do things with it which will not
work?


Reply to this email directly or view it on GitHub
#6960 (comment).

@darsain
darsain commented Jun 20, 2014

A little snippet. If this is representative of anything: https://sublime.wbond.net/packages/Nodejs

... than windows is the most popular platform to develop node on. And since the arrival of stuff like node-webkit, atom, lighttable, brackets, grunt, gulp, etc ... node is becoming also a platform that node is supposed to end up on.

Just to add to those well thought out ideas like "hurr durr, lets stop using/supporting windows".

That being said, this thread is becoming just a mosh pit. Would the intervention from maintainers be much to ask for? Where do you guys stand on this, which direction are you gonna move, etc...

There was a hint from isaacs that he is working on dedupe on install, but unfortunately without much details so I guess people choose to ignored it.

@efdee
efdee commented Jun 20, 2014

As a developer who both does Node.js development on Windows and uses tools based on Node.js in ASP.NET development flows, the amount of passive aggression towards Windows users in this thread is quite offensive. I have ran into the problem of nested directories going over the path name length limit on multiple occasions, both in CMD.EXE during Node.js development (the mocha example was given earlier in this thread) as in build tasks invoked by Visual Studio (through MSBuild, eg. for certain grunt tasks) and other third party tools.

If the official stance of the Node.js and NPM development teams is that this is "not a problem with Node.js or NPM" and it will not be fixed, this has a considerable impact on Node.js as a viable platform for development tools. Being part of an ecosystem means playing nice with the ecosystem, even if the problems "in theory" lies with other members.

I've been a fan of Node.js for a while and I would hate to have to let it go, but this thread is just extremely disappointing if not worrisome.

@manwood
manwood commented Jun 20, 2014

Another Windows dev for whom this is a showstopper.

@JeroMiya

I am a windows developer, and I wouldn't necessarily call this a showstopper personally. It's a pain point - one which requires some non-trivial amount of scripting and automation to work around within our continuous integration and code scanning environment, but it hasn't actually prevented us from using the product. At least not enough that I would go as far as labelling node "not windows compatible".

Our workaround involves:

  1. enabling long paths in git
  2. Using command line git or SourceTree, or any other git GUI that supports long paths (Visual Studio's git plugin is based on libgit2 which at the time of its creation did not support long file paths. I have an issue in MS connect for this already).
  3. Use rimraf to delete node_modules when you want a fresh install.
  4. After doing an npm install, do a dedupe, then create a tarball of node_modules. Check this into source control for CI and/or disaster recovery purposes.
  5. Incorporate as part of the CI build script a task that untars the node_modules tarball into node_modules before building.

I would find as an acceptable solution a utility (in node or from a third party) that automated this process without fundamentally changing the way node or npm works, if it is really THAT controversial to make changes to node for the benefit of windows users.

@isaacs
isaacs commented Jun 20, 2014

Hey, everybody. Just want to try to bring some sanity here, and make it clear what will (and won't) happen, and what we (node as well as npm-the-project and npm-the-company) do and don't care about. For those who don't know me, I am a Node core contributor, former Node BDFW, CEO of npm the company, and the BDFL of npm the OSS project. You can consider the following to be somewhat authoritative.

  1. Node almost certainly won't change its module system. I could be overruled on this by the rest of the project team, but that would be a much larger undertaking than just the code changes involved. If you're not prepared for a massive project with high costs and low chance of success, put that option out of your mind right now. Assume that node's module-loading logic is a fixed quantity, at least for the next year or two.
  2. npm will add dedupe-at-install-time by default. This is significantly more feasible than Node's module system changing, but it is still not exactly trivial, and involves a lot of reworking of some long-entrenched patterns. This requires a multi-stage install process to build the tree, then analyze it, and then reify it to the disk with as much atomicity as possible. We do need this refactor for many reasons, other than the Windows issues brought up here. (Eg, running out of memory, unnecessary fetches that can create race conditions, etc.) Work is underway. Most likely it will land in npm 1.5 or 1.6.
  3. Windows is a first-class citizen. It is extremely frustrating that the tools from Microsoft do not handle long directory paths as well as a third-party tool like Node, and that python and git (prior to 1.9) do not handle long paths properly, making node-gyp fail to build things deep in the tree. Even if we do everything right, we are harmed when others step on the landmines in our vicinity. But that frustration with git/del/cp/cmd/Explorer/etc must not be applied to Windows users, who we do care about very much. You are important to us, and "just use unix" is not a suitable answer.

With that in mind:

  1. Having this discussion here is probably less than useful. Node isn't going to change, so this isn't really a Node issue. The problem is not with the module loading semantics, but with the module installation semantics. (The two are related, but not identical.)
  2. There are a variety of ways to reify a dependency tree onto a fs structure, some flatter than others. Even deduped, there is still the possibility of eventually getting deep if there are deps that conflict with one another. However, rushing into a node_modules/module@version symlink nest is not something I'm eager to do without a lot of very careful planning. Anyone who remembers npm 0.x will understand why I am cautious. We may even want to make the reification strategy configurable, and have Windows default to a less-simple/flatter one, and Unix default to a simpler/less-flat one.
  3. Is it worth discussing setting a max folder depth within a package? All of our schemes here will be for naught if someone puts lib/a/b/c/d/a/b/c/d/a/b/c/d/a/b/c/d/a/b/c/d/... in their package folder. That's never been a problem so far, but it's worth pointing out that there is no way in principle to guarantee that npm never creates an excessively long path.

With these points in mind, I'm closing this issue, because it is not properly a Node issue, and the required npm work is already in progress. If other Node devs feel that there is more to discuss here regarding node, then we can always reopen it.

@isaacs isaacs closed this Jun 20, 2014
@dougwilson

Thanks @isaacs . Can someone please lock this GitHub issue, mainly so this answer will always be at the bottom when people come looking?

@phillip-haydon

Just finished reading this thread, so the conclusion is node's module system sucks but it won't be fixed? Lame.

@matejkramny

it doesn’t suck.

Matej Kramny

On Saturday, 21 June 2014 at 15:49, Phillip Haydon wrote:

Just finished reading this thread, so the conclusion is node's module system sucks but it won't be fixed? Lame.


Reply to this email directly or view it on GitHub (#6960 (comment)).

@phillip-haydon

It should pretty much drop the claim that it supports Windows.

@bmeck
bmeck commented Jun 21, 2014

I think instead of focusing on how to change the module system, it would be better for us to focus on what is broken. Mitigating the problem is the only solution; even with flattened modules it is just mitigation (what if I named my module something over 260 character [hyperbole]).

Lets get out there and fix em, it sounds like many developers either want to rewrite the module system (unlikely) or not change anything (unlikely). With some small fixes we can get ever closer to fixing people's ease of use problems without.

We should fix these problems and encourage other tools to do the same. As stated multiple times in this thread; Windows is supported, Microsoft uses Node, and there are tools we can fix :).

npm/npm#4012
npm/npm#5523
npm/npm#5525

@kleinfreund

Finished the thread. Is enabling longpaths with git config core.longpaths true the workaround to use? I'm hitting this while installing node packages for Gulp.js.

@innoscience

Was going nuts with this problem, glad to see I'm not alone. Wanted to add the note that using a VM isn't a solution if your project is in the shared path between your system and the VM (which is generally the case). The same errors occur as the files are not allowed to be written when installing the modules to your project when they hit the limit.

@edef1c
edef1c commented Jun 26, 2014

@innoscience We've seen npm breaking in varied, complicated ways with VM shared paths across different OSes. It's not really a supported thing (and very hard to support unfortunately, as much as I'd love to see that just magically work)

@LoveAndHappiness

Just wanted to say, how I solved it. I changed to Linux (Zorin Distribution). It costed me nearly two days to set up everything properly and I have a bunch of programs which aren't running, like the Adobe Collection, but at least gulp and yeoman are working, which is the reason for me to use node and npm. Btw, should't this threas been closed?

@phillip-haydon

The problem hasn't been solved. Just bad work around's. Core issue is unsolved because of reluctance to chance.

@bmeck
bmeck commented Jun 26, 2014

@phillip-haydon is there something that requires the change to the loader over making standard build tools / installation applications work?

@efdee
efdee commented Jun 26, 2014

@bmeck I'm not sure how you are planning to fix tools that are not open source. Eg: cmd.exe, MSBuild, Explorer.

@bmeck
bmeck commented Jun 26, 2014

@efdee only a concern with install / build phase of modules. Currently working on moving to a TMPDIR approach for building to fix most of those concerns (npm/npm#5523). Already have a PR for auto-enabling core.longpaths when npm shells out to git. Only issue left is the auto "dedup" (npm/npm#4012 ).

@efdee
efdee commented Jun 26, 2014

@bmeck It's not just an installation/build problem. We ran into errors due to long file names related to the nested node_modules structure when calling Grunt from inside an MSBuild buildfile. If you want I can look up what the exact scenario was.

@pixelshaded pixelshaded referenced this issue in jspm/jspm-cli Jun 26, 2014
Closed

Can't install on Mac #54

@StoneCypher

The amount of time people are spending trying to dust a design flaw under the rug as the operating system's fault is amazing.

It's the world's most common operating system. It doesn't matter whose fault it is. This is a critical and intolerable limitation in Node.

I can't check my Node projects into Git because the Windows Git client uses the default filesystem stuff.

This is because Node took an approach that is antagonistic to the world's most common operating system. This approach is not necessary or efficient.

Refusing to fix it by claiming the thing that was around 30 years earlier is just cheerleading.

Good engineers make tools that are usable, and do not defend unnecessarily broken design strategies as other peoples' fault.

@StoneCypher

@isaacs No. This is not Microsoft's fault. This is Node's. These design choices were in place before Node existed.

Playing games about wishing Microsoft would support things is beside the point. The Microsoft tools aren't the problem. We can't use you with Git. We can't use you with Bitbucket. We can't use you with compilers, with editors, with network tools.

We can't use you with anything because you broke the rules. Then you said "but there's this other secondary api, and we expect every tool that was ever written to change to conform to us!"

95% of the world's computers, dude. You're being a neckbeard and you're dramatically impeding Node's ability to grow so that you don't have to do a simple algorithm repair and admit you made a mistake.

This is not a legitimate choice.

@matejkramny

"This is a critical and intolerable limitation in Node.”

No, its a limitation of Windows.

"It's the world's most common operating system”

Of course, and the one developers use the least. In addition, 99% of servers are linux based.

Matej Kramny

On Thursday, 3 July 2014 at 03:50, John Haugeland wrote:

The amount of time people are spending trying to dust a design flaw under the rug as the operating system's fault is amazing.
It's the world's most common operating system. It doesn't matter whose fault it is. This is a critical and intolerable limitation in Node.
I can't check my Node projects into Git because the Windows Git client uses the default filesystem stuff.
This is because Node took an approach that is antagonistic to the world's most common operating system. This approach is not necessary or efficient.
Refusing to fix it by claiming the thing that was around 30 years earlier is just cheerleading.
Good engineers make tools that are usable, and do not defend unnecessarily broken design strategies as other peoples' fault.


Reply to this email directly or view it on GitHub (#6960 (comment)).

@phillip-haydon

Of course, and the one developers use the least. In addition, 99% of servers are linux based.

What absolute rubbish.

@othiym23
othiym23 commented Jul 3, 2014

@StoneCypher I understand your frustration, but I do believe this is an issue best handled by npm, simply because it's what's responsible for creating the nested folder structure in the first place. @isaacs and I have spent some time going over the various approaches described here and elsewhere, and I think we're in broad agreement that this problem can be resolved without making changes to the Node module loader.

That said, @isaacs, can you lock this issue now? The issue's been hashed over to death, and the arguing is no longer constructive or useful.

@StoneCypher

@othiym23 That was my initial reaction too. When I went to them, they were a lot more reasonable than the node people. They recognized that this was a fatal problem. However, their claim is that they have no choice but to use the node deployment system, which was frozen two years after this was first reported.

I tried asking #node.js about it, but I got a bunch of stuff like Matej, acting like if a 45 year old standard and everything written to it doesn't magically mutate to support this new package manager that can't cope with basic filesystem rules, it's everyone else's fault.

It is, actually, constructive and useful, and you should not lock it. The fundamental problem here is cultural: too many node devs are pretending that this isn't hurting people.

Please do not use censorship to suit your goal of pretending that Windows is a first class citizen. Either fix the package manager to work on the filesystem, or stop making the false claim that people are investing lost effort on the basis of.

If the ticket gets locked, people will just open new ones, because this is a very serious problem. Playing locking games leads to more noise, not less. Just unfollow.

@StoneCypher

Please do not close this ticket. The bug is unnecessary, serious, and fixable.

@phillip-haydon

Beginning to lose faith in Node, is this is the state of the project then I don't know how anyone can depend on it in production.

@khaledh
khaledh commented Jul 3, 2014

+1 for re-opening the ticket. This issue is not solved yet.

@StoneCypher

The NPM people state clearly that they have no choice but to follow the pattern that the Node people just said was NPM's fault.

This is not adequate behavior from a tool vendor. Tool vendors must keep high standards.

@StoneCypher

"To avoid this issue in the next release Lo-Dash core is inlining ~60 functions into various modules which greatly reduces the dep graphs (many to 0 deps) at the cost of duplication. It's a bummer of a band-aid but it's the state of things at the moment."

This is so broken that other libraries are starting to restructure themselves to cope.

@StoneCypher

@jdalton Is it possible to get the lo-dash that copes with this early?

@othiym23
othiym23 commented Jul 3, 2014

@StoneCypher I should have made clear in my earlier response that I work for (and on) npm and am the developer most likely to make whatever changes that the npm team / community deem best to better support Windows users. I want to see Windows users happy, because I think we make a good product and I want everyone to be able to take full advantage of it.

That said, this issue is already closed. It will not be reopened, nor should it be. I think that at this point all of the productive discussion has been exhausted, and instead of new ideas being brought to the table, the discussion is now circular and is doing nothing but frustrating people and causing conflict. Simply insisting that the issue is real, or saying that Windows is simply broken, is not a fruitful debate, it's just cheerleading.

If the Node core team thought that this was an issue that required their involvement, they would have weighed in at some point over the six months that this ticket has been around. Aside from a suggestion by @indutny and @trevnorris wondering why the discussion is happening here and not on npm's issue tracker, they haven't.

Without speaking on anyone's behalf, I'd suggest that's for a very simple reason: Node's module loader is probably the most frozen piece of its API surface, and it's the primary abstraction that connects the 80k+ modules that form the Node module ecosystem. npm has a certain amount of freedom to restructure how it installs modules in projects, as long as it follows the Node module system's rules; if it has that freedom but Node doesn't, why shouldn't it be the piece to change?

It is most definitely not censorship to close out a topic of discussion after the conversation has run out of gas. An issues thread isn't a democratic debate, nor is it a straw poll; it's a place to identify technical flaws and bring them to resolution. As I said earlier, I understand your frustration, and I agree that there is work to be done here. I don't think that either @isaacs or I have done anything that would suggest that we don't take this problem seriously. Beyond that, I don't see any suggestions or proposals in your comments. As you say, you're free to open new issues to keep hashing this out, but at this point, I consider this particular issue an attractive nuisance.

@innoscience

Pragmatically speaking, if there is a refusal or inability to fix this issue, then this needs to be disclosed plainly and openly to all people planning on using Node so everyone is clear on cross-platform compatibility. Something to the effect that the windows platform is not fully supported due to file path limitations and Node / NPM may fail under certain conditions.

@StoneCypher

@othiym23 If you choose not to repair or provide a successor to the module system, then please stop making claims of first class windows support.

Javascript libraries are now modifying their own structure to avoid the support requests that come from this design decision of yours.

Nothing's stopping you from implementing a node_mod2 system alongside the existing one to allow new work to transition to a safer model.

Please don't claim support that you don't actually offer. That does economic damage to others who believed you. Thanks.

@StoneCypher

@othiym23 I'm not entirely sure why you don't just provide a successor and let modules transition. That would be easy and straightforward, would provide no backwards breakage, would allow you to fix the problem that you suggest that you take seriously, would require no effort on your developers' parts other than in the future to put a 2 on some API call, and would also give you the opportunity to apply whatever other lessons you have learned in the meantime.

The reason I didn't offer any solutions is I discovered this ticket where you claimed the problem was Windows' fault, and after you had already made your decision.

I was pleading with you not to turn a blind eye this way.

I'm relatively new to node. I started two days ago, because someone did some work for me in it. This is my first experience with your platform. When I googled it I found dozens of bizarre stackoverflow and reddit posts of the form "help! node put down an illegal directory and I can't even delete it!"

And your response? "Windows is buggy because it behaves the same way it has since three years before JavaScript existed. If Windows doesn't change for us, thereby undermining every other piece of Windows software, then they are in the wrong!"

No, man, they really aren't.

There are tons of answers of what you can do. You're just not going "oh my god, red alarm, everyone is furious and it doesn't work!"

The reason people are speaking to you in the tone that they are is that you give the very strong impression of saying you take things seriously, but in the same thread, you say things like "This is the fault of a company that made this decision before the language we're working in was even created, and we actually genuinely expect them to break everyone else to satisfy us!"

Fundamentally, a boneheaded decision was made at the outset. You won't admit that. As a result, you can't actually get to the point of saying "well, now we need to do something about this."

If you actually wanted to do something about this, it would not be very hard to solve.

But you want to make sure that you don't feel like anyone made any mistakes, and if that means everything was right, then you end up ignoring it for six months then closing it unfixed, while telling yourself how seriously you take it.

We don't care how seriously you take it, if it ends up unfixed at the end of the day, and you end up telling us "go away, we don't want to hear about this anymore."

The problem doesn't end just because you blame someone else and close the ticket, buddy.

You see other people within an hour of calling for silence saying "I don't see how I can take these guys seriously in a production setting."

That's an expensive message to send. Are you really sure you're treating the community in a way that's going to result in them feeling like you're trying to make sure their problems get handled?

I would much rather have a vendor who didn't take me seriously at all, but fixed my problem, than one who took my problem very seriously, but closed the ticket with the issue unfixed six months later, with half a dozen perfectly fine suggestions in the thread, and literally several books written about this exact topic (ie, fixing bad choices made in the past without breaking legacy support.)

If you take this seriously, why isn't it fixed? It isn't unfixable.

Buddy, I don't know you. All I know is I told someone that they could do work for me in Node because Node claimed first class Windows support, except I can't install d3 or lodash or moustache or facebook react or graphael, all because the javascript engine chose a path system that was well defined to be wrong before they got started, and are now blaming Microsoft for their own mistake.

Dude, no, this isn't a bug in Windows. You're trying to hack around things by using the Unicode API for something it wasn't intended for, then saying "well all Windows software should be expected to update itself to obscure unofficial APIs that don't even exist in most languages so that we don't have to make a replacement module api!"

Your staff are busily closing all of your customers' and potential customers' avenues for telling you that they are unhappy because the tool you made has an unnecessary defect.

Please just fix the path strategy of your module system, or add a successor path system without the limitation so that library vendors can self-migrate past the problem. It's not a lot to ask for a tool vendor that claims first class support to be expected fundamentally to work at all.

@isaacs isaacs locked and limited conversation to collaborators Jul 3, 2014
@isaacs
isaacs commented Jul 3, 2014

For posterity, this was the resolution of the OP: #6960 (comment)

We are working on changing the way that npm lays modules out on the file system. Node is not the faulty technology here, it has nothing to do with package installation.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.