Node/npm MAX_PATH issues

[MarkStega]

There is a very well known issue with Node/NPM module management having to do with the maximum path length on Windows, see this exhausting thread: Node's nested node_modules approach is basically incompatible with Windows #6960 . So I am trying to build the new Aurelia framework (successor to Durandal and somewhat a competitor to Angular) and immediately ran into the issue when build of dependency modules failed because of the path. Is there any direction from Microsoft regarding this issue? With the seeming embrace of Node by MS having this flaw will be hurtful to adoption.

[mousetraps]

And by "direction", you mean "workaround", right? :-) There are a few options...

1. We have warnings when you install new packages that offer to run 'npm dedupe'. And now that you mention it... it would probably be good to light this up when the build fails.
2. sometimes that doesn't work, so we also suggest people start their projects in a short path like c:\src (embarassing, I know)
3. I haven't used this extensively, but some users have had success with a module called 'flatten-packages'.
4. Patiently await a flatter node_modules directory - http://blog.npmjs.org/post/111968476155/npm-weekly-6
5. groan very loudly when it happens - no good keeping those emotions bottled up ;-)

Luckily _#1/#_2 cover most cases.

One thing to be aware of with this is that dedupe, flatten-packages, and upcoming changes in npm@3 might have an adverse effect on IntelliSense due to the way the analysis engine currently analyzes dependencies. I haven't heard any complaints yet, but until we fix this, your mileage may vary.

With regards to the technicalities of max_path itself... the issue is with the .NET file I/O API, which doesn't allow you to use ?\ paths to work around it, and is too big for our team in particular to do much about. Here's a good article that might provide a little more context (it's a little old, and these "plans" they speak of never actually materialized.)
http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx

That said... we don't like it either, and we are still thinking about what we can do.

[MarkStega]

I am absolutely new to the environment of Node/NPM so bear with me for another set of questions.
The project that I am attempting to build/run starts with the download of a source tree that has a package.json. Instructions are to run "npm install" then "jspm install -y".

I successfully created a VS nodejs solution using the "From existing NodeJS code"

1. I right clicked the npm node and executed the 'install missing packages'; I did indeed get the popup indicating that I was past the 260 character path limit. Running dedup was not sufficient. A shorter starting path won't help as the socket-io package goes many characters past 260,not just a few so Add README.md file to repository #2 won't work. I installed and ran 'flatten' and it seems that it successfully moved all of the dependencies to the 'node_modules' directory. But the flatten takes the most recent version of a package and this introduced errors when running the solution. So Code improperly indents after if block inside inline callback function #3 doesn't work. It appears that in the case of this project that Update mocha.js to correct typos #4 & Startup file not preserved after being moved to another directory #5 are the solutions that will work (hopefully).

2. Are there plans to support jspm in a similar fashion to npm?

[mousetraps]

Np, they're good questions :-)

1. After flatten, you should be able to uninstall the root package with the version conflict, and install the proper versions under the packages that need it. If you're still hitting MAX_PATH, then you can delete the node_modules folder under the new version that you installed, cross your fingers the packages in the root node_modules folder don't conflict, rinse and repeat until you've resolved all the conflicts. Basically you'll be emulating what npm@3 will be doing by creating a flat directory structure, except when there are conflicts, in which case those conflicting versions are not bubbled all the way up to the project root.
2. The cmd line should play nice with VS, but no plans to support it through UI. We can certainly discuss if you're interested in contributing. :)

[mousetraps]

Also if you want support for jspm, the web essentials extension might be another route to making it happen: http://webessentials.uservoice.com/forums/140520-general/suggestions/7136766-add-support-for-jspm

[MarkStega]

1. The flatten, although causing other issues, made finding the error in the non-flattened form easier. When I removed socket-io and redid the install I realized that the build error was not due to path depth but rather a reference to VS2010 tools (which don't exist on my development system). A simple "npm config set msvs_version 2013 --global" followed by a new copy of the source and an "npm install" produced a working sample (even if I can't examine some of the dependency tree).

2. Thanks for the JSPM/Web Essentials thought; For now, the command line is not too onerous.

[phillipwei]

I was battling with path issues. I found that, for my situation, the following didn't work:

• using a shorter directory.
• dedupe
• flatten-package.
• npm install -g.

The only thing that did work was to install, trigger the path error, and use that to locate the most common offending module; uninstall; and then re-install the common module first.

[ghost]

There is no good way around it on Windows. Making paths shorter either through global install or subst or renaming are all hacks and do not address the issue. Sooner or later you'll run into a situation when none of those work. The only solution as of now is move away from developing in node on Windows as Node is not fully compatible with it. Period. It is best to migrate sooner rather than later to avoid running into this issue when the project is at a stage when moving to a different OS can cause a lot of headache ...

[glen-84]

Related: https://github.com/dotnet/corefx/issues/645

[itanex]

@mousetraps with npm@3.1.1 recently released the long path issue is expected to be resolved or at least mitigated by how packaging is applied into the node_module folder. Since it is still a bit beta-ish, you have to use npm version specific update. This creates a confusing aspect in the VS solution display now.

As expected a package in this folder not in the package.json file displays as &<package>@version (not listed in package.json) This presents a confusing aspect now as the previous expected result fulfilled the need for excessive unreferenced dependencies.

[mousetraps]

@itanex see #233. We're working on it :-)

[itanex]

@MarkStega and the rest landing here.

This feels a bit hacking in windows where I expected the NPM version that VS used would be the same as displayed on the command line. However, when updating NPM if you use npm install npm@3.1.1 -g as suggested by various SO and other pages. This updates the NPM located in your %AppDataRoaming%\npm location which is apparently not the version found by VS.

To fix this you have to be a bit brave. Here are 7 simple yet scary steps.

1. Locate the nodejs MSI install location usually C:\Program Files\nodejs\
2. Open the node_modules folder
3. Backup the NPM folder either copying it or sending it to a zip file
4. Delete the NPM folder
5. Open your command line tool and use npm install npm@3.1.1 -g or higher version if you prefer
6. Open the NPM\node_modules folder in your %AppDataRoaming% folder
7. Copy the NPM Module folder, NOT THE ROOT FOLDER, to the installed directory modules folder we removed it from.
8. Delete the NPM Module from your %AppDataRoaming% folder

I am using VS 2015RC and I didn't have to restart for this to take effect. I just deleted my projects node_module folder and ran the update missing packages command and everything worked with 3.1.1 referencing flat flat flat

If this worked for you your free to delete your backup as you won't be needing it anymore.

[itanex]

@mousetraps yeah, looks like your on this one. It'll be interesting to see what comes in the next few short weeks. Can't wait till RTM hits, these tools are invaluable for the VS Node Developer thanks!

[mousetraps]

@itanex thx for posting the steps here. Also check out the npm-windows-upgrade package that was recently released which makes things a bit easier.

• https://github.com/felixrieseberg/npm-windows-upgrade

Also 3.1.2 is out now which fixes some windows issues, so use the following instead.
npm install -g npm@v3.x-next

[NNM11]

@mousetraps - Hi. Does this npm-windows-upgrade package fix the same problem NTVS 1.1 Tools for Windows 10 IoT Core? Does this apply to the MSFT's implementation of Node.js runtime with Chakra engine for Windows IoT Core - i.e. the deployed node.js applications using npm to pull the required modules?

[mousetraps]

@NNM11 the package just makes it easier to upgrade to npm v3 (which is still very much in a beta state, and as I remarked above we are still working on supporting it fully in NTVS)

Re: IoT none of the proposed node changes to enable chakra involve npm, so presumably you can just hotswap npm v3 the same way you'd normally do (plus accounting for different paths and such if need be). @munyirik can provide more details on that since he's the dev on the IoT extension.

[munyirik]

Hi @NNM11,
Apologies for the delay.
***Does this npm-windows-upgrade package fix the same problem NTVS 1.1 Tools for Windows 10 IoT Core?
It should but it's not tested yet. We are working on this. Note that the IoT Extension uses the same npm functionality provided by NTVS. So the extension will use whatever version of npm that NTVS uses.

***Does this apply to the MSFT's implementation of Node.js runtime with Chakra engine for Windows IoT Core - i.e. the deployed node.js applications using npm to pull the required modules?
npm wasn't customized for Chakra (only node-gyp) so npm v3 should work by swapping node-gyp. This is also not fully tested yet.

[NNM11]

Hi @munyirik,

I would love to see a full support of the NPM@3 (to flatten the path) with NTVS 1.1 Tools with Win10 IoT Core.

I can help evaluating the NMP3 support, if you have the node-gyp swapping procedure for the node.js Chakra engine for IoT core - I did not find a documented way to modify the node.js implementation for the IoT OS image. Thx

[munyirik]

Thanks @NNM11 . The swapping procedure should only be required if you're building a native node.js addon - I'll keep you posted when we have something working. For a pure JS package (e.g. Express), you can upgrade to npmv3 and use node.js (chakra) with it. BUT this is independent of NTVS (i.e. it works when testing through command line) - the integration of npmv3 and NTVS is what mousetraps is working on.

[NNM11]

Thanks @munyirik for the explanation.
@mousetraps - let me know if I can help with the npmv3 and NTVS (UWP) integration. Thx.

[mousetraps]

@MarkStega re: jspm - it's not full support, but check out this new VS extension
https://visualstudiogallery.msdn.microsoft.com/753b9720-1638-4f9a-ad8d-2c45a410fd74

[kant2002]

1.1 RC2 is out with NPM3 support. That could helps with MAX_PATH issue.

[MarkStega]

Oops - Forgot to comment; I installed the NPM3 beta (removing the NPM2 installation) a few weeks ago. It does solve the MAX_PATH issue (at least for the Aurelia navigation project).

[linquize]

after flatten, restart VS, rebuild project

[graphnode]

👍 for @linquize tip

After I npm install npm -g to get the latest version, deleted the project's node_modules and did npm install I was still having MAX_PATH problems even when what npm ls was showing was pretty flat.

Restarting VS solved the problem.

Edit: I also reordered external web tools paths in the options so that PATH was above the vs one. Not sure if helped but I leave it here.

[AdamAndersonFalafelSoftware]

I simply installed the latest npm with "npm install npm -g" and restarted Visual Studio. No "scary steps" necessary. Confirmed by deleting the old node_modules folder (which requires special treatment to delete the very long paths) and using Visual Studio to install missing packages. The newly installed packages used the new flattened structure as expected for NPM 3.x.

[mousetraps]

Feel free to continue discussing, but I'm going to go ahead and close this issue in favor of the guidance available in our new Node.js Guidelines repo, which sums up many of the tips in this thread: https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#max_path-explanation-and-workarounds

And of course, PRs are welcome 😃

[karlhorky]

Apparently the long paths issue on Windows is still not resolved with modern Node.js 20.9.0, on a machine with LongPathsEnabled set to true

Opened an issue to revisit this MAX_PATH explanation and workarounds guide in case anything has changed with Node.js support for long 260+ character paths on Windows:

• Update MAX_PATH guidance for LongPathsEnabled? nodejs-guidelines#108