New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node/npm MAX_PATH issues #69

Closed
MarkStega opened this Issue Apr 16, 2015 · 26 comments

Comments

Projects
None yet
@MarkStega

MarkStega commented Apr 16, 2015

There is a very well known issue with Node/NPM module management having to do with the maximum path length on Windows, see this exhausting thread: Node's nested node_modules approach is basically incompatible with Windows #6960 . So I am trying to build the new Aurelia framework (successor to Durandal and somewhat a competitor to Angular) and immediately ran into the issue when build of dependency modules failed because of the path. Is there any direction from Microsoft regarding this issue? With the seeming embrace of Node by MS having this flaw will be hurtful to adoption.

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Apr 16, 2015

Contributor

And by "direction", you mean "workaround", right? :-) There are a few options...

  1. We have warnings when you install new packages that offer to run 'npm dedupe'. And now that you mention it... it would probably be good to light this up when the build fails.
  2. sometimes that doesn't work, so we also suggest people start their projects in a short path like c:\src (embarassing, I know)
  3. I haven't used this extensively, but some users have had success with a module called 'flatten-packages'.
  4. Patiently await a flatter node_modules directory - http://blog.npmjs.org/post/111968476155/npm-weekly-6
  5. groan very loudly when it happens - no good keeping those emotions bottled up ;-)

Luckily _#1/#_2 cover most cases.

One thing to be aware of with this is that dedupe, flatten-packages, and upcoming changes in npm@3 might have an adverse effect on IntelliSense due to the way the analysis engine currently analyzes dependencies. I haven't heard any complaints yet, but until we fix this, your mileage may vary.

With regards to the technicalities of max_path itself... the issue is with the .NET file I/O API, which doesn't allow you to use ?\ paths to work around it, and is too big for our team in particular to do much about. Here's a good article that might provide a little more context (it's a little old, and these "plans" they speak of never actually materialized.)
http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx

That said... we don't like it either, and we are still thinking about what we can do.

Contributor

mousetraps commented Apr 16, 2015

And by "direction", you mean "workaround", right? :-) There are a few options...

  1. We have warnings when you install new packages that offer to run 'npm dedupe'. And now that you mention it... it would probably be good to light this up when the build fails.
  2. sometimes that doesn't work, so we also suggest people start their projects in a short path like c:\src (embarassing, I know)
  3. I haven't used this extensively, but some users have had success with a module called 'flatten-packages'.
  4. Patiently await a flatter node_modules directory - http://blog.npmjs.org/post/111968476155/npm-weekly-6
  5. groan very loudly when it happens - no good keeping those emotions bottled up ;-)

Luckily _#1/#_2 cover most cases.

One thing to be aware of with this is that dedupe, flatten-packages, and upcoming changes in npm@3 might have an adverse effect on IntelliSense due to the way the analysis engine currently analyzes dependencies. I haven't heard any complaints yet, but until we fix this, your mileage may vary.

With regards to the technicalities of max_path itself... the issue is with the .NET file I/O API, which doesn't allow you to use ?\ paths to work around it, and is too big for our team in particular to do much about. Here's a good article that might provide a little more context (it's a little old, and these "plans" they speak of never actually materialized.)
http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx

That said... we don't like it either, and we are still thinking about what we can do.

@MarkStega

This comment has been minimized.

Show comment
Hide comment
@MarkStega

MarkStega Apr 17, 2015

I am absolutely new to the environment of Node/NPM so bear with me for another set of questions.
The project that I am attempting to build/run starts with the download of a source tree that has a package.json. Instructions are to run "npm install" then "jspm install -y".

I successfully created a VS nodejs solution using the "From existing NodeJS code"

  1. I right clicked the npm node and executed the 'install missing packages'; I did indeed get the popup indicating that I was past the 260 character path limit. Running dedup was not sufficient. A shorter starting path won't help as the socket-io package goes many characters past 260,not just a few so #2 won't work. I installed and ran 'flatten' and it seems that it successfully moved all of the dependencies to the 'node_modules' directory. But the flatten takes the most recent version of a package and this introduced errors when running the solution. So #3 doesn't work. It appears that in the case of this project that #4 & #5 are the solutions that will work (hopefully).

  2. Are there plans to support jspm in a similar fashion to npm?

MarkStega commented Apr 17, 2015

I am absolutely new to the environment of Node/NPM so bear with me for another set of questions.
The project that I am attempting to build/run starts with the download of a source tree that has a package.json. Instructions are to run "npm install" then "jspm install -y".

I successfully created a VS nodejs solution using the "From existing NodeJS code"

  1. I right clicked the npm node and executed the 'install missing packages'; I did indeed get the popup indicating that I was past the 260 character path limit. Running dedup was not sufficient. A shorter starting path won't help as the socket-io package goes many characters past 260,not just a few so #2 won't work. I installed and ran 'flatten' and it seems that it successfully moved all of the dependencies to the 'node_modules' directory. But the flatten takes the most recent version of a package and this introduced errors when running the solution. So #3 doesn't work. It appears that in the case of this project that #4 & #5 are the solutions that will work (hopefully).

  2. Are there plans to support jspm in a similar fashion to npm?

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Apr 17, 2015

Contributor

Np, they're good questions :-)

  1. After flatten, you should be able to uninstall the root package with the version conflict, and install the proper versions under the packages that need it. If you're still hitting MAX_PATH, then you can delete the node_modules folder under the new version that you installed, cross your fingers the packages in the root node_modules folder don't conflict, rinse and repeat until you've resolved all the conflicts. Basically you'll be emulating what npm@3 will be doing by creating a flat directory structure, except when there are conflicts, in which case those conflicting versions are not bubbled all the way up to the project root.
  2. The cmd line should play nice with VS, but no plans to support it through UI. We can certainly discuss if you're interested in contributing. :)
Contributor

mousetraps commented Apr 17, 2015

Np, they're good questions :-)

  1. After flatten, you should be able to uninstall the root package with the version conflict, and install the proper versions under the packages that need it. If you're still hitting MAX_PATH, then you can delete the node_modules folder under the new version that you installed, cross your fingers the packages in the root node_modules folder don't conflict, rinse and repeat until you've resolved all the conflicts. Basically you'll be emulating what npm@3 will be doing by creating a flat directory structure, except when there are conflicts, in which case those conflicting versions are not bubbled all the way up to the project root.
  2. The cmd line should play nice with VS, but no plans to support it through UI. We can certainly discuss if you're interested in contributing. :)

@mousetraps mousetraps changed the title from Node/NPM path issues to Node/npm MAX_PATH issues Apr 17, 2015

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Apr 18, 2015

Contributor

Also if you want support for jspm, the web essentials extension might be another route to making it happen: http://webessentials.uservoice.com/forums/140520-general/suggestions/7136766-add-support-for-jspm

Contributor

mousetraps commented Apr 18, 2015

Also if you want support for jspm, the web essentials extension might be another route to making it happen: http://webessentials.uservoice.com/forums/140520-general/suggestions/7136766-add-support-for-jspm

@MarkStega

This comment has been minimized.

Show comment
Hide comment
@MarkStega

MarkStega Apr 18, 2015

  1. The flatten, although causing other issues, made finding the error in the non-flattened form easier. When I removed socket-io and redid the install I realized that the build error was not due to path depth but rather a reference to VS2010 tools (which don't exist on my development system). A simple "npm config set msvs_version 2013 --global" followed by a new copy of the source and an "npm install" produced a working sample (even if I can't examine some of the dependency tree).

  2. Thanks for the JSPM/Web Essentials thought; For now, the command line is not too onerous.

MarkStega commented Apr 18, 2015

  1. The flatten, although causing other issues, made finding the error in the non-flattened form easier. When I removed socket-io and redid the install I realized that the build error was not due to path depth but rather a reference to VS2010 tools (which don't exist on my development system). A simple "npm config set msvs_version 2013 --global" followed by a new copy of the source and an "npm install" produced a working sample (even if I can't examine some of the dependency tree).

  2. Thanks for the JSPM/Web Essentials thought; For now, the command line is not too onerous.

@phillipwei

This comment has been minimized.

Show comment
Hide comment
@phillipwei

phillipwei May 7, 2015

I was battling with path issues. I found that, for my situation, the following didn't work:

  • using a shorter directory.
  • dedupe
  • flatten-package.
  • npm install -g.

The only thing that did work was to install, trigger the path error, and use that to locate the most common offending module; uninstall; and then re-install the common module first.

phillipwei commented May 7, 2015

I was battling with path issues. I found that, for my situation, the following didn't work:

  • using a shorter directory.
  • dedupe
  • flatten-package.
  • npm install -g.

The only thing that did work was to install, trigger the path error, and use that to locate the most common offending module; uninstall; and then re-install the common module first.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 9, 2015

There is no good way around it on Windows. Making paths shorter either through global install or subst or renaming are all hacks and do not address the issue. Sooner or later you'll run into a situation when none of those work. The only solution as of now is move away from developing in node on Windows as Node is not fully compatible with it. Period. It is best to migrate sooner rather than later to avoid running into this issue when the project is at a stage when moving to a different OS can cause a lot of headache ...

ghost commented May 9, 2015

There is no good way around it on Windows. Making paths shorter either through global install or subst or renaming are all hacks and do not address the issue. Sooner or later you'll run into a situation when none of those work. The only solution as of now is move away from developing in node on Windows as Node is not fully compatible with it. Period. It is best to migrate sooner rather than later to avoid running into this issue when the project is at a stage when moving to a different OS can cause a lot of headache ...

@glen-84

This comment has been minimized.

Show comment
Hide comment
@glen-84

glen-84 commented May 22, 2015

@itanex

This comment has been minimized.

Show comment
Hide comment
@itanex

itanex Jul 14, 2015

@mousetraps with npm@3.1.1 recently released the long path issue is expected to be resolved or at least mitigated by how packaging is applied into the node_module folder. Since it is still a bit beta-ish, you have to use npm version specific update. This creates a confusing aspect in the VS solution display now.

As expected a package in this folder not in the package.json file displays as &<package>@version (not listed in package.json) This presents a confusing aspect now as the previous expected result fulfilled the need for excessive unreferenced dependencies.

itanex commented Jul 14, 2015

@mousetraps with npm@3.1.1 recently released the long path issue is expected to be resolved or at least mitigated by how packaging is applied into the node_module folder. Since it is still a bit beta-ish, you have to use npm version specific update. This creates a confusing aspect in the VS solution display now.

As expected a package in this folder not in the package.json file displays as &<package>@version (not listed in package.json) This presents a confusing aspect now as the previous expected result fulfilled the need for excessive unreferenced dependencies.

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Jul 14, 2015

Contributor

@itanex see #233. We're working on it :-)

Contributor

mousetraps commented Jul 14, 2015

@itanex see #233. We're working on it :-)

@itanex

This comment has been minimized.

Show comment
Hide comment
@itanex

itanex Jul 14, 2015

@MarkStega and the rest landing here.

This feels a bit hacking in windows where I expected the NPM version that VS used would be the same as displayed on the command line. However, when updating NPM if you use npm install npm@3.1.1 -g as suggested by various SO and other pages. This updates the NPM located in your %AppDataRoaming%\npm location which is apparently not the version found by VS.

To fix this you have to be a bit brave. Here are 7 simple yet scary steps.

  1. Locate the nodejs MSI install location usually C:\Program Files\nodejs\
  2. Open the node_modules folder
  3. Backup the NPM folder either copying it or sending it to a zip file
  4. Delete the NPM folder
  5. Open your command line tool and use npm install npm@3.1.1 -g or higher version if you prefer
  6. Open the NPM\node_modules folder in your %AppDataRoaming% folder
  7. Copy the NPM Module folder, NOT THE ROOT FOLDER, to the installed directory modules folder we removed it from.
  8. Delete the NPM Module from your %AppDataRoaming% folder

I am using VS 2015RC and I didn't have to restart for this to take effect. I just deleted my projects node_module folder and ran the update missing packages command and everything worked with 3.1.1 referencing flat flat flat

If this worked for you your free to delete your backup as you won't be needing it anymore.

itanex commented Jul 14, 2015

@MarkStega and the rest landing here.

This feels a bit hacking in windows where I expected the NPM version that VS used would be the same as displayed on the command line. However, when updating NPM if you use npm install npm@3.1.1 -g as suggested by various SO and other pages. This updates the NPM located in your %AppDataRoaming%\npm location which is apparently not the version found by VS.

To fix this you have to be a bit brave. Here are 7 simple yet scary steps.

  1. Locate the nodejs MSI install location usually C:\Program Files\nodejs\
  2. Open the node_modules folder
  3. Backup the NPM folder either copying it or sending it to a zip file
  4. Delete the NPM folder
  5. Open your command line tool and use npm install npm@3.1.1 -g or higher version if you prefer
  6. Open the NPM\node_modules folder in your %AppDataRoaming% folder
  7. Copy the NPM Module folder, NOT THE ROOT FOLDER, to the installed directory modules folder we removed it from.
  8. Delete the NPM Module from your %AppDataRoaming% folder

I am using VS 2015RC and I didn't have to restart for this to take effect. I just deleted my projects node_module folder and ran the update missing packages command and everything worked with 3.1.1 referencing flat flat flat

If this worked for you your free to delete your backup as you won't be needing it anymore.

@itanex

This comment has been minimized.

Show comment
Hide comment
@itanex

itanex Jul 14, 2015

@mousetraps yeah, looks like your on this one. It'll be interesting to see what comes in the next few short weeks. Can't wait till RTM hits, these tools are invaluable for the VS Node Developer thanks!

itanex commented Jul 14, 2015

@mousetraps yeah, looks like your on this one. It'll be interesting to see what comes in the next few short weeks. Can't wait till RTM hits, these tools are invaluable for the VS Node Developer thanks!

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Jul 15, 2015

Contributor

@itanex thx for posting the steps here. Also check out the npm-windows-upgrade package that was recently released which makes things a bit easier.

Also 3.1.2 is out now which fixes some windows issues, so use the following instead.
npm install -g npm@v3.x-next

Contributor

mousetraps commented Jul 15, 2015

@itanex thx for posting the steps here. Also check out the npm-windows-upgrade package that was recently released which makes things a bit easier.

Also 3.1.2 is out now which fixes some windows issues, so use the following instead.
npm install -g npm@v3.x-next

@NNM11

This comment has been minimized.

Show comment
Hide comment
@NNM11

NNM11 Jul 20, 2015

@mousetraps - Hi. Does this npm-windows-upgrade package fix the same problem NTVS 1.1 Tools for Windows 10 IoT Core? Does this apply to the MSFT's implementation of Node.js runtime with Chakra engine for Windows IoT Core - i.e. the deployed node.js applications using npm to pull the required modules?

NNM11 commented Jul 20, 2015

@mousetraps - Hi. Does this npm-windows-upgrade package fix the same problem NTVS 1.1 Tools for Windows 10 IoT Core? Does this apply to the MSFT's implementation of Node.js runtime with Chakra engine for Windows IoT Core - i.e. the deployed node.js applications using npm to pull the required modules?

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Jul 27, 2015

Contributor

@NNM11 the package just makes it easier to upgrade to npm v3 (which is still very much in a beta state, and as I remarked above we are still working on supporting it fully in NTVS)

Re: IoT none of the proposed node changes to enable chakra involve npm, so presumably you can just hotswap npm v3 the same way you'd normally do (plus accounting for different paths and such if need be). @munyirik can provide more details on that since he's the dev on the IoT extension.

Contributor

mousetraps commented Jul 27, 2015

@NNM11 the package just makes it easier to upgrade to npm v3 (which is still very much in a beta state, and as I remarked above we are still working on supporting it fully in NTVS)

Re: IoT none of the proposed node changes to enable chakra involve npm, so presumably you can just hotswap npm v3 the same way you'd normally do (plus accounting for different paths and such if need be). @munyirik can provide more details on that since he's the dev on the IoT extension.

@munyirik

This comment has been minimized.

Show comment
Hide comment
@munyirik

munyirik Aug 5, 2015

Contributor

Hi @NNM11,
Apologies for the delay.
***Does this npm-windows-upgrade package fix the same problem NTVS 1.1 Tools for Windows 10 IoT Core?
It should but it's not tested yet. We are working on this. Note that the IoT Extension uses the same npm functionality provided by NTVS. So the extension will use whatever version of npm that NTVS uses.

***Does this apply to the MSFT's implementation of Node.js runtime with Chakra engine for Windows IoT Core - i.e. the deployed node.js applications using npm to pull the required modules?
npm wasn't customized for Chakra (only node-gyp) so npm v3 should work by swapping node-gyp. This is also not fully tested yet.

Contributor

munyirik commented Aug 5, 2015

Hi @NNM11,
Apologies for the delay.
***Does this npm-windows-upgrade package fix the same problem NTVS 1.1 Tools for Windows 10 IoT Core?
It should but it's not tested yet. We are working on this. Note that the IoT Extension uses the same npm functionality provided by NTVS. So the extension will use whatever version of npm that NTVS uses.

***Does this apply to the MSFT's implementation of Node.js runtime with Chakra engine for Windows IoT Core - i.e. the deployed node.js applications using npm to pull the required modules?
npm wasn't customized for Chakra (only node-gyp) so npm v3 should work by swapping node-gyp. This is also not fully tested yet.

@NNM11

This comment has been minimized.

Show comment
Hide comment
@NNM11

NNM11 Aug 7, 2015

Hi @munyirik,

I would love to see a full support of the NPM@3 (to flatten the path) with NTVS 1.1 Tools with Win10 IoT Core.

I can help evaluating the NMP3 support, if you have the node-gyp swapping procedure for the node.js Chakra engine for IoT core - I did not find a documented way to modify the node.js implementation for the IoT OS image. Thx

NNM11 commented Aug 7, 2015

Hi @munyirik,

I would love to see a full support of the NPM@3 (to flatten the path) with NTVS 1.1 Tools with Win10 IoT Core.

I can help evaluating the NMP3 support, if you have the node-gyp swapping procedure for the node.js Chakra engine for IoT core - I did not find a documented way to modify the node.js implementation for the IoT OS image. Thx

@munyirik

This comment has been minimized.

Show comment
Hide comment
@munyirik

munyirik Aug 7, 2015

Contributor

Thanks @NNM11 . The swapping procedure should only be required if you're building a native node.js addon - I'll keep you posted when we have something working. For a pure JS package (e.g. Express), you can upgrade to npmv3 and use node.js (chakra) with it. BUT this is independent of NTVS (i.e. it works when testing through command line) - the integration of npmv3 and NTVS is what mousetraps is working on.

Contributor

munyirik commented Aug 7, 2015

Thanks @NNM11 . The swapping procedure should only be required if you're building a native node.js addon - I'll keep you posted when we have something working. For a pure JS package (e.g. Express), you can upgrade to npmv3 and use node.js (chakra) with it. BUT this is independent of NTVS (i.e. it works when testing through command line) - the integration of npmv3 and NTVS is what mousetraps is working on.

@NNM11

This comment has been minimized.

Show comment
Hide comment
@NNM11

NNM11 Aug 13, 2015

Thanks @munyirik for the explanation.
@mousetraps - let me know if I can help with the npmv3 and NTVS (UWP) integration. Thx.

NNM11 commented Aug 13, 2015

Thanks @munyirik for the explanation.
@mousetraps - let me know if I can help with the npmv3 and NTVS (UWP) integration. Thx.

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Aug 21, 2015

Contributor

@MarkStega re: jspm - it's not full support, but check out this new VS extension
https://visualstudiogallery.msdn.microsoft.com/753b9720-1638-4f9a-ad8d-2c45a410fd74

Contributor

mousetraps commented Aug 21, 2015

@MarkStega re: jspm - it's not full support, but check out this new VS extension
https://visualstudiogallery.msdn.microsoft.com/753b9720-1638-4f9a-ad8d-2c45a410fd74

@kant2002

This comment has been minimized.

Show comment
Hide comment
@kant2002

kant2002 Sep 4, 2015

Contributor

1.1 RC2 is out with NPM3 support. That could helps with MAX_PATH issue.

Contributor

kant2002 commented Sep 4, 2015

1.1 RC2 is out with NPM3 support. That could helps with MAX_PATH issue.

@MarkStega

This comment has been minimized.

Show comment
Hide comment
@MarkStega

MarkStega Sep 4, 2015

Oops - Forgot to comment; I installed the NPM3 beta (removing the NPM2 installation) a few weeks ago. It does solve the MAX_PATH issue (at least for the Aurelia navigation project).

MarkStega commented Sep 4, 2015

Oops - Forgot to comment; I installed the NPM3 beta (removing the NPM2 installation) a few weeks ago. It does solve the MAX_PATH issue (at least for the Aurelia navigation project).

@mousetraps mousetraps modified the milestone: Future Sep 24, 2015

@linquize

This comment has been minimized.

Show comment
Hide comment
@linquize

linquize Oct 1, 2015

after flatten, restart VS, rebuild project

linquize commented Oct 1, 2015

after flatten, restart VS, rebuild project

@graphnode

This comment has been minimized.

Show comment
Hide comment
@graphnode

graphnode Oct 30, 2015

👍 for @linquize tip

After I npm install npm -g to get the latest version, deleted the project's node_modules and did npm install I was still having MAX_PATH problems even when what npm ls was showing was pretty flat.

Restarting VS solved the problem.

Edit: I also reordered external web tools paths in the options so that PATH was above the vs one. Not sure if helped but I leave it here.

graphnode commented Oct 30, 2015

👍 for @linquize tip

After I npm install npm -g to get the latest version, deleted the project's node_modules and did npm install I was still having MAX_PATH problems even when what npm ls was showing was pretty flat.

Restarting VS solved the problem.

Edit: I also reordered external web tools paths in the options so that PATH was above the vs one. Not sure if helped but I leave it here.

@AdamAndersonFalafelSoftware

This comment has been minimized.

Show comment
Hide comment
@AdamAndersonFalafelSoftware

AdamAndersonFalafelSoftware Jan 22, 2016

I simply installed the latest npm with "npm install npm -g" and restarted Visual Studio. No "scary steps" necessary. Confirmed by deleting the old node_modules folder (which requires special treatment to delete the very long paths) and using Visual Studio to install missing packages. The newly installed packages used the new flattened structure as expected for NPM 3.x.

AdamAndersonFalafelSoftware commented Jan 22, 2016

I simply installed the latest npm with "npm install npm -g" and restarted Visual Studio. No "scary steps" necessary. Confirmed by deleting the old node_modules folder (which requires special treatment to delete the very long paths) and using Visual Studio to install missing packages. The newly installed packages used the new flattened structure as expected for NPM 3.x.

@mousetraps

This comment has been minimized.

Show comment
Hide comment
@mousetraps

mousetraps Jan 23, 2016

Contributor

Feel free to continue discussing, but I'm going to go ahead and close this issue in favor of the guidance available in our new Node.js Guidelines repo, which sums up many of the tips in this thread: https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#max_path-explanation-and-workarounds

And of course, PRs are welcome 😃

Contributor

mousetraps commented Jan 23, 2016

Feel free to continue discussing, but I'm going to go ahead and close this issue in favor of the guidance available in our new Node.js Guidelines repo, which sums up many of the tips in this thread: https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#max_path-explanation-and-workarounds

And of course, PRs are welcome 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment