npm-folders(1) -- Folder Structures Used by npm
This functionality is not yet implemented. It is a plan, not reality. It is not the map, nor the territory, but a blueprint with blank areas.
In particular, this is the scheme that will be used starting in firstname.lastname@example.org, which will require email@example.com or above.
Don't splat stuff across the filesystem so much. Just specify a single root location, and be done with it.
Work on windows.
Minimize shim/symlink usage.
Do not rely on any global system path for node modules.
Remove the "activation" concept.
npm metadata lives in the
root setting. There is the cache folder,
and the contents of all installed packages.
The default npm root folder is
TODO: What should the root folder be on windows?
The cache folder is a mirror of the data in the registry, as well as a working space for unpacking and creating tarballs.
Files and folders created in the cache are owned by the executing user, often "root".
Files are created with 0666 and folders with 0777, so that they can be modified by any user.
root/cache/foo/cache.jsonExpirable cache of registry/foo json data
root/cache/foo/1.2.3/packagePristine copy of foo package contents
root/cache/foo/1.2.3/package.tgztarball of firstname.lastname@example.org
In the npm root folder, package contents are unpacked, built, and then moved into the desired location.
root/packages/foo/1.2.3Metadata and contents of email@example.com
root/packages/foo/1.2.3/packageBuild location of firstname.lastname@example.org
root/packages/foo/1.2.3/node_modulesLinks to (or, on windows, copies of) dependencies of email@example.com
root/packages/foo/1.2.3/metadata.jsonMetadata about the foo package.
When you run
npm install firstname.lastname@example.org it downloads and builds the
package, and then, if there is a package.json file in the current
working directory, it copies it to
$PWD/node_modules/foo, so that your
current package will get it when you do
When this is done, it also installs all of foo's dependencies to
./node_modules/foo/node_modules/, so that it will get its dependencies
appropriately when it calls
require(). If foo depends on bar, and bar
depends on baz, then there will also be a
./node_modules/foo/node_modules/bar/node_modules/baz, and so on.
If there is not a package.json in the current working directory, then
npm walks up the working dir parent paths looking for a package.json,
indicating the root of a package, or a node_modules folder,
indicating an npm package deployment location, and then take the party to that
location. This behavior may be suppressed by setting the
config value to false.
If no package root is found, then a global installation is performed.
The global installation may be supressed by setting the
configuration to false, in which case, the install will fail.
global configuration is set to true, or if it is not explicitly
set false and no suitable node_modules folder was found, then npm will
install packages "globally".
This means that the module contents are symlinked (or, on windows,
When installing globally, executables are linked (or, on windows,
shimmed with a .bat file) to
When doing a
localized installation, executables are linked (or, on windows, shimmed)
./node_modules/.bin. This also applies to the case when a globally
installed package's dependents are being installed into it. Basically,
.../node_modules/foo, and the "foo"
package has an executable named "bar", it'll write it to
It is up to the user to update their PATH environment variable
appropriately for globally installed executables. When running package
lifecycle scripts (for example, to build, start, test, etc.), npm will
./node_modules/.bin as the first item in the PATH environ.
npm will install man pages to
root/share/man. It is up to the user to
make sure that their man program searches this location.
Cycles, Conflicts, and Folder Parsimony
Cycles are handled using the property of node's module system that it walks up the directories looking for node_modules folders. So, at every stage, if a package is already installed in an ancestor node_modules folder, then it is not installed at the current location.
Consider the case above, where
foo -> bar -> baz. Imagine if, in
addition to that, baz depended on bar, so you'd have:
foo -> bar -> baz -> bar -> baz .... However, since the folder
structure is: foo/node_modules/bar/node_modules/baz, there's no need to
put another copy of bar into .../baz/node_modules, since when it calls
require("bar"), it will get the copy that is installed in
This shortcut is only used if the exact same
version would be installed in multiple nested node_modules folders. It
is still possible to have
a/node_modules/b/node_modules/a if the two
"a" packages are different versions. However, without repeating the
exact same package multiple times, an infinite regress will always be
Another optimization can be made by installing dependencies at the highest level possible, below the localized "target" folder.
For example, consider this dependency graph:
foo +-- email@example.com | +-- firstname.lastname@example.org | | `-- email@example.com | `-- asdf@* `-- firstname.lastname@example.org `-- email@example.com
In this case, we'd expect a folder structure like this:
foo +-- node_modules +-- bar (1.2.3) | `-- node_modules | `-- baz (2.0.2) +-- quux (3.2.0) +-- asdf (0.2.5) `-- baz (1.2.3)
Since foo depends directly on firstname.lastname@example.org and email@example.com, those are installed in foo's node_modules folder.
Since firstname.lastname@example.org depends on email@example.com, a satisfying version is placed in foo's node_modules folder, because there are no conflicts.
Since firstname.lastname@example.org depends on asdf@*, a satisfying version is placed in foo's node_modules folder. It also depends on email@example.com, but this conflicts with the version already installed in foo's node_modules folder, so it is installed into the node_modules folder under firstname.lastname@example.org.
email@example.com depends on firstname.lastname@example.org, but this dependency is already satisfied by the quux version installed in foo's node_modules folder, so nothing further needs to be done.
npm snapshot command is run, the package.json file is
updated to include the versions of all of the packages in the
./node_modules folder as dependencies.
Upon publishing, npm will look in the node_modules folder. If any of
the items there are on the "dependencies" or "devDependencies" list, and
are unmodified copies of the corresponding packages in
root/node_modules/<name>/<version>/package, then they will not be
included in the package tarball.
If the package has been modified, then it is left as-is, and included in the package.
This allows a package maintainer to install all of their dependencies (and dev dependencies) locally, but only publish those items that cannot be found elsewhere.
npm keeps track of every installation of email@example.com in
When updating in a package folder (see algorithm for determining this in
node_modules Folders"), npm updates the packages in
the local folder to the latest versions that are compatible with the
requirements in the package.json file.
If global is set to
"super", then npm will attempt to update
all copies of packages installed anywhere and everywhere that it is
global is set to `true, or implied by not being in a package folder
at the time, then npm will update the globally installed packages.
If you do
npm update foo, and you're in a local package folder, but
foo is not installed there, and it is installed globally, then that
will be equivalent to doing
npm update foo --global.
Uninstalling works like updating.
If global is set to "super", then it removes all traces from everywhere.
If global is set to "true", or implied, then it removes the global copy.
If global is set to "false", or unset and the command is in a package folder with a copy of the package being removed, then it removes it from the local folder.
npm link is run without any arguments in a package folder, then a
symbolic link is created from that folder to
Additionally, its package dependencies are installed to its
./node_modules folder as necessary.
npm link <name> is run in another package folder, a symbolic
link is created from
its dependencies are also installed if necessary. (Generally, it will
not be necessary, as the package will already have its own node_modules
folder containing its dependencies.)
When publishing, linked package dependencies are resolved to their current state. It is assumed that the linked folder was linked for a reason, and may contain changes required for the proper functioning of the host program.
npm link use case
# create a linked "request" package cd ~/projects/request npm link # now write a program that uses request mkdir -p ~/projects/my-program cd ~/projects/my-program git init # do your git stuff... npm init # enter some package.json values # now we're ready to rock. # use redis, but don't need bleeding edge. npm install redis # use the linked copy of request npm link request # now any changes to ~/projects/request will # be immediately effective in my-program when # I do require("request")
link on Windows
Not sure how this will work. Maybe linking simply isn't possible?