WIP multi stage install

[iarna]

• Load existing tree with read-package-tree
• Isolate and extract package metadata fetching
• Recursively determine dependencies
  • From package.json
  • From arguments
  • From developer dependencies
• Account for shrinkwrap in dependency loader
  • Replace deps at the top level
  • Include similar processing for children too
• (partially complete) Dedup tree
  • (partially complete) Implement the current strategy, merging back to the lowest common ancestor
    • A version of this exists, but suffers from inexplicable differences from run to run, despite containing no async code. PLUS, adding accesses to certain objects seem to make the differences go away. I put this aside while I work further down the list and'll come back to it with a clear head later.
  • Implement a "as flat as possible" strategy where we merge back to the root when possible
• Validate the tree
  • Existing tree validations
  • Peer depenedencies
• Compare the trees and generate a list of high level actions (ie, install, update, remove)
• Optimize the action list (that is, fold remove+install into update or move as appropriate)
• Decompose the action list into the lowlevel actions (untar, extract, gyp, lifecycle scripts, etc)
• Sort the action list
• Possibly further optimize the action list
• Execute the actions
  • add locking to these where appropriate
  • fetch
  • extract
  • preinstall
  • build
  • link binaries
  • install (the install lifecycle)
  • postinstall
  • test (w/ npat)
  • remove
  • finalize (move from staging to final location)
• Update top-level package.json and shrinkwrap when --save is used.
• Emit wrapped error messages like the old npm-install
• Load bundled dependenci
• Add progress bar to replace spinner
  • Integrate progress bar code into npm-log
  • Make progress bar prettier
  • Integrate progress bar into multi-stage install
  • Deeper integration of progress bar into other related tasks, eg, cache, request, rimraf, etc.
• Implement linked dependency semantics
• Update this here list to reference the tickets the various bits are addressing
• Make an external tool to prototype submitting ideal trees to nodesecurity for validation. (Consider as a prototype for potential inclusion as an npm command.) http://blog.nodesecurity.io/2014/02/01/new-feature-validate-modules-with-npm-shrinkwrap
  This will ultimately fix multi-stage install #5919 et al.

Builds now occur in node_modules/.staging and modules are only moved to their respective locations at the very end of the process.

See also the design doc on install semantics.

[iarna]

Closing this as its tasks have been split into issues and we won't be landing it via pull request anyway.

[Fishrock123]

Is there a good way to track the progress of this?

[prettycode]

I'm also looking for the progress. I'm on Windows and running into the "path too long" problems of nested node_module dependencies. My understanding was that dedup in its current form does not solve this problem and that "multi-stage-install" was going to solve this. Where are we at now with this?

[Fishrock123]

Looks like it's now on this branch: https://github.com/npm/npm/commits/multi-stage

[othiym23]

@Fishrock123 @prettycode Yeah, sorry for not responding earlier. There's a multi-stage branch which represents the current state of the code, and sometime in the next few days @iarna is going to refresh the multi-stage install milestone to look less like a wish list and more like a snapshot of the current issues with the code base. The good news is that this is a pretty thorough revision of the npm installation process, and is pretty much entirely superior to what we had before. The bad news is that it's a pretty thorough revision of the npm installation process and we keep finding new things that need thorough reworking to work with the new code.

[orangemocha]

Any updates on when this work will make it into master / an official release?