Skip to content
This repository has been archived by the owner. It is now read-only.

WIP multi stage install #6191

Closed
wants to merge 14 commits into from
Closed

WIP multi stage install #6191

wants to merge 14 commits into from

Conversation

@iarna
Copy link
Contributor

@iarna iarna commented Sep 15, 2014

  • Load existing tree with read-package-tree
  • Isolate and extract package metadata fetching
  • Recursively determine dependencies
    • From package.json
    • From arguments
    • From developer dependencies
  • Account for shrinkwrap in dependency loader
    • Replace deps at the top level
    • Include similar processing for children too
  • (partially complete) Dedup tree
    • (partially complete) Implement the current strategy, merging back to the lowest common ancestor
      • A version of this exists, but suffers from inexplicable differences from run to run, despite containing no async code. PLUS, adding accesses to certain objects seem to make the differences go away. I put this aside while I work further down the list and'll come back to it with a clear head later.
    • Implement a "as flat as possible" strategy where we merge back to the root when possible
  • Validate the tree
    • Existing tree validations
    • Peer depenedencies
  • Compare the trees and generate a list of high level actions (ie, install, update, remove)
  • Optimize the action list (that is, fold remove+install into update or move as appropriate)
  • Decompose the action list into the lowlevel actions (untar, extract, gyp, lifecycle scripts, etc)
  • Sort the action list
  • Possibly further optimize the action list
  • Execute the actions
    • add locking to these where appropriate
    • fetch
    • extract
    • preinstall
    • build
    • link binaries
    • install (the install lifecycle)
    • postinstall
    • test (w/ npat)
    • remove
    • finalize (move from staging to final location)
  • Update top-level package.json and shrinkwrap when --save is used.
  • Emit wrapped error messages like the old npm-install
  • Load bundled dependenci
  • Add progress bar to replace spinner
    • Integrate progress bar code into npm-log
    • Make progress bar prettier
    • Integrate progress bar into multi-stage install
    • Deeper integration of progress bar into other related tasks, eg, cache, request, rimraf, etc.
  • Implement linked dependency semantics
  • Update this here list to reference the tickets the various bits are addressing
  • Make an external tool to prototype submitting ideal trees to nodesecurity for validation. (Consider as a prototype for potential inclusion as an npm command.) http://blog.nodesecurity.io/2014/02/01/new-feature-validate-modules-with-npm-shrinkwrap
    This will ultimately fix #5919 et al.

Builds now occur in node_modules/.staging and modules are only moved to their respective locations at the very end of the process.

See also the design doc on install semantics.

@iarna iarna added the in-progress label Sep 15, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 4 times, most recently Sep 15, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 5 times, most recently Oct 1, 2014
@iarna iarna force-pushed the iarna:multi-stage branch Oct 8, 2014
@iarna iarna added this to the multi-stage install milestone Oct 13, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 2 times, most recently Oct 17, 2014
@iarna iarna force-pushed the iarna:multi-stage branch Oct 20, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 8 times, most recently Oct 22, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 3 times, most recently Oct 28, 2014
@iarna iarna force-pushed the iarna:multi-stage branch Nov 6, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 2 times, most recently Nov 24, 2014
@iarna iarna force-pushed the iarna:multi-stage branch 4 times, most recently Dec 2, 2014
@iarna iarna force-pushed the iarna:multi-stage branch to a8d4306 Dec 12, 2014
@iarna
Copy link
Contributor Author

@iarna iarna commented Dec 15, 2014

Closing this as its tasks have been split into issues and we won't be landing it via pull request anyway.

@iarna iarna closed this Dec 15, 2014
@iarna iarna removed the in-progress label Dec 15, 2014
@Fishrock123
Copy link
Contributor

@Fishrock123 Fishrock123 commented Jan 7, 2015

Is there a good way to track the progress of this?

@prettycode
Copy link

@prettycode prettycode commented Mar 5, 2015

I'm also looking for the progress. I'm on Windows and running into the "path too long" problems of nested node_module dependencies. My understanding was that dedup in its current form does not solve this problem and that "multi-stage-install" was going to solve this. Where are we at now with this?

@Fishrock123
Copy link
Contributor

@Fishrock123 Fishrock123 commented Mar 20, 2015

Looks like it's now on this branch: https://github.com/npm/npm/commits/multi-stage

@othiym23
Copy link
Contributor

@othiym23 othiym23 commented Mar 21, 2015

@Fishrock123 @prettycode Yeah, sorry for not responding earlier. There's a multi-stage branch which represents the current state of the code, and sometime in the next few days @iarna is going to refresh the multi-stage install milestone to look less like a wish list and more like a snapshot of the current issues with the code base. The good news is that this is a pretty thorough revision of the npm installation process, and is pretty much entirely superior to what we had before. The bad news is that it's a pretty thorough revision of the npm installation process and we keep finding new things that need thorough reworking to work with the new code.

@orangemocha
Copy link
Contributor

@orangemocha orangemocha commented Jun 4, 2015

Any updates on when this work will make it into master / an official release?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.