This repository has been archived by the owner. It is now read-only.

WIP multi stage install #6191

Closed
wants to merge 14 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@iarna
Member

iarna commented Sep 15, 2014

  • Load existing tree with read-package-tree
  • Isolate and extract package metadata fetching
  • Recursively determine dependencies
    • From package.json
    • From arguments
    • From developer dependencies
  • Account for shrinkwrap in dependency loader
    • Replace deps at the top level
    • Include similar processing for children too
  • (partially complete) Dedup tree
    • (partially complete) Implement the current strategy, merging back to the lowest common ancestor
      • A version of this exists, but suffers from inexplicable differences from run to run, despite containing no async code. PLUS, adding accesses to certain objects seem to make the differences go away. I put this aside while I work further down the list and'll come back to it with a clear head later.
    • Implement a "as flat as possible" strategy where we merge back to the root when possible
  • Validate the tree
    • Existing tree validations
    • Peer depenedencies
  • Compare the trees and generate a list of high level actions (ie, install, update, remove)
  • Optimize the action list (that is, fold remove+install into update or move as appropriate)
  • Decompose the action list into the lowlevel actions (untar, extract, gyp, lifecycle scripts, etc)
  • Sort the action list
  • Possibly further optimize the action list
  • Execute the actions
    • add locking to these where appropriate
    • fetch
    • extract
    • preinstall
    • build
    • link binaries
    • install (the install lifecycle)
    • postinstall
    • test (w/ npat)
    • remove
    • finalize (move from staging to final location)
  • Update top-level package.json and shrinkwrap when --save is used.
  • Emit wrapped error messages like the old npm-install
  • Load bundled dependenci
  • Add progress bar to replace spinner
    • Integrate progress bar code into npm-log
    • Make progress bar prettier
    • Integrate progress bar into multi-stage install
    • Deeper integration of progress bar into other related tasks, eg, cache, request, rimraf, etc.
  • Implement linked dependency semantics
  • Update this here list to reference the tickets the various bits are addressing
  • Make an external tool to prototype submitting ideal trees to nodesecurity for validation. (Consider as a prototype for potential inclusion as an npm command.) http://blog.nodesecurity.io/2014/02/01/new-feature-validate-modules-with-npm-shrinkwrap
    This will ultimately fix #5919 et al.

Builds now occur in node_modules/.staging and modules are only moved to their respective locations at the very end of the process.

See also the design doc on install semantics.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Dec 15, 2014

Member

Closing this as its tasks have been split into issues and we won't be landing it via pull request anyway.

Member

iarna commented Dec 15, 2014

Closing this as its tasks have been split into issues and we won't be landing it via pull request anyway.

@iarna iarna closed this Dec 15, 2014

@iarna iarna removed the in-progress label Dec 15, 2014

@Fishrock123

This comment has been minimized.

Show comment
Hide comment
@Fishrock123

Fishrock123 Jan 7, 2015

Contributor

Is there a good way to track the progress of this?

Contributor

Fishrock123 commented Jan 7, 2015

Is there a good way to track the progress of this?

@prettycode

This comment has been minimized.

Show comment
Hide comment
@prettycode

prettycode Mar 5, 2015

I'm also looking for the progress. I'm on Windows and running into the "path too long" problems of nested node_module dependencies. My understanding was that dedup in its current form does not solve this problem and that "multi-stage-install" was going to solve this. Where are we at now with this?

prettycode commented Mar 5, 2015

I'm also looking for the progress. I'm on Windows and running into the "path too long" problems of nested node_module dependencies. My understanding was that dedup in its current form does not solve this problem and that "multi-stage-install" was going to solve this. Where are we at now with this?

@Fishrock123

This comment has been minimized.

Show comment
Hide comment
@Fishrock123

Fishrock123 Mar 20, 2015

Contributor

Looks like it's now on this branch: https://github.com/npm/npm/commits/multi-stage

Contributor

Fishrock123 commented Mar 20, 2015

Looks like it's now on this branch: https://github.com/npm/npm/commits/multi-stage

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Mar 21, 2015

Contributor

@Fishrock123 @prettycode Yeah, sorry for not responding earlier. There's a multi-stage branch which represents the current state of the code, and sometime in the next few days @iarna is going to refresh the multi-stage install milestone to look less like a wish list and more like a snapshot of the current issues with the code base. The good news is that this is a pretty thorough revision of the npm installation process, and is pretty much entirely superior to what we had before. The bad news is that it's a pretty thorough revision of the npm installation process and we keep finding new things that need thorough reworking to work with the new code.

Contributor

othiym23 commented Mar 21, 2015

@Fishrock123 @prettycode Yeah, sorry for not responding earlier. There's a multi-stage branch which represents the current state of the code, and sometime in the next few days @iarna is going to refresh the multi-stage install milestone to look less like a wish list and more like a snapshot of the current issues with the code base. The good news is that this is a pretty thorough revision of the npm installation process, and is pretty much entirely superior to what we had before. The bad news is that it's a pretty thorough revision of the npm installation process and we keep finding new things that need thorough reworking to work with the new code.

@orangemocha

This comment has been minimized.

Show comment
Hide comment
@orangemocha

orangemocha Jun 4, 2015

Contributor

Any updates on when this work will make it into master / an official release?

Contributor

orangemocha commented Jun 4, 2015

Any updates on when this work will make it into master / an official release?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.