Add support for `npm ci` to `lerna bootstrap` #1324

casey-speer · 2018-03-14T20:43:04Z

Npm 5.7 includes a new install command, npm ci, that installs dependencies directly by scanning the package-lock.json and foregoes any additional dependency resolution. This can be much much faster than running the usual npm install, saving time on build. Lerna should have an option to use this command instead of npm install when running lerna bootstrap.

Expected Behavior

Able to use npm ci on request to install packages with lerna bootstrap.

Current Behavior

Currently bootstrap uses only the npm install command to install packages.

Possible Solution

Add a --npm-ci flag or similar to the bootstrap command that causes it to use npm ci instead of npm install when present.

Context

In a CI environment, we wish to have builds run as quickly as possible and npm ci is faster than npm install.

The text was updated successfully, but these errors were encountered:

evocateur · 2018-03-14T20:58:00Z

I would just skip lerna bootstrap completely. Use relative file: specifiers between your dependencies, npm i packages/* in the root, and call npm ci* directly. Lerna itself does exactly this.

* Caveat:
Until npm 5.8.0 becomes latest, there is a bug in npm ci that mangles local file: directory package.jsons. Installing either npm@next or cipm (and running it instead of npm ci) will workaround this problem for the time being.

casey-speer · 2018-03-15T16:35:55Z

Hmmm maybe I'm missing something about file: specifiers, but wouldn't this mean that I have to always manually include the file: dependencies of a package X from my lerna repo in any host app that wants to use X?

evocateur · 2018-03-15T16:47:01Z

Nope, the file: specifiers are overwritten with the current version range during lerna publish, so they're fully-functional npm packages that you treat like any other when consuming. (The versioned source code always preserves the file: specifiers)

Compare @lerna/cli's dependencies:

source:

  "dependencies": {
    "@lerna/add": "file:../../commands/add",
    "@lerna/bootstrap": "file:../../commands/bootstrap",
    "@lerna/changed": "file:../../commands/changed",
    "@lerna/clean": "file:../../commands/clean",
    "@lerna/diff": "file:../../commands/diff",
    "@lerna/exec": "file:../../commands/exec",
    "@lerna/global-options": "file:../global-options",
    "@lerna/import": "file:../../commands/import",
    "@lerna/init": "file:../../commands/init",
    "@lerna/link": "file:../../commands/link",
    "@lerna/list": "file:../../commands/list",
    "@lerna/publish": "file:../../commands/publish",
    "@lerna/run": "file:../../commands/run",
    "dedent": "^0.7.0",
    "is-ci": "^1.0.10",
    "npmlog": "^4.1.2",
    "yargs": "^11.0.0"
  }

npm v --json @lerna/cli dependencies:

{
  "@lerna/add": "^3.0.0-beta.3",
  "@lerna/bootstrap": "^3.0.0-beta.3",
  "@lerna/changed": "^3.0.0-beta.3",
  "@lerna/clean": "^3.0.0-beta.3",
  "@lerna/diff": "^3.0.0-beta.3",
  "@lerna/exec": "^3.0.0-beta.3",
  "@lerna/global-options": "^3.0.0-beta.3",
  "@lerna/import": "^3.0.0-beta.3",
  "@lerna/init": "^3.0.0-beta.3",
  "@lerna/link": "^3.0.0-beta.3",
  "@lerna/list": "^3.0.0-beta.3",
  "@lerna/publish": "^3.0.0-beta.3",
  "@lerna/run": "^3.0.0-beta.3",
  "dedent": "^0.7.0",
  "is-ci": "^1.0.10",
  "npmlog": "^4.1.2",
  "yargs": "^11.0.0"
}

casey-speer · 2018-03-15T17:46:30Z

Perfect I'll give this a shot, thanks!

reggi · 2018-03-16T08:31:14Z

When I use this method mainly switching to file and npm i packages/* the binaries are not located for each file, so scripts are failing :(

npm i packages/* && lerna bootstrap --hoist

This sorta fills in the cracks, but it's throwing a lot of errors.

evocateur · 2018-03-16T15:09:07Z

the binaries are not located for each file, so scripts are failing

Do you have a specific example? In most cases this is a dev script that should be moved to the root anyway. No leaf devDependencies are installed when relative file specifiers are used in the root.

reggi · 2018-03-16T17:15:02Z

@evocateur My issue with npm i packages/* is that it's not installing dev-deps, any idea how to fix that?

evocateur · 2018-03-16T17:18:34Z

Move all the devDependencies to the root package.json.

reggi · 2018-03-19T18:49:26Z

@evocateur when I run npm i packages/* then edit my package the packages that used the edited package still are referencing the version that was installed. How am I supposed to bypass this to allow the packages to be updated and have all the references to file: reflect the "live" copy of the module.

evocateur · 2018-03-19T18:52:58Z

@reggi I need an actual example to debug, I have no (possible) answer other than "works on my repo". Local file: dependencies should show up as symlinks under the root node_modules, otherwise they're being clobbered by other dependencies.

You also need all of the leaf packages to specify their sibling dependencies as relative file: specifiers; if they are caret versions (like lerna bootstrap --hoist needs) they will fail because they're installing from npm.

reggi · 2018-03-19T19:28:18Z

Is it possible that npm 3 does not symlink filemodules, while 5 does?

evocateur · 2018-03-19T19:29:00Z

Correct, this pattern is npm5 only.

jeremiahrhall · 2018-03-23T02:32:04Z

For repos with a large number of entries in lerna.json's packages configuration, this approach is complicated. Is there really no desire to make it possible to use npm ci instead of install?

evocateur · 2018-03-23T03:11:13Z

There are no PRs yet, if that’s what you mean.

…

On Mar 22, 2018, at 19:32, Jeremiah Hall ***@***.***> wrote: For repos with a large number of entries in lerna.json's packages configuration, this approach is complicated. Is there really no desire to make it possible to use npm ci instead of install? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

…nged to `npm ci` once this is fixed lerna/lerna#1324 .

* develop: (68 commits) Added support for absolute paths. update develop config Added Error boundary component which should catch all render errors and uncaught throws. Fixes #409 - Faucet link now opens in new window/tab Some style tweaks Commented out the warning header Fix issue where user profile showed all milestones and marked reviewer incorrectly Donations, campaign and DAC required transactions Campaign managers can now accept proposed milestones. Fix eslint issues. Fixed travis by switching back to npm. The install step should be changed to `npm ci` once this is fixed lerna/lerna#1324 . Added required confirmation count Update EditDAC.jsx update bridge addy add beta configuration Add validations for templates Initial changes to views for campaign, Donations Use default quill dropdown fix feathers endpoint in release config donations and roles on profile page ... # Conflicts: # src/index.js # src/services/DonationService.js

RobertWSaunders · 2018-09-17T16:47:04Z

I think this can be closed now that #1360 added it?

evocateur · 2018-09-17T16:58:09Z

@RobertWSaunders Yes, thanks!

RobertWSaunders · 2018-09-17T17:15:37Z

@evocateur Can you think of any reason why running lerna bootstrap --ci locally works fine but when running the same command in CircleCI it's failing. I get this error:

npm install --unsafe-perm || ((if [ -f npm-debug.log ]; then cat npm-debug.log; fi) && false) && lerna bootstrap --ci


> husky@1.0.0-rc.14 install /home/circleci/project/node_modules/husky
> node husky install

husky > setting up git hooks
CI detected, skipping Git hooks installation.
added 739 packages from 427 contributors and audited 37313 packages in 21.674s
found 0 vulnerabilities

lerna notice cli v3.4.0
lerna info Bootstrapping 2 packages
lerna info Installing external dependencies
lerna info hoist Installing hoisted dependencies into root
lerna info hoist Pruning hoisted dependencies
lerna info hoist Finished pruning hoisted dependencies
lerna ERR! npm ci exited 1 in 'qhacks-dashboard'
lerna ERR! npm ci stderr:
npm WARN prepare removing existing node_modules/ before installation
npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! 
npm ERR! Missing: @sendgrid/mail@^6.3.1
npm ERR! Missing: axios@^0.18.0
npm ERR! Missing: babel-core@^6.26.0
npm ERR! Missing: babel-jest@^23.2.0
npm ERR! Missing: babel-loader@^7.1.2
npm ERR! Missing: babel-polyfill@^6.26.0
npm ERR! Missing: babel-preset-env@^1.6.1
npm ERR! Missing: babel-preset-react@^6.24.1
npm ERR! Missing: babel-preset-stage-2@^6.24.1
npm ERR! Missing: bcrypt@^3.0.0
npm ERR! Missing: body-parser@^1.18.3
npm ERR! Missing: camelcase@^5.0.0
npm ERR! Missing: compression@^1.7.3
npm ERR! Missing: compression-webpack-plugin@^1.1.3
npm ERR! Missing: connect-history-api-fallback@^1.5.0
npm ERR! Missing: css-loader@^0.28.9
npm ERR! Missing: dotenv@^6.0.0
npm ERR! Missing: enzyme@^3.3.0
npm ERR! Missing: enzyme-adapter-react-16@^1.1.1
npm ERR! Missing: enzyme-to-json@^3.3.4
npm ERR! Missing: express@^4.16.3
npm ERR! Missing: favicons-webpack-plugin@0.0.9
npm ERR! Missing: file-loader@^1.1.6
npm ERR! Missing: helmet@^3.13.0
npm ERR! Missing: html-webpack-plugin@^3.2.0
npm ERR! Missing: jest@^23.6.0
npm ERR! Missing: jest-environment-node@^23.4.0
npm ERR! Missing: jsonwebtoken@^8.3.0
npm ERR! Missing: less@2.7.3
npm ERR! Missing: less-loader@^4.0.5
npm ERR! Missing: mini-css-extract-plugin@^0.4.1
npm ERR! Missing: mjml@^4.1.2
npm ERR! Missing: moment@^2.22.2
npm ERR! Missing: mongodb-memory-server@^2.2.0
npm ERR! Missing: mongoose@^5.2.14
npm ERR! Missing: mustache@^2.3.2
npm ERR! Missing: nodemon@^1.18.4
npm ERR! Missing: querystring@^0.2.0
npm ERR! Missing: react@^16.2.0
npm ERR! Missing: react-dom@^16.2.0
npm ERR! Missing: react-emoji-render@^0.4.4
npm ERR! Missing: react-redux@^5.0.6
npm ERR! Missing: react-router-dom@^4.2.2
npm ERR! Missing: redux@^4.0.0
npm ERR! Missing: redux-form@^7.2.1
npm ERR! Missing: redux-form-validators@^2.3.2
npm ERR! Missing: redux-saga@^0.16.0
npm ERR! Missing: style-loader@^0.21.0
npm ERR! Missing: supertest@^3.3.0
npm ERR! Missing: url-loader@^1.0.1
npm ERR! Missing: validator@^10.7.1
npm ERR! Missing: webpack@^4.14.0
npm ERR! Missing: webpack-cli@^3.0.8
npm ERR! Missing: webpack-dev-server@^3.1.4
npm ERR! Missing: winston@^3.1.0
npm ERR!

As you can see, the command being run is npm install --unsafe-perm || ((if [ -f npm-debug.log ]; then cat npm-debug.log; fi) && false) && lerna bootstrap --ci. Only reason I am calling npm install before lerna bootstrap is because lerna bootstrap requires lerna to be installed. Is that logic wrong?

evocateur · 2018-09-17T17:22:18Z

No, but you can't --hoist if you're running npm ci, it's trying to install a bunch of hoisted dependencies in the root first. Not sure why you need the --unsafe-perm?

RobertWSaunders · 2018-09-17T19:10:02Z

Thanks @evocateur! I had tried without hoisting but taking a second look I was able to get it. Yeah I got rid of --unsafe-perm, thanks for the tip, don't know how that got there.

vidartf · 2018-10-12T15:52:57Z

If you cannot use ci with --hoist, then shouldn't --hoist imply --no-ci ?

Signed-off-by: André Jaenisch <andre.jaenisch@posteo.de>

lock · 2018-12-27T02:23:17Z

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.

mndewitt mentioned this issue Apr 4, 2018

feat(bootstrap): Use npm ci with --ci option #1360

Merged

9 tasks

vojtechsimetka added a commit to Giveth/giveth-dapp that referenced this issue Jul 2, 2018

Fixed travis by switching back to npm. The install step should be cha…

0ff4fbf

…nged to `npm ci` once this is fixed lerna/lerna#1324 .

ErikBjare mentioned this issue Sep 5, 2018

Added use of npm ci in build-production SuperuserLabs/thankful#170

Merged

evocateur closed this as completed Sep 17, 2018

Ryuno-Ki added a commit to lobsangnet/lobsang.js that referenced this issue Nov 4, 2018

build: drop hoist flag. kudos lerna/lerna#1324~422099684. Refs #6

0e3b1cf

Ryuno-Ki added a commit to lobsangnet/lobsang.js that referenced this issue Nov 4, 2018

build: drop hoist flag. kudos lerna/lerna#1324~422099684. Refs #6

307ab4c

Signed-off-by: André Jaenisch <andre.jaenisch@posteo.de>

joaquimsantmrf mentioned this issue Nov 5, 2018

fix lerna issue. DevDependencies in root Marfeel/MarfeelCodingStandards#60

Merged

BeniCheni mentioned this issue Nov 8, 2018

Travis "Flaky" Failure On CI Instance priceline/design-system#411

Closed

lock bot locked as resolved and limited conversation to collaborators Dec 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `npm ci` to `lerna bootstrap` #1324

Add support for `npm ci` to `lerna bootstrap` #1324

casey-speer commented Mar 14, 2018 •

edited

Loading

evocateur commented Mar 14, 2018

casey-speer commented Mar 15, 2018

evocateur commented Mar 15, 2018

casey-speer commented Mar 15, 2018

reggi commented Mar 16, 2018 •

edited

Loading

evocateur commented Mar 16, 2018 via email

reggi commented Mar 16, 2018

evocateur commented Mar 16, 2018

reggi commented Mar 19, 2018

evocateur commented Mar 19, 2018

reggi commented Mar 19, 2018

evocateur commented Mar 19, 2018

jeremiahrhall commented Mar 23, 2018

evocateur commented Mar 23, 2018 via email

RobertWSaunders commented Sep 17, 2018

evocateur commented Sep 17, 2018

RobertWSaunders commented Sep 17, 2018

evocateur commented Sep 17, 2018

RobertWSaunders commented Sep 17, 2018

vidartf commented Oct 12, 2018

lock bot commented Dec 27, 2018

Add support for npm ci to lerna bootstrap #1324

Add support for npm ci to lerna bootstrap #1324

Comments

casey-speer commented Mar 14, 2018 • edited Loading

Expected Behavior

Current Behavior

Possible Solution

Context

evocateur commented Mar 14, 2018

casey-speer commented Mar 15, 2018

evocateur commented Mar 15, 2018

casey-speer commented Mar 15, 2018

reggi commented Mar 16, 2018 • edited Loading

evocateur commented Mar 16, 2018 via email

reggi commented Mar 16, 2018

evocateur commented Mar 16, 2018

reggi commented Mar 19, 2018

evocateur commented Mar 19, 2018

reggi commented Mar 19, 2018

evocateur commented Mar 19, 2018

jeremiahrhall commented Mar 23, 2018

evocateur commented Mar 23, 2018 via email

RobertWSaunders commented Sep 17, 2018

evocateur commented Sep 17, 2018

RobertWSaunders commented Sep 17, 2018

evocateur commented Sep 17, 2018

RobertWSaunders commented Sep 17, 2018

vidartf commented Oct 12, 2018

lock bot commented Dec 27, 2018

Add support for `npm ci` to `lerna bootstrap` #1324

Add support for `npm ci` to `lerna bootstrap` #1324

casey-speer commented Mar 14, 2018 •

edited

Loading

reggi commented Mar 16, 2018 •

edited

Loading