Hoisting seems broken with npm5 #856

jquense · 2017-05-30T17:55:18Z

Expected Behavior

lerna bootstrap --hoist should hoist dependencies when using npm@5

Current Behavior

The correct package.json is generated in the root directory but none of the hoisted deps are installed when a package-lock.json is present....which as far as I can tell (thanks npm docs!) can't be toggled off.

lerna.json

{
  "lerna": "2.0.0-rc.5",
  "version": "4.0.0-rc.6",
  "hoist": true,
  "packages": [
    "packages/*"
  ]
}

Your Environment

Executable	Version
`lerna --version`	2.0.0-rc.5
`npm --version`	5.0.0

OS	Version
macOS	?

The text was updated successfully, but these errors were encountered:

hzoo · 2017-05-30T17:59:48Z

Yeah it's definetely not handled yet (nothing on npm 5 is supported since we haven't tested)

evocateur · 2017-05-30T17:59:55Z

This is not surprising, but definitely needs to be addressed. Hopefully we can delegate most/all of the bootstrapping logic to npm@5.

jquense · 2017-05-30T18:02:14Z

might be worth making a note of somewhere, node8 is coming out today and ships with (or soon will ship with npm5) Don't want ya'll overrun by folks with the same issue :)

tivac · 2017-05-30T18:56:23Z

#851 is another npm@5 issue similar to this one.

Lerna doesn't really support npm@5 yet, see lerna/lerna#856

garthk · 2017-06-05T02:08:04Z

Workaround to force npm 4.6 despite a global npm 5, at least when running lerna in the root of your monorepo:

npm install --save-dev npm@4.6.1
set npmClient to node_modules/.bin/npm in your lerna.json

rgbkrk · 2017-06-06T22:46:25Z

Workaround to force npm 4.6 despite a global npm 5

This works well on macOS and Linux, does not work well on Windows as the npm bin references the npm-cli used by *nix systems:


Error: Cannot find module 'C:\projects\nteract\node_modules\.bin\node_modules\npm\bin\npm-cli.js'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Function.Module.runMain (module.js:604:10)
    at Function.runMain (C:\Users\appveyor\.node-spawn-wrap-952-2fdd5baf427b\node:40:10)
    at Object.<anonymous> (C:\projects\nteract\node_modules\nyc\bin\wrap.js:23:4)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

We only noticed that via our AppVeyor setup.

evocateur · 2017-06-08T16:19:07Z

Notably, if you "manually" hoist all the managed dependencies into the root (thus ensuring lerna bootstrap is mostly a no-op, aside from symlinking), it works with npm@5. Not ideal, but it's a more robust cross-platform workaround.

MoOx · 2017-06-10T22:09:21Z

@evocateur what are you calling "hoisting manually"?

evocateur · 2017-06-11T00:59:50Z

All dependencies in ./packages/*/package.json are literally added to ./package.json Thus, when bootstrapping, everything already exists in the root and only symlinking is necessary.

…

On Jun 10, 2017, at 15:09, Maxime Thirouin ***@***.***> wrote: @evocateur what are you calling "hoisting manually"? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rgbkrk · 2017-06-11T23:44:38Z

What would be an ideal way of combining all the dependencies of the subpackages to add to the main package?

blink1073 · 2017-06-13T18:22:27Z

@rgbkrk, something like this should work (adapted from a JuypterLab script):

var path = require('path');
var glob = require('glob');

/**
 * Handle an individual package on the path - get its dependencies.
 */
function handlePackage(packagePath, data) {
  // Read in the package.json.
  var packagePath = path.join(packagePath, 'package.json');
  try {
    var package = require(packagePath);
  } catch (e) {
    console.log('Skipping package ' + packagePath);
    return;
  }

  var deps = package.dependencies || [];
  for (let dep in deps) {
    data.dependencies[dep] = deps[dep];
  }
}

var data = require('./package.json');

// Handle all of the packages.
var basePath = path.resolve('.');
var files = glob.sync(path.join(basePath, 'packages/*'));
for (var j = 0; j < files.length; j++) {
  handlePackage(files[j], data);
}
console.log(JSON.stringify(data.dependencies, null, 2));

blink1073 · 2017-06-13T18:43:54Z

Follow up: I just tried the above for JupyterLab here, and not all of the packages are ending up in the top level node_modules after hoisting, meaning we still can't use npm@5.

* Use node 6 instead of node 8 / npm 5 because of lerna/lerna#856 * Use urijs instead of the node 7+ URI package. * Don't ask lerna to hoist all dependencies. This makes our docker images smaller because we don't need to put the union of all dependencies in every image. tsmonad still needs to be hoisted to work around microsoft/TypeScript#6496. * Remove yarn.lock and package-lock.json files.

Tallyb · 2017-07-18T16:07:08Z

package-lock.json can be disabled using npm config set package-lock false
You probably also want to add in npm config set save false

blink1073 · 2017-07-18T16:21:43Z

Confirmed that npm config set package-lock false is enough to get us going for now, thanks @Tallyb!

garthk · 2017-07-24T00:38:17Z

Anyone figured how to set that locally?

tivac · 2017-07-24T15:24:15Z

It seems like you could add a .npmrc file to the root of your lerna install, and then set

package-lock = false

in it? Can't test it atm but seems plausible.

blink1073 · 2017-07-24T15:32:45Z

That should work, package-lock=false is what ends up in my ~/.npmrc with the npm config command.

tivac · 2017-07-24T15:37:20Z

It does work, though you need a .npmrc file for the root and every child package. A little annoying, but better than not being able to use bootstrap --hoist.

redonkulus · 2018-03-24T00:09:49Z

@jquense @evocateur is there a minimally reproduce-able example we can provide npm so they can debug this issue?

evocateur · 2018-03-24T00:18:10Z

I just use relative file: specifiers for sibling dependencies now, don't even use lerna bootstrap anymore. Lerna's own source code demonstrates the usage. npm i packages/* and you're done.

tivac · 2018-03-24T01:59:10Z

Or lerna exec -- npm i for cross platform support.

Should we switch to recommending that by default? I'm biased but think so.

evocateur · 2018-03-24T02:02:22Z

@tivac Since lerna itself has no problem at all using npm ci with relative file specifiers in its windows builds, I'm inclined to believe that your recent issues migrating modular-css are due to dependencies misbehaving, not cross-platform incompatibility (that would be an issue for npm to fix).

joebowbeer · 2018-08-19T22:37:49Z

@tivac Is lerna exec -- npm i really equivalent to npm i packages/*?

evocateur · 2018-08-20T03:59:27Z

No, it isn't. Again, it's more than likely a symptom of misbehaving dependencies, not Windows platform-ness. Relative file: specifiers in the root results in basically the same symlink structure as yarn workspaces.

tivac · 2018-08-20T05:57:27Z

Yup, sorry everybody, I totally misunderstood the purpose of npm i packages/*. It isn't to run npm i within each of those directories, it's to add them as a file dependency there.

I ran npm i packages\<package> manually for each one of my packages. It added new file dependencies at the root level for each of the child packages. I also previously had manually hoisted all my devDependencies to the root.

But now what? Should npm i from the root install the dependencies of those file: packages?

evocateur · 2018-08-20T15:47:51Z

Should npm i from the root install the dependencies of those file: packages?

Yep! The whole tree becomes serialized in the package-lock.json, including relative file: siblings. It's pretty sweet.

tivac · 2018-08-20T16:01:47Z

I was running into some trouble with that workflow last night but think I finally got things cleaned up and functioning more cleanly!

Thanks for your patience as I stumbled through this process.

joebowbeer · 2018-08-27T01:56:55Z

@evocateur please excuse the follow-ons; I am hoping you can clarify your comment above regarding file: -- I just came across npm/pull/15900 which claims link: is the new file:

BTW it would be great to update the lerna doc to include a description of this golden path free of bootstrap.

evocateur · 2018-08-27T02:09:43Z

Yeah, that PR eventually went back to file: without updating the summary. If you run lerna link convert (only needed once), it basically sets you up on the golden path.

I agree I really need to get around to a blog post or something...

redonkulus · 2018-08-27T23:03:55Z

@evocateur is lerna link convert an easter egg? Should it be documented on https://github.com/lerna/lerna/tree/master/commands/link#readme?

evocateur · 2018-08-27T23:48:48Z

Yeah, sometimes I forget the CLI help isn't automatically shown in the README. And in this case, could definitely use more context. Along with that Lerna 3.0 blog post I've been meaning to write for a few months now...

$ npx lerna link -h
lerna link

Symlink together all packages that are dependencies of each other

Commands:
  lerna link convert  Replace local sibling version ranges with relative file: specifiers

Command Options:
  --force-local  Force local sibling links regardless of version range match                                             [boolean]

(cut off repetitive global options)

stale · 2018-12-27T04:39:29Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

revelt · 2019-05-30T14:59:13Z

hi all, I don't see any issues with Lerna hoisting on npm 6.9.0. Please confirm can we close this issue.

* Use node 6 instead of node 8 / npm 5 because of lerna/lerna#856 * Use urijs instead of the node 7+ URI package. * Don't ask lerna to hoist all dependencies. This makes our docker images smaller because we don't need to put the union of all dependencies in every image. tsmonad still needs to be hoisted to work around microsoft/TypeScript#6496. * Remove yarn.lock and package-lock.json files.

JamesHenry · 2022-11-29T11:37:57Z

Hi Folks 👋

Please take a look at our published roadmap for Lerna v7 here: #3410

One of the key items covered at length on there (please do read it for full context) is that now that we find ourselves in late 2022, it no longer makes sense for lerna to supplement package management concerns (such as installation, boostrapping, linking etc) which are covered reliably for monorepo workspaces by the three main package managers: npm, yarn and pnpm. lerna bootstrap et al were developed in completely different era of the JavaScript ecosystem.

If you have any specific concerns please do join in on that discussion, and provide as much context as possible.

Many thanks 🙏

evocateur added bug labels May 30, 2017

tivac added a commit to tivac/modular-css that referenced this issue Jun 2, 2017

chore: no more hoisting

6487d7b

Lerna doesn't really support npm@5 yet, see lerna/lerna#856

lgeiger mentioned this issue Jun 4, 2017

chore: switch to npm@5, test node@8 nteract/nteract#1728

Closed

jeffposnick mentioned this issue Jun 7, 2017

Update to npm 5? GoogleChrome/workbox#606

Closed

blink1073 mentioned this issue Jun 13, 2017

npm@5 breaks linking to extensions jupyterlab/jupyterlab#2435

Closed

swashcap mentioned this issue Jun 20, 2017

npm fixes: lockfiles & electron versioning trendscenter/coinstac#232

Merged

broccolini mentioned this issue Jun 23, 2017

Primer monorepo primer/css#230

Merged

9 tasks

vhf mentioned this issue Jul 2, 2017

Fix dependencies zestedesavoir/zmarkdown#106

Closed

akosyakov mentioned this issue Jul 18, 2017

Restructuring eclipse-theia/theia#299

Merged

gziolo mentioned this issue May 15, 2018

Packages: Move element to packages maintained by Lerna WordPress/gutenberg#6756

Merged

3 tasks

tivac mentioned this issue Aug 20, 2018

chore: correctly use file: dependencies tivac/modular-css#484

Merged

stale bot added the stale label Dec 27, 2018

evocateur removed the help wanted label Dec 27, 2018

stale bot removed the stale label Dec 27, 2018

evocateur removed the help wanted label Dec 27, 2018

JamesHenry removed upstream labels Jun 3, 2022

JamesHenry added the scope: package management Issues with the bootstrap/add/link commands that relate to package management label Jun 14, 2022

JamesHenry closed this as not planned Won't fix, can't repro, duplicate, stale Nov 29, 2022

lerna locked and limited conversation to collaborators Nov 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hoisting seems broken with npm5 #856

Hoisting seems broken with npm5 #856

jquense commented May 30, 2017

hzoo commented May 30, 2017

evocateur commented May 30, 2017

jquense commented May 30, 2017

tivac commented May 30, 2017

garthk commented Jun 5, 2017 •

edited

Loading

rgbkrk commented Jun 6, 2017 •

edited

Loading

evocateur commented Jun 8, 2017

MoOx commented Jun 10, 2017

evocateur commented Jun 11, 2017 via email

rgbkrk commented Jun 11, 2017

blink1073 commented Jun 13, 2017

blink1073 commented Jun 13, 2017

Tallyb commented Jul 18, 2017

blink1073 commented Jul 18, 2017

garthk commented Jul 24, 2017

tivac commented Jul 24, 2017

blink1073 commented Jul 24, 2017

tivac commented Jul 24, 2017

redonkulus commented Mar 24, 2018

evocateur commented Mar 24, 2018

tivac commented Mar 24, 2018 •

edited

Loading

evocateur commented Mar 24, 2018

joebowbeer commented Aug 19, 2018

evocateur commented Aug 20, 2018

tivac commented Aug 20, 2018

evocateur commented Aug 20, 2018

tivac commented Aug 20, 2018

joebowbeer commented Aug 27, 2018 •

edited

Loading

evocateur commented Aug 27, 2018

redonkulus commented Aug 27, 2018

evocateur commented Aug 27, 2018

stale bot commented Dec 27, 2018

revelt commented May 30, 2019

JamesHenry commented Nov 29, 2022

Hoisting seems broken with npm5 #856

Hoisting seems broken with npm5 #856

Comments

jquense commented May 30, 2017

Expected Behavior

Current Behavior

Your Environment

hzoo commented May 30, 2017

evocateur commented May 30, 2017

jquense commented May 30, 2017

tivac commented May 30, 2017

garthk commented Jun 5, 2017 • edited Loading

rgbkrk commented Jun 6, 2017 • edited Loading

evocateur commented Jun 8, 2017

MoOx commented Jun 10, 2017

evocateur commented Jun 11, 2017 via email

rgbkrk commented Jun 11, 2017

blink1073 commented Jun 13, 2017

blink1073 commented Jun 13, 2017

Tallyb commented Jul 18, 2017

blink1073 commented Jul 18, 2017

garthk commented Jul 24, 2017

tivac commented Jul 24, 2017

blink1073 commented Jul 24, 2017

tivac commented Jul 24, 2017

redonkulus commented Mar 24, 2018

evocateur commented Mar 24, 2018

tivac commented Mar 24, 2018 • edited Loading

evocateur commented Mar 24, 2018

joebowbeer commented Aug 19, 2018

evocateur commented Aug 20, 2018

tivac commented Aug 20, 2018

evocateur commented Aug 20, 2018

tivac commented Aug 20, 2018

joebowbeer commented Aug 27, 2018 • edited Loading

evocateur commented Aug 27, 2018

redonkulus commented Aug 27, 2018

evocateur commented Aug 27, 2018

stale bot commented Dec 27, 2018

revelt commented May 30, 2019

JamesHenry commented Nov 29, 2022

garthk commented Jun 5, 2017 •

edited

Loading

rgbkrk commented Jun 6, 2017 •

edited

Loading

tivac commented Mar 24, 2018 •

edited

Loading

joebowbeer commented Aug 27, 2018 •

edited

Loading