Support Yarn workspaces to replace bootstrap command #899

bestander · Jun 27, 2017

Description + Motivation and Context

Yarn has a Workspaces feature that implements Lerna's bootstrap logic in a nice "more atomic" and more native from a package manager point of view way.
The progress and all the links to the feature is here yarnpkg/yarn#3294.

Workspaces are still considered "experimental" but we are moving forward and are ready to turn it on for a couple of projects, e.g. facebook/jest#3906.

This PR integrates Yarn workspaces with Lerna by replacing all the bootstrap/hoisting logic with a simple call to yarn install.
Also it makes Lerna use package.json/workspaces field as Yarn does.

How Has This Been Tested?

Manual:

update to Yarn 0.27.2
checkout jest with this PR facebook/jest#3906
run ../lerna/bin/lerna.js bootstrap
verify that workspace symlinks were installed in jest/node_modules
verify that unhoistable dependencies got installed in workspaces (jest/packages/jest-runtime/node_modules/strip-bom)
run yarn test-ci

added integration test

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

hzoo · Jun 27, 2017

Cool thanks @bestander!

evocateur · Jun 27, 2017

Thanks @bestander!

bestander · Jun 29, 2017

@hzoo @evocateur this is ready for review.

Bnaya · Jun 30, 2017

@bestander Thanks for the great work!
Where i can see example for this setup?

bestander · Jun 30, 2017

@Bnaya we are turning it on for jest here facebook/jest#3906.
Waiting for yarn 0.27 to become stable

jquense · Jul 5, 2017

yarn 27.0 is out yeah? can we merge this? I'd like to give it a shot

bestander · Jul 5, 2017

Yes, 0.27 is out and marked as stable.
Waiting for a review and merge please

evocateur · Jul 5, 2017

looking now, thanks for understanding

evocateur · Jul 5, 2017

evocateur requested changes Jul 5, 2017

View changes

Apologies for the delay!

evocateur · Jul 5, 2017

README.md

@@ -840,6 +840,23 @@ May also be configured in `lerna.json`:
 }
 ```

+#### --yarnWorkspaces


I think this would be clearer as --use-workspaces, with the long-term goal of making it client-independent.

evocateur · Jul 5, 2017

README.md

+{
+  ...
+  "npmClient": "yarn",
+  "yarnWorkspaces": ["bootstrap"]


I don't think this is an appropriate place to whitelist commands; that's an implementation detail if workspaces are enabled at all. In this PR alone there's already an implicit "ls" member in this array, and that's not very clear. The useWorkspaces (the camelization of --use-workspaces) value should be a boolean.

evocateur · Jul 5, 2017

src/Repository.js

@@ -55,6 +55,9 @@ export default class Repository {
  }

  get packageConfigs() {
+    if (this.lernaJson.yarnWorkspaces) {


This isn't sufficient because it doesn't pick up any potential --use-workspaces flag in the CLI args, only the "durable" config from a lerna.json (mostly equivalent to the relationship of yarn to a .yarnrc file, CLI flags always supersede the lerna.json values).

evocateur · Jul 5, 2017

src/commands/BootstrapCommand.js

+    const { yarnWorkspaces } = this.options;
+
+    if (yarnWorkspaces && (yarnWorkspaces.indexOf("bootstrap") >= 0)) {
+      this.installRootPackageOnly(callback);


No symlinking happens in this case? Or is that handled by Yarn now? (I dug through yarnpkg/yarn#3294 but my eyes glazed over before I could figure it out, sorry)

Yep, workspaces feature in Yarn creates symlinks in node_modules.
It also treats all workspaces as one installation, so it would hoist symlinks to the root node_modules unless there is a conflict, e.g. jest@next might be using jest-cli@stable in the root.

Awesome, thanks for the explanation!

bestander · Jul 5, 2017

Thanks @evocateur, no need to apologize.

bestander · Jul 5, 2017

@evocateur, I've switched to --use-workspaces and tested that it works as a CLI config as well.

bestander · Jul 6, 2017

I don't think my PR is responsible for appveyor fail

SimenB · Jul 6, 2017

Oh, npm 5 strikes again.

    "lerna info version __TEST_VERSION__
    +package-3: ┌───────────────────────────────────────────────────────────────┐
    +package-3: │                    npm update check failed                    │
    +package-3: │              Try running with sudo or get access              │
    +package-3: │             to the local update config store via              │
    +package-3: │ sudo chown -R $USER:$(id -gn $USER) C:\\Users\\appveyor\\.config │
    +package-3: └───────────────────────────────────────────────────────────────┘

Export NO_UPDATE_NOTIFIER in the tests?

evocateur · Jul 6, 2017

That's a great idea! Is that documented anywhere? I'm not against including that fix in this PR.

…

On Wed, Jul 5, 2017 at 9:53 PM, Simen Bekkhus ***@***.***> wrote: Oh, npm 5 strikes again. "lerna info version __TEST_VERSION__ +package-3: ┌───────────────────────────────────────────────────────────────┐ +package-3: │ npm update check failed │ +package-3: │ Try running with sudo or get access │ +package-3: │ to the local update config store via │ +package-3: │ sudo chown -R $USER:$(id -gn $USER) C:\\Users\\appveyor\\.config │ +package-3: └───────────────────────────────────────────────────────────────┘ Export NO_UPDATE_NOTIFIER in the tests? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#899 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAV5SyAgrZCxbecRqfYQsfiidSqZLXDks5sLGhigaJpZM4OGFNC> .

SimenB · Jul 6, 2017

https://github.com/yeoman/update-notifier#user-settings

I also sent a PR to skip the check on CI automatically. Not sure if it will be accepted, though. yeoman/update-notifier#116

evocateur · Jul 6, 2017

evocateur approved these changes Jul 6, 2017

View changes

Thanks @bestander!

SimenB · Jul 6, 2017

SimenB reviewed Jul 6, 2017

View changes

SimenB · Jul 6, 2017

README.md

+{
+  ...
+  "npmClient": "yarn",
+  "use-workspaces": true


Weird that this is not camel case

Whoops, I can fix momentarily.

bestander · Jul 6, 2017

Thanks a lot, guys!

hzoo · Jul 6, 2017

hzoo reviewed Jul 6, 2017

View changes

hzoo · Jul 6, 2017

README.md

@@ -840,6 +840,22 @@ May also be configured in `lerna.json`:
 }
 ```

+#### --use-workspaces
+
+Enables integration with [Yarn Workspaces](https://github.com/yarnpkg/rfcs/blob/master/implemented/0000-workspaces-install-phase-1.md) (available since yarn@0.27+).  


Is there a a docs page on yarnpkg.com for workspaces?

Not yet, I'll work on it in the next 3-4 weeks.

hzoo · Jul 6, 2017

Nice work everyone!

jquense · Jul 6, 2017

@bestander sorry to ping you here, but it doesn't seem like bins from workspaces are linked or generated. I've got a workspace packages/foo with a bin, but after an install there isn't anything in node_modules/.bin in the root or any workspaces that depend on foo.

wonderful feature tho, i'm really excited about it !

bestander · Jul 6, 2017

@jquense, might be a bug in Yarn.
File an issues there, we'll resolve it.

Bnaya · Jul 6, 2017

I have it up and running in my project, with my .bin files in the appropriate places

SimenB · Jul 7, 2017

SimenB reviewed Jul 7, 2017

View changes

SimenB · Jul 7, 2017

src/Repository.js

@@ -55,6 +55,9 @@ export default class Repository {
  }

  get packageConfigs() {
+    if (this.lernaJson.useWorkspaces) {
+      return this.packageJson.workspaces;


|| [DEFAULT_PACKAGE_GLOB];? This is not mentioned in the readme, so everything went boom when I tried to use this

Yarn does not have a default (PR welcome ;), so it would boom there if this setting is missing.
There is a comment about package.json/workspaces being used instead of packages https://github.com/lerna/lerna#--use-workspaces

You're correct, I'm just blind! 😀

PR welcome ;)

No doubt

thomasfr · Jul 10, 2017

Is it possible to exclude some node modules from beeing hoisted to the root folder? Like the lerna no-hoist flag did before?

But besides that. Awesome work. Cuts the installation time by half with yarn workspaces for us!! From ~2minutes down to 58secs

jquense · Jul 10, 2017

I don't think so regarding no-hoist since that's how yarn workspaces work and lerna is defering to it.

jquense · Jul 11, 2017

So i've done some more testing. It appears like yarns install with workspaces doesn't different things from lerna bootstrap. The big things i've notice are:

yarn doesn't link local bins
yarn doesn't run local prepublish/prepare scripts in linked packages, which is required for build steps

I've tried a bunch of different configs and setups with the same results. As a replacement for bootstrap it doesn't do all the things :)

bestander · Jul 11, 2017

Thanks, @jquense.
Feel free to raise issues in https://github.com/yarnpkg/yarn/issues with repro steps and ping me.

thomasfr · Jul 12, 2017

Just fyi i created an issue for --no-hoist at yarn: yarnpkg/yarn#3882

Hypnosphi · Aug 7, 2017

So is lerna bootstrap --use-workspaces a "shortcut" to yarn command?

lock · Dec 27, 2018

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.

FezVrasta referenced this pull request Jun 28, 2017
Merged
[refactor] Build improvements #326

bestander referenced this pull request Jun 29, 2017
Merged
Integrated with Yarn workspaces #3906

bestander changed the title ~~WIP: Support Yarn workspaces to replace bootstrap command~~ Support Yarn workspaces to replace bootstrap command Jun 29, 2017

bestander force-pushed the bestander:yarn-workspaces branch from 8d93488 to 3d6f874 Jun 30, 2017

bestander added some commits Jun 27, 2017

WIP: yarn workspaces

a6e8b9a

changed to --use-workspaces flag after feedback

Loading status checks…

a003740

bestander force-pushed the bestander:yarn-workspaces branch from 2a663ac to a003740 Jul 5, 2017

SimenB referenced this pull request Jul 6, 2017
Merged
Skip update notifiers on CI #907
4 of 9 tasks complete

fix typo [skip ci]

b801d57

evocateur merged commit 616e44c into lerna:master Jul 6, 2017

evocateur added the Tag: Enhancement label Jul 6, 2017

Bnaya referenced this pull request Jul 7, 2017
Closed
When using with workspaces, yarn ignores nohoist settings #909

marionebl referenced this pull request Nov 18, 2017
Closed
[question] Using yarn commands with non-published lerna dependencies #501

kiriappeee referenced this pull request Feb 7, 2018
Open
Upgrading Node to `8.9.4` and/or upgrading the `yarn` version in the image #262

evocateur referenced this pull request Mar 13, 2018
Closed
lerna bootstrap failing as a postinstall step with yarn workspaces #1308

tech4him1 referenced this pull request Sep 1, 2018
Open
Package has both yarn.lock and package-lock.json #1516

bd82 referenced this pull request Nov 9, 2018
Closed
Convert from yarn to npm #93

lock bot locked as resolved and limited conversation to collaborators Dec 27, 2018

lerna/lerna

Join GitHub today

Support Yarn workspaces to replace bootstrap command #899

Conversation

bestander commented Jun 27, 2017 • edited

Description + Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

This comment has been minimized.

hzoo commented Jun 27, 2017

This comment has been minimized.

evocateur commented Jun 27, 2017

FezVrasta referenced this pull request Jun 28, 2017

bestander referenced this pull request Jun 29, 2017

bestander changed the title WIP: Support Yarn workspaces to replace bootstrap command Support Yarn workspaces to replace bootstrap command Jun 29, 2017

This comment has been minimized.

bestander commented Jun 29, 2017

This comment has been minimized.

Bnaya commented Jun 30, 2017

This comment has been minimized.

bestander commented Jun 30, 2017

bestander force-pushed the bestander:yarn-workspaces branch from 8d93488 to 3d6f874 Jun 30, 2017

This comment has been minimized.

jquense commented Jul 5, 2017

This comment has been minimized.

bestander commented Jul 5, 2017

This comment has been minimized.

evocateur commented Jul 5, 2017

evocateur requested changes Jul 5, 2017 View changes

evocateur left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bestander commented Jul 5, 2017

This comment has been minimized.

bestander commented Jul 5, 2017

bestander added some commits Jun 27, 2017

bestander force-pushed the bestander:yarn-workspaces branch from 2a663ac to a003740 Jul 5, 2017

This comment has been minimized.

bestander commented Jul 6, 2017

This comment has been minimized.

SimenB commented Jul 6, 2017

This comment has been minimized.

evocateur commented Jul 6, 2017

This comment has been minimized.

SimenB commented Jul 6, 2017

SimenB referenced this pull request Jul 6, 2017

evocateur approved these changes Jul 6, 2017 View changes

evocateur left a comment

evocateur merged commit 616e44c into lerna:master Jul 6, 2017

evocateur added the Tag: Enhancement label Jul 6, 2017

SimenB reviewed Jul 6, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bestander commented Jul 6, 2017

hzoo reviewed Jul 6, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hzoo commented Jul 6, 2017

This comment has been minimized.

jquense commented Jul 6, 2017 • edited

This comment has been minimized.

bestander commented Jul 6, 2017

This comment has been minimized.

Bnaya commented Jul 6, 2017

SimenB reviewed Jul 7, 2017 View changes

This comment has been minimized.

SimenB Jul 7, 2017 • edited

This comment has been minimized.

This comment has been minimized.

bestander commented Jun 27, 2017 •

edited

bestander force-pushed the bestander:yarn-workspaces branch from `8d93488` to `3d6f874` Jun 30, 2017

evocateur requested changes Jul 5, 2017

View changes

bestander force-pushed the bestander:yarn-workspaces branch from `2a663ac` to `a003740` Jul 5, 2017

evocateur approved these changes Jul 6, 2017

View changes

evocateur merged commit `616e44c` into lerna:master Jul 6, 2017

SimenB reviewed Jul 6, 2017

View changes

hzoo reviewed Jul 6, 2017

View changes

jquense commented Jul 6, 2017 •

edited

SimenB reviewed Jul 7, 2017

View changes

SimenB Jul 7, 2017 •

edited

thomasfr commented Jul 10, 2017 •

edited