New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrated with Yarn workspaces #3906

Merged
merged 6 commits into from Jul 7, 2017

Conversation

Projects
None yet
6 participants
@bestander
Contributor

bestander commented Jun 25, 2017

Summary

Yarn 0.27 has workspaces feature that should be a better alternative to Lerna boostrapping.
Yarn workspaces still don't replace Lerna (and probably won't do it completely for a long time), so Lerna publishing is still used for Jest.

This PR replaces Lerna bootstrap phase with Yarn workspaces.

Things to discuss:

  1. Unlike Lerna Yarn hoists workspaces to the root node_modules.

It means that integration/examples test don't run in isolation from the workspace root so I removed symlinking to babel-jest et al. at the tests boostrapping.
One test integration_tests/__tests__/transform.test.js required to be running in isolation, so I changed the test bootstrap to execute on a copy on a temp folder.

  1. As of Yarn 0.27.2 workspaces are still fresh, I am working on making it snappy.
    Known issues:
  • changes in workspaces (non root) package.json are not automatically invalidating integrity check
  • logs may have confusing lines when working with workspaces
  • yarn add/upgrade/remove is not fully tested from within workspaces
  1. PR to support Yarn workspaces as first class citizen in Lerna lerna/lerna#899

Test plan

  • yarn test-ci
  • lint/flow/build tasks work per workspaces correctly
  • ran yarn run publish and lerna seems to handle version bumping and npm publishing correctly
@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jun 25, 2017

Contributor

We need to bump Yarn 0.27 to stable to fix CI.

Contributor

bestander commented Jun 25, 2017

We need to bump Yarn 0.27 to stable to fix CI.

@bestander bestander referenced this pull request Jun 27, 2017

Merged

Support Yarn workspaces to replace bootstrap command #899

7 of 7 tasks complete
@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jun 27, 2017

Contributor

This is super sweet @bestander!

Mind rebasing and giving me an update on what the remaining problems are on this diff?

Contributor

cpojer commented Jun 27, 2017

This is super sweet @bestander!

Mind rebasing and giving me an update on what the remaining problems are on this diff?

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jun 27, 2017

Contributor

@cpojer will do.
What do you think of integration_tests/__tests__/transform.test.js change?
Yarn hoists all packages as symlinks to the top so all tests now have babel-jest available by default and we can't test integration/example tests in isolation anymore.

Contributor

bestander commented Jun 27, 2017

@cpojer will do.
What do you think of integration_tests/__tests__/transform.test.js change?
Yarn hoists all packages as symlinks to the top so all tests now have babel-jest available by default and we can't test integration/example tests in isolation anymore.

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jun 27, 2017

Collaborator

If I recall correctly we're already using os. tmpdir in some of integration tests so I'm not opposed to do it in this case.

Collaborator

thymikee commented Jun 27, 2017

If I recall correctly we're already using os. tmpdir in some of integration tests so I'm not opposed to do it in this case.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jun 27, 2017

Contributor

Yep, I'm fine with it also.

Contributor

cpojer commented Jun 27, 2017

Yep, I'm fine with it also.

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jun 29, 2017

Contributor

Waiting for Yarn 0.27.2 release to be stable so that CI would run installation correctly

Contributor

bestander commented Jun 29, 2017

Waiting for Yarn 0.27.2 release to be stable so that CI would run installation correctly

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jul 4, 2017

Collaborator

@bestander is yarn stable now?

Collaborator

thymikee commented Jul 4, 2017

@bestander is yarn stable now?

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 4, 2017

Contributor
Contributor

bestander commented Jul 4, 2017

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jul 4, 2017

Collaborator

Would you mind rebasing this diff when you have a while? :)

Collaborator

thymikee commented Jul 4, 2017

Would you mind rebasing this diff when you have a while? :)

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 4, 2017

Contributor
Contributor

bestander commented Jul 4, 2017

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jul 4, 2017

Collaborator

It passes locally, I've rebased and restarted the build, we'll see :)

Collaborator

thymikee commented Jul 4, 2017

It passes locally, I've rebased and restarted the build, we'll see :)

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 4, 2017

Contributor
Contributor

bestander commented Jul 4, 2017

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jul 4, 2017

Collaborator

Tried rebuilding without cache, but still the same. Not sure why it happens. cc @SimenB @skovhus

Collaborator

thymikee commented Jul 4, 2017

Tried rebuilding without cache, but still the same. Not sure why it happens. cc @SimenB @skovhus

@skovhus

This comment has been minimized.

Show comment
Hide comment
@skovhus

skovhus Jul 4, 2017

Contributor

Sound weird that this affects the rollup build, but now I'm super happy that I added the eslint check. 😊

Have you looked into the es5 build for each package and see if they all contains un-transpiled code?

Contributor

skovhus commented Jul 4, 2017

Sound weird that this affects the rollup build, but now I'm super happy that I added the eslint check. 😊

Have you looked into the es5 build for each package and see if they all contains un-transpiled code?

@skovhus

This comment has been minimized.

Show comment
Hide comment
@skovhus

skovhus Jul 4, 2017

Contributor

@thymikee @bestander

jest-matchers/build-es5/index.js
  4593:1  error  Parsing error: The keyword 'const' is reserved

✖ 1 problem (1 error, 0 warnings)

Looking into the bundle reveals that ansi-styles@3.0.0 contains ES2015 code (e.g. const and for... of)... 😐

I guess this will happen as node library authors are targeting newer version of node.

Contributor

skovhus commented Jul 4, 2017

@thymikee @bestander

jest-matchers/build-es5/index.js
  4593:1  error  Parsing error: The keyword 'const' is reserved

✖ 1 problem (1 error, 0 warnings)

Looking into the bundle reveals that ansi-styles@3.0.0 contains ES2015 code (e.g. const and for... of)... 😐

I guess this will happen as node library authors are targeting newer version of node.

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 4, 2017

Contributor
Contributor

bestander commented Jul 4, 2017

@skovhus

This comment has been minimized.

Show comment
Hide comment
@skovhus

skovhus Jul 4, 2017

Contributor

This patch solves the problem by also running Babel on node_modules (except for node_modules/babel-runtime that Babel ironically fails on with a A module cannot import itself error):

From b62040420421da46de054e7b9afc25649b45700d Mon Sep 17 00:00:00 2001
From: Kenneth Skovhus <kenneth.skovhus@gmail.com>
Date: Tue, 4 Jul 2017 22:39:04 +0200
Subject: [PATCH] Fix browser build (ansi-styles contains ES2015 code)

---
 scripts/browserBuild.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/browserBuild.js b/scripts/browserBuild.js
index ac43ca0..db1b4a2 100644
--- a/scripts/browserBuild.js
+++ b/scripts/browserBuild.js
@@ -20,7 +20,7 @@ const babelEs5Options = Object.assign(
   {},
   {
     babelrc: false,
-    exclude: 'node_modules/**',
+    exclude: 'node_modules/babel-runtime/**',
     plugins: [
       'syntax-trailing-function-commas',
       'transform-flow-strip-types',
-- 
2.10.2

@thymikee @bestander
But the underlying issues seems to be that we are upgrading dependencies in the packages (example ansi-styles@2 to ansi-styles@3) due to all the lock files being removed. See #3906 (comment)

Contributor

skovhus commented Jul 4, 2017

This patch solves the problem by also running Babel on node_modules (except for node_modules/babel-runtime that Babel ironically fails on with a A module cannot import itself error):

From b62040420421da46de054e7b9afc25649b45700d Mon Sep 17 00:00:00 2001
From: Kenneth Skovhus <kenneth.skovhus@gmail.com>
Date: Tue, 4 Jul 2017 22:39:04 +0200
Subject: [PATCH] Fix browser build (ansi-styles contains ES2015 code)

---
 scripts/browserBuild.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/browserBuild.js b/scripts/browserBuild.js
index ac43ca0..db1b4a2 100644
--- a/scripts/browserBuild.js
+++ b/scripts/browserBuild.js
@@ -20,7 +20,7 @@ const babelEs5Options = Object.assign(
   {},
   {
     babelrc: false,
-    exclude: 'node_modules/**',
+    exclude: 'node_modules/babel-runtime/**',
     plugins: [
       'syntax-trailing-function-commas',
       'transform-flow-strip-types',
-- 
2.10.2

@thymikee @bestander
But the underlying issues seems to be that we are upgrading dependencies in the packages (example ansi-styles@2 to ansi-styles@3) due to all the lock files being removed. See #3906 (comment)

@@ -5,15 +5,17 @@ environment:
install:
- ps: Install-Product node $env:nodejs_version x64
- node --version
- yarn --version
- yarn install
- curl -fsSL -o yarn.js https://github.com/yarnpkg/yarn/releases/download/v0.27.5/yarn-0.27.5.js

This comment has been minimized.

@skovhus

skovhus Jul 4, 2017

Contributor

Wouldn't npm install --global yarn@0.27.5 be nicer here than node ./yarn.js?

@skovhus

skovhus Jul 4, 2017

Contributor

Wouldn't npm install --global yarn@0.27.5 be nicer here than node ./yarn.js?

This comment has been minimized.

@bestander

bestander Jul 5, 2017

Contributor

Then we would need to be sure that npm global dir precedes appveyor's global Yarn installation in PATH.
I don't have a strong opinion on this and happy to change to anything that maintainers think is best.

@bestander

bestander Jul 5, 2017

Contributor

Then we would need to be sure that npm global dir precedes appveyor's global Yarn installation in PATH.
I don't have a strong opinion on this and happy to change to anything that maintainers think is best.

This comment has been minimized.

@skovhus

skovhus Jul 6, 2017

Contributor

To me it is just more standard to rely on a global yarn installation. I hope appveyour global Yarn installation will be overridden by a npm install -g.

@skovhus

skovhus Jul 6, 2017

Contributor

To me it is just more standard to rely on a global yarn installation. I hope appveyour global Yarn installation will be overridden by a npm install -g.

This comment has been minimized.

@bestander

bestander Jul 7, 2017

Contributor

So what should I do here, npm install -g?

@bestander

bestander Jul 7, 2017

Contributor

So what should I do here, npm install -g?

@@ -1,727 +0,0 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.

This comment has been minimized.

@skovhus

skovhus Jul 4, 2017

Contributor

Does this mean that each package dependencies are not locked by a yarn.lock file?

@skovhus

skovhus Jul 4, 2017

Contributor

Does this mean that each package dependencies are not locked by a yarn.lock file?

This comment has been minimized.

@skovhus

skovhus Jul 4, 2017

Contributor

Ah, yes. 😎

Unlike Lerna Yarn hoists workspaces to the root node_modules.

@skovhus

skovhus Jul 4, 2017

Contributor

Ah, yes. 😎

Unlike Lerna Yarn hoists workspaces to the root node_modules.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jul 4, 2017

Contributor

Can we whitelist only the node_moduels that are not ES5 compliant and transform those? I'd much rather compile only one or two that need it, rather than all of them.

Contributor

cpojer commented Jul 4, 2017

Can we whitelist only the node_moduels that are not ES5 compliant and transform those? I'd much rather compile only one or two that need it, rather than all of them.

@skovhus

This comment has been minimized.

Show comment
Hide comment
@skovhus

skovhus Jul 4, 2017

Contributor

@cpojer a whitelist might be a moving target, but probably a better option. 👍🏻

This works:

-    exclude: 'node_modules/**',
+    exclude: 'node_modules/!(ansi-styles|remove-trailing-separator)/**',

remove-trailing-separator can be fixed by upgrading from 1.0.1 to 1.0.2, see darsain/remove-trailing-separator@6dd7a71

Contributor

skovhus commented Jul 4, 2017

@cpojer a whitelist might be a moving target, but probably a better option. 👍🏻

This works:

-    exclude: 'node_modules/**',
+    exclude: 'node_modules/!(ansi-styles|remove-trailing-separator)/**',

remove-trailing-separator can be fixed by upgrading from 1.0.1 to 1.0.2, see darsain/remove-trailing-separator@6dd7a71

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jul 5, 2017

Contributor

Yeah let's do this. It's fine with me if we have to update this list from time to time.

Contributor

cpojer commented Jul 5, 2017

Yeah let's do this. It's fine with me if we have to update this list from time to time.

Show outdated Hide outdated lerna.json
@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 5, 2017

Contributor

Thanks a lot, @skovhus, this fixed the build.
Indeed it is great that we have es5 tests now.

The way rollup picks modules is a bit concerning though, it all goes down to what structure is inside the root node_modules and it is conforms with Node.js resolution, not for browser builds for each package in isolation

We might rethink how those jest-mock and jest-matchers are built.
They probably should do it in isolation with --flat mode.

Contributor

bestander commented Jul 5, 2017

Thanks a lot, @skovhus, this fixed the build.
Indeed it is great that we have es5 tests now.

The way rollup picks modules is a bit concerning though, it all goes down to what structure is inside the root node_modules and it is conforms with Node.js resolution, not for browser builds for each package in isolation

We might rethink how those jest-mock and jest-matchers are built.
They probably should do it in isolation with --flat mode.

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Jul 5, 2017

Codecov Report

Merging #3906 into master will increase coverage by 0.26%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3906      +/-   ##
==========================================
+ Coverage   59.87%   60.13%   +0.26%     
==========================================
  Files         196      195       -1     
  Lines        6794     6736      -58     
  Branches        6        6              
==========================================
- Hits         4068     4051      -17     
+ Misses       2723     2682      -41     
  Partials        3        3
Impacted Files Coverage Δ
packages/jest-config/src/index.js 0% <0%> (-36.37%) ⬇️
...ages/jest-config/src/reporter_validation_errors.js 16.66% <0%> (-16.67%) ⬇️
packages/jest-config/src/find_config.js 0% <0%> (-15.16%) ⬇️
packages/jest-config/src/defaults.js 85.71% <0%> (-14.29%) ⬇️
packages/jest-config/src/vendor/jsonlint.js 0% <0%> (-5.73%) ⬇️
packages/jest-config/src/utils.js 73.33% <0%> (-3.75%) ⬇️
packages/jest-config/src/normalize.js 84.66% <0%> (-0.89%) ⬇️
packages/jest-resolve-dependencies/src/index.js 100% <0%> (ø) ⬆️
packages/jest-config/src/valid_config.js 100% <0%> (ø) ⬆️
packages/jest-diff/src/constants.js 100% <0%> (ø) ⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 286b877...3d63117. Read the comment docs.

codecov-io commented Jul 5, 2017

Codecov Report

Merging #3906 into master will increase coverage by 0.26%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3906      +/-   ##
==========================================
+ Coverage   59.87%   60.13%   +0.26%     
==========================================
  Files         196      195       -1     
  Lines        6794     6736      -58     
  Branches        6        6              
==========================================
- Hits         4068     4051      -17     
+ Misses       2723     2682      -41     
  Partials        3        3
Impacted Files Coverage Δ
packages/jest-config/src/index.js 0% <0%> (-36.37%) ⬇️
...ages/jest-config/src/reporter_validation_errors.js 16.66% <0%> (-16.67%) ⬇️
packages/jest-config/src/find_config.js 0% <0%> (-15.16%) ⬇️
packages/jest-config/src/defaults.js 85.71% <0%> (-14.29%) ⬇️
packages/jest-config/src/vendor/jsonlint.js 0% <0%> (-5.73%) ⬇️
packages/jest-config/src/utils.js 73.33% <0%> (-3.75%) ⬇️
packages/jest-config/src/normalize.js 84.66% <0%> (-0.89%) ⬇️
packages/jest-resolve-dependencies/src/index.js 100% <0%> (ø) ⬆️
packages/jest-config/src/valid_config.js 100% <0%> (ø) ⬆️
packages/jest-diff/src/constants.js 100% <0%> (ø) ⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 286b877...3d63117. Read the comment docs.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jul 6, 2017

Contributor

Are we good to go on this one? Is the latest version stable and rolled out so that people can use this feature?

Contributor

cpojer commented Jul 6, 2017

Are we good to go on this one? Is the latest version stable and rolled out so that people can use this feature?

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jul 6, 2017

Collaborator

I was able to install yarn 0.27.5 through brew, so pretty sure it's rolled out 👍

Collaborator

thymikee commented Jul 6, 2017

I was able to install yarn 0.27.5 through brew, so pretty sure it's rolled out 👍

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jul 6, 2017

Collaborator

we can put "engines" entry in package.json though

Collaborator

thymikee commented Jul 6, 2017

we can put "engines" entry in package.json though

@skovhus

This comment has been minimized.

Show comment
Hide comment
@skovhus

skovhus Jul 6, 2017

Contributor

Are we good to go on this one? Is the latest version stable and rolled out so that people can use this feature?

A lot of dependencies might have been bumped in all packages as dependencies are resolved from outer package.json file now. But I guess this is fairly safe due to the test coverage we have.

Contributor

skovhus commented Jul 6, 2017

Are we good to go on this one? Is the latest version stable and rolled out so that people can use this feature?

A lot of dependencies might have been bumped in all packages as dependencies are resolved from outer package.json file now. But I guess this is fairly safe due to the test coverage we have.

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 7, 2017

Contributor

@thymikee, "engines" probably won't do much, in latest npm it is not strict.

Contributor

bestander commented Jul 7, 2017

@thymikee, "engines" probably won't do much, in latest npm it is not strict.

bestander added some commits Jun 24, 2017

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 7, 2017

Contributor

Updates Lerna to 2.0.0, it supports workspaces now

Contributor

bestander commented Jul 7, 2017

Updates Lerna to 2.0.0, it supports workspaces now

@rarkins rarkins referenced this pull request Jul 7, 2017

Closed

Yarn workspaces #473

0 of 3 tasks complete
@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jul 7, 2017

Contributor

And I'm getting another error about const. If you fix CI, happy if you merge this. Love the change.

Contributor

cpojer commented Jul 7, 2017

And I'm getting another error about const. If you fix CI, happy if you merge this. Love the change.

@cpojer

cpojer approved these changes Jul 7, 2017

bestander added some commits Jul 7, 2017

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jul 7, 2017

Contributor

Should be green now again.

Contributor

bestander commented Jul 7, 2017

Should be green now again.

@facebook-github-bot

This comment has been minimized.

Show comment
Hide comment
@facebook-github-bot

facebook-github-bot Jul 7, 2017

@bestander has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot commented Jul 7, 2017

@bestander has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bestander bestander merged commit e6704bf into facebook:master Jul 7, 2017

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jul 7, 2017

Contributor

Woohoo, awesome work @bestander!

Contributor

cpojer commented Jul 7, 2017

Woohoo, awesome work @bestander!

@SimenB SimenB referenced this pull request Jul 8, 2017

Merged

Remove unused yarn.lock #3989

@richburdon richburdon referenced this pull request Jul 18, 2017

Closed

Workspaces issue progress #3294

18 of 22 tasks complete

tushardhole pushed a commit to tushardhole/jest that referenced this pull request Aug 21, 2017

Integrated with Yarn workspaces (facebook#3906)
* enabled workspaces

* updated lerna to 2.0.0 with workspaces support

@shawnbot shawnbot referenced this pull request Aug 29, 2017

Closed

[WIP] Update to lerna@2.1.1; use Yarn + workspaces #331

0 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment