Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: improve Yarn 2 cache on CI #11781

Merged
merged 6 commits into from Aug 26, 2020
Merged

Conversation

@JLHwung
Copy link
Contributor

@JLHwung JLHwung commented Jul 2, 2020

Q                       A
Fixed Issues? Add @actions/cache to our GitHub coverage workflow
License MIT

Based on example: https://github.com/actions/cache/blob/master/examples.md#node---yarn

The cache is working as expected. We can compare the Yarn fetch time between main and this PR.

main This PR
2.4m 0.91s

@babel-bot
Copy link
Collaborator

@babel-bot babel-bot commented Jul 2, 2020

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/27863/

@codesandbox
Copy link

@codesandbox codesandbox bot commented Jul 2, 2020

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit b990407:

Sandbox Source
babel-repl-custom-plugin Configuration
@JLHwung JLHwung marked this pull request as ready for review Jul 2, 2020
@JLHwung JLHwung force-pushed the JLHwung:add-github-actions-cache branch from 06b433b to 411800f Aug 15, 2020
@JLHwung JLHwung changed the base branch from next-8-dev to main Aug 15, 2020
@JLHwung JLHwung force-pushed the JLHwung:add-github-actions-cache branch from 411800f to 771e85b Aug 15, 2020
@JLHwung JLHwung changed the title chore: add cache actions chore: improve Yarn 2 cache on CI Aug 15, 2020
@@ -3,13 +3,45 @@ name: Node CI
on: [push, pull_request]

jobs:
validate-yarn-cache:

This comment has been minimized.

@JLHwung

JLHwung Aug 15, 2020
Author Contributor

This step ensures the test jobs later will always be served with proper yarn cache.

cc @arcanis Do you think yarn 2 will ever support yarn install --fetch-only, which skips the linking stage and only fetche new packages to yarn cache folder. The idea is to run yarn install --fetch-only only if yarn.lock is changed, and the CI jobs afterward will be served with a proper yarn cache. In all it can save network requests and reduce carbon emissions.

Currently when yarn.lock is updated, the validate-yarn-cache job will introduce 20s (linking) overhead to our CI -- since we meant to update the yarn cache folder only.

This comment has been minimized.

@arcanis

arcanis Aug 15, 2020
Contributor

I'm not sure it'll be in the default distribution. One problem with the v1 was that installs could be done in various slightly different ways that made the surface for unforeseen edge cases much larger. I'd like to avoid that with one standardized path as much as possible.

That being said, a plugin to do this would be trivial, so we probably can find something eventually, once we've figured out a good way to develop/host such plugins (it's already possible, but there's no comprehensive documentation).

This comment has been minimized.

@merceyz

merceyz Aug 16, 2020
Contributor

You can sort of get this behaviour if you set this specific job to install using PnP and with build scripts disabled, setting these env variable should do the trick:

YARN_NODE_LINKER=pnp
YARN_ENABLE_SCRIPTS=false

This comment has been minimized.

@JLHwung

JLHwung Aug 16, 2020
Author Contributor

@merceyz We opt-out of PnP because Flow does not support PnP: yarnpkg/berry#634 (comment).

This comment has been minimized.

@merceyz

merceyz Aug 16, 2020
Contributor

Indeed, but for this specific job, validate-yarn-cache, its job is to generate a valid cache, not run Flow, so installing using PnP would do that without the cost of linking. The cache for PnP and node-modules is the same.

This comment has been minimized.

@JLHwung

JLHwung Aug 16, 2020
Author Contributor

Awesome: https://github.com/babel/babel/pull/11781/checks?check_run_id=991006280#step:6:27 The pnp linker costs only 4s compared to 20s of node-modules.

This comment has been minimized.

@merceyz

merceyz Aug 16, 2020
Contributor

You can reduce it even more if you also set YARN_ENABLE_SCRIPTS=false, you can do this since you don't need to run postinstalls to get a valid cache

This comment has been minimized.

@JLHwung

JLHwung Aug 16, 2020
Author Contributor

@JLHwung JLHwung mentioned this pull request Aug 15, 2020
@JLHwung JLHwung force-pushed the JLHwung:add-github-actions-cache branch 6 times, most recently from 34bb1a4 to ee7912c Aug 16, 2020
@JLHwung JLHwung force-pushed the JLHwung:add-github-actions-cache branch from ee7912c to ff13492 Aug 24, 2020
@JLHwung JLHwung force-pushed the JLHwung:add-github-actions-cache branch 3 times, most recently from e3dbdc4 to d484dbb Aug 25, 2020
@@ -3,13 +3,46 @@ name: Node CI
on: [push, pull_request]

jobs:
validate-yarn-cache:

This comment has been minimized.

@nicolo-ribaudo

nicolo-ribaudo Aug 25, 2020
Member

Nit: maybe this should be called prepare-yarn-cache, since "validate" implies that it would fail the build if, for example, the cache is old.

This comment has been minimized.

@JLHwung JLHwung force-pushed the JLHwung:add-github-actions-cache branch from d484dbb to b990407 Aug 25, 2020
@JLHwung JLHwung merged commit b8f3453 into babel:main Aug 26, 2020
11 of 12 checks passed
11 of 12 checks passed
Prepare Cache
Details
Test on Node.js Latest
Details
Test on Node.js (6.x)
Details
Test on Node.js (8.x)
Details
test262-pr Workflow: test262-pr
Details
Gitpod Open an online workspace in Gitpod
Details
Travis CI - Pull Request Build Passed
Details
babel/repl REPL preview is available
Details
build-standalone Workflow: build-standalone
Details
ci/codesandbox Building packages succeeded.
Details
codecov/project 91.69% (target 90.00%)
Details
e2e Workflow: e2e
Details
@JLHwung JLHwung deleted the JLHwung:add-github-actions-cache branch Aug 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.