Travis optimization #2404

Munter · 2018-07-28T14:03:44Z

I've leveraged as much caching on Travis as possible, which have sped up yarn installs and proselint runs by a lot.

I've switched the travis setup to run on the cheap and fast container setup, which means the test start faster.

The lint+build stage runs all linting except proselint, which runs in parallel in a separate container.

The lint+build stage saves build artefacts in a cache.

After the build has succeeded the external link check runs on the build artefacts in a separate container. This is a non-blocking test run to avoid massive blockage of PR's based on external links changing.

After the build has succeeded and if the build is of the master branch, a deploy will be initialized with the cached artefacts from the build in a previous container.

The fetching of the repositories for loaders and plugins, which previously caused our github api usage to exeed the rate limit has now been removed from the build pipeline. Instead it is a script you will run manually as a maintainer or contributor (yarn update-repos) , which will save the updated repos in version control. The build uses that list of repos to fetch the readmes without ever hitting the github api, and thus avoiding the rate limitation.

All of the above means we can increase the number of parallel builds on Travis, which are currently limited to 1. I think 4 is the maximum we can get, which will give us significant improvement on feedback time.

Latest build of this branch: https://travis-ci.org/webpack/webpack.js.org/builds/409489808

Build times are around 5 minutes for the blocking part and 4 minutes for the non-blocking part. I've set travis up to `fast_finish, which should give a green light in Github as soon as the blocking part of the tests are finished.

…se made the checker go into CPU overdrive

…version

montogeek · 2018-07-28T19:55:42Z

It worked?

montogeek · 2018-07-28T19:56:41Z

.travis.yml

+
+    - stage: post-build
+      name: Deploy
+      if: branch = master


This condition saves deploy.sh internal condition, right?

I'll clean that up in a bit

montogeek · 2018-07-28T19:57:35Z

.travis.yml

+      name: Link check
+      script: yarn linkcheck
+
+    - stage: post-build


Why it this one post build? We shouldn't build if this command fails.

We can re-order as we see fit later. Right now I just need to get the pieces working, then we can discuss what the pipeline should do.

…. Try to centralize github API usage to avoid rate limitations

montogeek · 2018-07-28T22:44:34Z

.travis.yml

@@ -51,6 +51,7 @@ jobs:
      cache:
        pip: true
        directories:
+          - dist


This should be revisited. At this point it was already deployed, right?

Yeah, I also forgot that the dist folder is actually not even checked with proselint at all. Just src/content. So this is just useless

…script

…ecked" This reverts commit fc7676d.

Munter · 2018-07-29T10:11:50Z

This branch is ready to go!

I've leveraged as much caching on Travis as possible, which have sped up yarn installs and proselint runs by a lot.

I've switched the travis setup to run on the cheap and fast container setup, which means the test start faster.

The lint+build stage runs all linting except proselint, which runs in parallel in a separate container.

The lint+build stage saves build artefacts in a cache.

After the build has succeeded the external link check runs on the build artefacts in a separate container. This is a non-blocking test run to avoid massive blockage of PR's based on external links changing.

After the build has succeeded and if the build is of the master branch, a deploy will be initialized with the cached artefacts from the build in a previous container.

The fetching of the repositories for loaders and plugins, which previously caused our github api usage to exeed the rate limit has now been removed from the build pipeline. Instead it is a script you will run manually as a maintainer or contributor (yarn update-repos) , which will save the updated repos in version control. The build uses that list of repos to fetch the readmes without ever hitting the github api, and thus avoiding the rate limitation.

All of the above means we can increase the number of parallel builds on Travis, which are currently limited to 1. I think 4 is the maximum we can get, which will give us significant improvement on feedback time.

Latest build of this branch: https://travis-ci.org/webpack/webpack.js.org/builds/409489808

Build times are around 5 minutes for the blocking part and 4 minutes for the non-blocking part. I've set travis up to `fast_finish, which should give a green light in Github as soon as the blocking part of the tests are finished.

Munter · 2018-07-29T10:12:32Z

Not that this PR merges to rebuild-hyperlink, since the changes there are prerequisites

byzyk · 2018-07-29T10:21:27Z

.travis.yml

+  include:
+    - stage: Build
+      name: Lint and Build
+      before_install: npm install --global yarn


Travis will automatically use yarn once it sees yarn.lock file, we don't need to install it explicitly.

byzyk · 2018-07-29T10:25:23Z

package.json

    "test": "npm run lint",
    "lint": "run-s lint:*",
-    "lint:js": "eslint src --ext .js,.jsx,.md",
+    "lint:js": "eslint src --ext .js,.jsx,.md --cache true --cache-location .cache/.eslintcache",


Nice one! 👍🏽

byzyk · 2018-07-29T10:33:02Z

src/utilities/fetch-package-readmes.js

+
+          await writeFile(fileName, headmatter + body);
+
+          console.log('Generated:', path.relative(cwd, fileName));


Don't mean to sound too picky but perhaps we could add space after Generated: here?

The console logging adds a space between arguments :)

byzyk · 2018-07-29T10:34:43Z

src/utilities/fetch-package-repos.js

+    fs.writeFile(jsonPath, json, (err) => {
+      if (err) {
+        throw err;
+      }


Do you think it will make sense to add some basic logging here, the same as one in fetch-package-readmes script?

That would indeed make sense. I'd prefer if someone else adds that. I'm about to be offline for a week

byzyk · 2018-07-29T10:41:34Z

Awesome work @Munter, can't wait to try this out! I like the idea of not having to fetch GH repos every 5 minutes and it's gonna solve the API limit issue finally 🎉

montogeek · 2018-07-31T10:28:50Z

@jeremenichelli @EugeneHlushko Any feedback? I would like to merge this one.

montogeek · 2018-07-31T10:30:01Z

@Munter There is a cron job that deploys every day, How would this affect it?

Munter · 2018-07-31T16:37:16Z

@montogeek since I kept the script names intact I think there will be no impact on any cron job

montogeek · 2018-07-31T16:52:54Z

@Munter Yep

montogeek · 2018-08-02T09:43:17Z

I will merge this one later today

* Update link checking to use latest hyperlink. Replaces #1582 * Stop appending empty fragment to urls pointing ad no fragment identifier * Resolve relative hrefs from README's to their github.com url instead of linking locally inside docs * Prefer https where available and also use https in examples * Make non-fragment links inside headings visible * Update webpack docs china link * Fixed invalid edit links to index pages * skip fragment check for vimdoc that is known to fail * Fix broken link to migration guide from configuration/resolve * Also resolve github README relative links when they don't start with a dot * Add missing pages: Vote, Organization, Starter kits * Remove unused code * Fix link that was redirected * Open all target='_blank' links with rel='noopener' for security * Skip link checking img.shields.io * Change order of markdown url rewriting to avoid errors * Use lowercase variable name for node url module. See #2382 (comment) * Update link to file-loader github repo * Rewrite links from https://npmjs.com to https://www.npmjs.com to avoid the inevitable redirect * Skip checking links to npmjs.com/package that causes hyperlink to get http 500 responses. This should probably be reverted and fixed later * Updated organization projects urls to their redirect target * Update tool-list to get correct links for projects * Updated links to their redirect target * Remove jscs-loader. The project is deprecated, merged with eslint and the domain we link to is for sale * Update link to karma-webpack project * Remove link to named-modules-plugin, which is now in webpack core * Added missing mini-css-extract-plugin to fetch script * Skip checking links to known deleted github users * Update broken fragments to work with new markdown generator * Fixed more broken links * infra(site) Copy assets when building * Update github location for css-loader * Remove dead github accounts from contributors listings * infra(site) Fix fetchPackages script when running locally * Fix internal fragmtn links in optimization.md * Skip link checking or opencollective.com/webpack. Massive html response made the checker go into CPU overdrive * Remove link to non-existing named-modules-plugin * Use new release of hyperlink * feat(infra) Travis optimization (#2404) * Fix internal fragmtn links in optimization.md * Skip link checking or opencollective.com/webpack. Massive html response made the checker go into CPU overdrive * Try out travis staged build * Add proselint * Upgrade pip before using it * Move before-hooks around to try and make proselint install * Try adding python as a language as well to geet an updated version * More messing with config * Manually handle node versioning * Add node minor version to nvm install. Defaulted to slightly too low version * Manually install node 8.11 * Try a matrix build to separate node and python stuff * Add linkcheck task and try a different cahce setup * Minor name change to test if cache works correctly across builds * Attempt to combine matrix and staged build * Attempt going back to staged build * Bump travis * Ping Travis. You alive? * Fix broken travis.yml * Fix wrong stage order * Explicitly run specific lintings, exclude proselint * Allow failures for link checker * Change proselint cache directory. Maybe this works * Add new script to fetch repository names that the docs should contain. Try to centralize github API usage to avoid rate limitations * Add caching to eslint * Remove parts of deploy.sh that are now run by travis * Added new ./repositories to store github api results * Replace fetch.sh with npm scripts and fetch-package-readmes.js * Attempt to make caches more specific to the containers and stages they refer to * Remove deprecaed fetch_packages script. Now replaced by two-step fetch-package-repos and fetch-package-readmes * Attempt a more windows friendly build by using npm and webpack instead of shell commands * Disable link checking for now to speed up builds * Revert "Disable link checking for now to speed up builds" This reverts commit 7d4bb06. * Add dist to proselint cache so generated content also gets checked * Remove unnessessary GITHUB_TOKEN env variable check in repo fetching script * Revert "Add dist to proselint cache so generated content also gets checked" This reverts commit fc7676d. * Rework pipeline for better speed and make proselint a deployment blocker * Rename build stage to reflect its actions * Run content-tree only after generating all external content * Remove link to non-existing named-modules-plugin * Fix double slashes in edit links * Rename stages * Add new internal link check as a blocking check * Fix wrong name in allowed_failures config

Munter added 23 commits July 27, 2018 12:07

Fix internal fragmtn links in optimization.md

cabd893

Skip link checking or opencollective.com/webpack. Massive html respon…

5321534

…se made the checker go into CPU overdrive

Try out travis staged build

03e0dcf

Add proselint

b31f42e

Upgrade pip before using it

e616a32

Move before-hooks around to try and make proselint install

64d5ac9

Try adding python as a language as well to geet an updated version

012b534

More messing with config

3efa21d

Manually handle node versioning

2bcfb86

Add node minor version to nvm install. Defaulted to slightly too low …

10f18df

…version

Manually install node 8.11

ba535b5

Try a matrix build to separate node and python stuff

315cdef

Add linkcheck task and try a different cahce setup

ef4cafa

Minor name change to test if cache works correctly across builds

1bfbcc8

Attempt to combine matrix and staged build

f8a6cf8

Attempt going back to staged build

df21ca2

Bump travis

b344006

Ping Travis. You alive?

625c6e7

Fix broken travis.yml

3416ecb

Fix wrong stage order

3545917

Explicitly run specific lintings, exclude proselint

5d238f4

Allow failures for link checker

2dc8216

Change proselint cache directory. Maybe this works

d0c36fc

montogeek reviewed Jul 28, 2018

View reviewed changes

Munter added 4 commits July 28, 2018 21:57

Add new script to fetch repository names that the docs should contain…

0c18717

…. Try to centralize github API usage to avoid rate limitations

Add caching to eslint

b88f90b

Remove parts of deploy.sh that are now run by travis

08dbaef

Added new ./repositories to store github api results

9f1a1cc

montogeek reviewed Jul 28, 2018

View reviewed changes

Munter added 3 commits July 29, 2018 00:55

Remove unnessessary GITHUB_TOKEN env variable check in repo fetching …

993c998

…script

Revert "Add dist to proselint cache so generated content also gets ch…

8332fe3

…ecked" This reverts commit fc7676d.

Rework pipeline for better speed and make proselint a deployment blocker

fbd7a2f

Munter force-pushed the rebuild-travis-stages branch from 17c5d9f to fbd7a2f Compare July 29, 2018 06:18

Munter added 6 commits July 29, 2018 08:19

Rename build stage to reflect its actions

65644a1

Run content-tree only after generating all external content

a2e6146

Remove link to non-existing named-modules-plugin

1ec6c28

Fix double slashes in edit links

80685a7

Rename stages

42f0b1d

Add new internal link check as a blocking check

dc053d9

Munter changed the title ~~DO NOT MERGE: Travis optimization attempt~~ Travis optimization Jul 29, 2018

Munter requested review from byzyk, skipjack and EugeneHlushko July 29, 2018 10:00

Fix wrong name in allowed_failures config

ea7f606

byzyk reviewed Jul 29, 2018

View reviewed changes

montogeek approved these changes Jul 31, 2018

View reviewed changes

montogeek added the Infrastructure label Jul 31, 2018

montogeek merged commit ab6f6c1 into rebuild-hyperlink Aug 2, 2018

Munter deleted the rebuild-travis-stages branch August 2, 2018 16:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Travis optimization #2404

Travis optimization #2404

Munter commented Jul 28, 2018 •

edited

montogeek commented Jul 28, 2018

montogeek Jul 28, 2018

Munter Jul 28, 2018

Munter Jul 28, 2018

montogeek Jul 28, 2018

Munter Jul 28, 2018

montogeek Jul 28, 2018

Munter Jul 28, 2018

Munter commented Jul 29, 2018

Munter commented Jul 29, 2018

byzyk Jul 29, 2018

byzyk Jul 29, 2018

byzyk Jul 29, 2018

Munter Jul 29, 2018

byzyk Jul 29, 2018

Munter Jul 29, 2018

byzyk commented Jul 29, 2018

montogeek commented Jul 31, 2018

montogeek commented Jul 31, 2018

Munter commented Jul 31, 2018

montogeek commented Jul 31, 2018

montogeek commented Aug 2, 2018


		await writeFile(fileName, headmatter + body);

		console.log('Generated:', path.relative(cwd, fileName));

Travis optimization #2404

Travis optimization #2404

Conversation

Munter commented Jul 28, 2018 • edited

montogeek commented Jul 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Munter commented Jul 29, 2018

Munter commented Jul 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

byzyk commented Jul 29, 2018

montogeek commented Jul 31, 2018

montogeek commented Jul 31, 2018

Munter commented Jul 31, 2018

montogeek commented Jul 31, 2018

montogeek commented Aug 2, 2018

Munter commented Jul 28, 2018 •

edited