area/circleci: Speed up circleci build process for branches and pr #16778

briangann · 2019-04-26T05:24:54Z

What this PR does / why we need it:

Build times for branches and pull requests are slow

Special notes for your reviewer:

This is focused on streamlining the PR process by speeding up builds in circleci

bergquist · 2019-04-26T05:36:33Z

Love it!

torkelo · 2019-04-26T06:43:34Z

Yes love it as well, discussed doing this kind of change with @xlson a couple of weeks ago and we both agreed that this was a good idea!

xlson · 2019-04-26T15:10:53Z

Nice work @briangann! Looks like you'll be able to merge build.s and build-all.sh into one file pretty soon as well as speeding up the build a lot.

briangann · 2019-04-26T15:16:00Z

Nice work @briangann! Looks like you'll be able to merge build.s and build-all.sh into one file pretty soon as well as speeding up the build a lot.

Looks possible. I'm looking for any other areas that can be done in parallel, or anything that will benefit from caching result.

xlson · 2019-04-26T17:31:13Z

It looks like you've covered most things right now. It would be great if we could parallelize the frontend build somehow.

briangann · 2019-04-26T17:39:12Z

It looks like you've covered most things right now. It would be great if we could parallelize the frontend build somehow.

definitely room for optimization in the frontend, will work on that next.

the frontend-test take a long time also, will see if that can be split up.

…0426_circleci_speedup

briangann · 2019-04-27T06:10:43Z

the frontend build spends the majority of the time in webpack (linting takes about 8 seconds of that time)

briangann · 2019-04-27T15:16:54Z

build time is now down from 21 minutes to 12 minutes :)

bergquist · 2019-04-29T06:28:26Z

Great work! Any chance we can merge this first and iterate more in other PR's? You can always base the new PR on this branch so you can start now without waiting.

briangann · 2019-04-30T03:22:42Z

adding the webpack speed measurement plugin shows this (for future speedup work)

 SMP  ⏱
General output time took 3 mins, 38.095 secs

 SMP  ⏱  Plugins
TerserPlugin took 2 mins, 30.62 secs
ngAnnotatePlugin took 2 mins, 26.66 secs
HtmlWebpackPlugin took 6.48 secs
OptimizeCssAssetsWebpackPlugin took 4.69 secs
SpeedMeasurePlugin took 2.65 secs
MiniCssExtractPlugin took 0.726 secs
ForkTsCheckerWebpackPlugin took 0.05 secs

 SMP  ⏱  Loaders
modules with no loaders took 26.41 secs
  module count = 1913
css-loader, and
postcss-loader, and
sass-loader took 26.29 secs
  module count = 10
expose-loader, and
expose-loader took 22.19 secs
  module count = 2
ts-loader, and
tslint-loader took 20.81 secs
  module count = 792
mini-css-extract-plugin, and
css-loader, and
postcss-loader, and
sass-loader took 4.6 secs
  module count = 2
ngtemplate-loader, and
html-loader took 3.027 secs
  module count = 137
html-webpack-plugin took 0.027 secs
  module count = 2

xlson

Looks great! I agree with @bergquist, let's merge it.

bergquist

LGTM!

* grafana/master: prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778)

* grafana/master: FormLabel: allow any rather than just a string for tooltip (grafana#16841) prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778)

* grafana/master: FormLabel: allow any rather than just a string for tooltip (grafana#16841) prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778) DataProxy: Restore Set-Cookie header after proxy request (grafana#16838) docs: clarify page parameter version support for folder/dashboard search (grafana#16836) Chore: revise some of the gosec rules (grafana#16713) Refactor: consistant plugin/meta usage (grafana#16834)

* grafana/master: FormLabel: allow any rather than just a string for tooltip (grafana#16841) prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778) DataProxy: Restore Set-Cookie header after proxy request (grafana#16838) docs: clarify page parameter version support for folder/dashboard search (grafana#16836) Chore: revise some of the gosec rules (grafana#16713) Refactor: consistant plugin/meta usage (grafana#16834) Explore: Use SeriesData format for loki/logs (grafana#16793) Refactor: move NavModel to @grafana/ui (grafana#16813)

* grafana/master: (27 commits) FormLabel: allow any rather than just a string for tooltip (grafana#16841) prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778) DataProxy: Restore Set-Cookie header after proxy request (grafana#16838) docs: clarify page parameter version support for folder/dashboard search (grafana#16836) Chore: revise some of the gosec rules (grafana#16713) Refactor: consistant plugin/meta usage (grafana#16834) Explore: Use SeriesData format for loki/logs (grafana#16793) Refactor: move NavModel to @grafana/ui (grafana#16813) Auth: Enable retries and transaction for some db calls for auth tokens (grafana#16785) Provisioning: Show file path of provisioning file in save/delete dialogs (grafana#16706) Add tracing headers for prometheus datasource (grafana#16724) Config: Fixes bug where timeouts for alerting was not parsed correctly (grafana#16784) build: removes gopkg files from dev docker file (grafana#16817) Provisioning: Trying to fix failing test (grafana#16800) Table: React table fix rotate support in storybook (grafana#16816) TestData: add log level in dummy message (grafana#16815) removes gopkg.lock from root folder Explore: Support user timezone (grafana#16469) Plugins: rename vizPlugin to panelPlugin (grafana#16802) ...

…MetricPanelCtrl * grafana/master: (24 commits) CSV: escape quotes in toCSV (grafana#16874) Dashboard: Lazy load out of view panels (grafana#15554) LDAP: Added reload endpoint for LDAP config (grafana#15470) PluginsList: Removed icons and updated snapshots (grafana#16872) Panels: Fixed issue with panel type change and data updates (grafana#16871) Chore: fix modes for non-executable files (grafana#16864) DataSourceSettings: Minor fix for uncontrolled input (grafana#16863) Chore: Lowered implicit anys limit to 5623 TestData: Add dashboards to testdata (grafana#16855) Input Datasource: convert from angular config to react ConfigEditor (grafana#16856) DataSources: minor typescript cleanups and comments (grafana#16860) TestDataDatasource: Add config editor (grafana#16861) App Plugins: support react pages and tabs (grafana#16586) Add Windows MSI generation to build process (grafana#16502) Datasources: add support for POST HTTP verb for InfluxDB (grafana#16690) Add pattern validation in configs (grafana#16837) Search: Enable filtering dashboards in search by current folder (grafana#16790) FormLabel: allow any rather than just a string for tooltip (grafana#16841) prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778) ...

* grafana/master: (44 commits) CSV: escape quotes in toCSV (grafana#16874) Dashboard: Lazy load out of view panels (grafana#15554) LDAP: Added reload endpoint for LDAP config (grafana#15470) PluginsList: Removed icons and updated snapshots (grafana#16872) Panels: Fixed issue with panel type change and data updates (grafana#16871) Chore: fix modes for non-executable files (grafana#16864) DataSourceSettings: Minor fix for uncontrolled input (grafana#16863) Chore: Lowered implicit anys limit to 5623 TestData: Add dashboards to testdata (grafana#16855) Input Datasource: convert from angular config to react ConfigEditor (grafana#16856) DataSources: minor typescript cleanups and comments (grafana#16860) TestDataDatasource: Add config editor (grafana#16861) App Plugins: support react pages and tabs (grafana#16586) Add Windows MSI generation to build process (grafana#16502) Datasources: add support for POST HTTP verb for InfluxDB (grafana#16690) Add pattern validation in configs (grafana#16837) Search: Enable filtering dashboards in search by current folder (grafana#16790) FormLabel: allow any rather than just a string for tooltip (grafana#16841) prometheus: fix regression after adding support for tracing headers (grafana#16829) area/circleci: Speed up circleci build process for branches and pr (grafana#16778) ...

briangann added 4 commits April 26, 2019 00:20

speed up circleci build

2dab761

add build-fast to jobs

b88f7ce

fix script

3008dd5

remove build from jobs

2fb8544

briangann added 6 commits April 26, 2019 01:06

convert to functions and add args for different build types

20db229

fix typo

6003b2e

add frontend

c1797cf

add yarn cache

6bb6bb8

split build fast into pieces

8aa92f9

split build process

b083aed

briangann added 12 commits April 26, 2019 01:47

ensure dist exists for backend build

a799a85

backend should persist bin

fb28193

add build process output

354152d

add debug to frontend output

49ecdc1

additional debug output for frontend build

168137e

more debug output and set path for cache

5b98c85

update frontend build artifacts

2ac4ca9

fix frontend artifact path

29f18a6

attach workspace in package step

b3c94db

package needs node_modules for grunt

77fa9dd

add node_moduels to persist

5ee3386

omit latest

027fdc1

briangann added 2 commits April 26, 2019 10:13

set noninteractive flag for docker build

da34183

add fast option to docker build

d63cf47

briangann added 2 commits April 26, 2019 10:47

update path

3eaea9a

packaing needs grunt

799011c

add checkout to package step

555f712

briangann added 10 commits April 26, 2019 22:12

fix cache

df80fcf

debug workspace

f32b3d2

update cache

8305054

debugging workspace

2bbee8d

more debugging

225021d

add build latest to packaging step

70b649c

fast build should not try to tag non-amd builds

7e3376a

remove debug steps, remove signing step for prs-and-branches

bfc55f6

remove debug code

6ce6f64

Merge branch 'master' of https://github.com/grafana/grafana into 2019…

4e8c04c

…0426_circleci_speedup

frontend also pulls phantomjs, include in workspace

269c436

briangann marked this pull request as ready for review April 30, 2019 03:21

xlson approved these changes Apr 30, 2019

View reviewed changes

bergquist approved these changes Apr 30, 2019

View reviewed changes

briangann merged commit 157f330 into grafana:master May 1, 2019

briangann deleted the 20190426_circleci_speedup branch May 1, 2019 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

area/circleci: Speed up circleci build process for branches and pr #16778

area/circleci: Speed up circleci build process for branches and pr #16778

briangann commented Apr 26, 2019

bergquist commented Apr 26, 2019

torkelo commented Apr 26, 2019

xlson commented Apr 26, 2019

briangann commented Apr 26, 2019

xlson commented Apr 26, 2019

briangann commented Apr 26, 2019 •

edited

briangann commented Apr 27, 2019

briangann commented Apr 27, 2019

bergquist commented Apr 29, 2019

briangann commented Apr 30, 2019

xlson left a comment

bergquist left a comment

area/circleci: Speed up circleci build process for branches and pr #16778

area/circleci: Speed up circleci build process for branches and pr #16778

Conversation

briangann commented Apr 26, 2019

bergquist commented Apr 26, 2019

torkelo commented Apr 26, 2019

xlson commented Apr 26, 2019

briangann commented Apr 26, 2019

xlson commented Apr 26, 2019

briangann commented Apr 26, 2019 • edited

briangann commented Apr 27, 2019

briangann commented Apr 27, 2019

bergquist commented Apr 29, 2019

briangann commented Apr 30, 2019

xlson left a comment

Choose a reason for hiding this comment

bergquist left a comment

Choose a reason for hiding this comment

briangann commented Apr 26, 2019 •

edited