Make CI builds faster

[Hoverbear]

This PR explores how to make the Circle CI builds faster.

Diagnosis

This is not necessarily a new problem, and does not seem to be caused by #2993:

• Circle 1.0 https://circleci.com/gh/pingcap/tikv/10072
• Circle 2.0 https://circleci.com/gh/pingcap/tikv/10119

Currently caching is a massive portion of the build, taking around 21 minutes (nearly half the build time):

Implementation

• Follow caching configuration from https://docs.travis-ci.com/user/caching/#Rust-Cargo-cache
• Split format and test into different builds.
  • Format only requires the Rust toolchain and can be done in a matter of a couple minutes.
  • Testing does not require rusftmt.
  • Splitting off clippy and bench might help as well but since they need to do the build compile of dependencies I suspect it might be even longer since it would add queuing issues.

Failed Experiments

• Tried sccache however it didn't yield tangible build tiime improvements.

    Finished dev [unoptimized + debuginfo] target(s) in 409.90 secs
RUSTC_WRAPPER="sccache" cargo build  1642.19s user 151.41s system 437% cpu 6:50.27 total
# ...
    Finished dev [unoptimized + debuginfo] target(s) in 434.92 secs
RUSTC_WRAPPER="sccache" cargo build  1699.46s user 158.76s system 426% cpu 7:15.30 tota
# 
    Finished dev [unoptimized + debuginfo] target(s) in 395.50 secs
RUSTC_WRAPPER="sccache" cargo build  1685.84s user 159.83s system 466% cpu 6:35.94 tota

• Docker builds aren't working since we're using the default machine size (https://circleci.com/docs/2.0/configuration-reference/#resource_class) which only has 4GB of RAM. RocksDB gets killed.

[23102.855272] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[23102.855360] [23609]     0 23609     7678     5173      20       5        0         -1000 circleci
[23102.855364] [24107] 165536 24107     1157       17       8       3        0             0 sh
[23102.855367] [24205] 165536 24205    15439    11691      34       5        0             0 picard
[23102.855411] [74133] 165536 74133   150931     9060     265       4        0             0 cargo
[23102.855413] [80263] 165536 80263     4627      114      16       3        0             0 bash
[23102.855419] [93708] 165536 93708     4549       86      13       3        0             0 build-script-bu
[23102.855459] [97626] 165536 97626    43647      376      69       3        0             0 cmake
[23102.855461] [97627] 165536 97627     1965      322      11       3        0             0 make
[23102.855464] [97629] 165536 97629     1839      169       9       3        0             0 make
[23102.855466] [97632] 165536 97632     1839      169       9       3        0             0 make
[23102.855469] [97690] 165536 97690     2515      865      10       3        0             0 make
[23102.855471] [97783] 165536 97783     1411       18       8       3        0             0 c++
[23102.855473] [97784] 165536 97784     1411       17       8       3        0             0 c++
[23102.855476] [97786] 165536 97786     1411       17       8       3        0             0 c++
[23102.855478] [97787] 165536 97787     1411       18       8       3        0             0 c++
[23102.855480] [97840] 165536 97840    83909    71535     166       3        0             0 cc1plus
[23102.855482] [97841] 165536 97841     1411       18       7       3        0             0 c++
[23102.855484] [97846] 165536 97846    86771    74115     172       3        0             0 cc1plus
[23102.855487] [97847] 165536 97847     1411       18       8       3        0             0 c++
[23102.855489] [97848] 165536 97848     1411       18       8       3        0             0 c++
[23102.855491] [97854] 165536 97854     1411       18       9       3        0             0 c++
[23102.855493] [97857] 165536 97857    86774    74107     172       4        0             0 cc1plus
[23102.855495] [97858] 165536 97858     1411       17       8       3        0             0 c++
[23102.855497] [97876] 165536 97876    86768    74097     172       4        0             0 cc1plus
[23102.855499] [97887] 165536 97887     1411       18       8       3        0             0 c++
[23102.855502] [97888] 165536 97888    86229    73519     170       4        0             0 cc1plus
[23102.855504] [97892] 165536 97892     1411       18       9       3        0             0 c++
[23102.855506] [97893] 165536 97893    84909    72621     168       3        0             0 cc1plus
[23102.855508] [97896] 165536 97896    86778    74176     171       3        0             0 cc1plus
[23102.855511] [97923] 165536 97923     1411       18       8       3        0             0 c++
[23102.855514] [97926] 165536 97926    86752    74130     172       3        0             0 cc1plus
[23102.855516] [97927] 165536 97927    83958    71273     167       3        0             0 cc1plus
[23102.855518] [97928] 165536 97928    82282    69495     163       3        0             0 cc1plus
[23102.855520] [97929] 165536 97929    86808    74114     171       3        0             0 cc1plus
[23102.855522] [97931] 165536 97931    86776    74135     171       3        0             0 cc1plus
[23102.855524] [98759] 165536 98759     1411       17       8       3        0             0 c++
[23102.855526] [98760] 165536 98760    84941    72276     168       3        0             0 cc1plus
[23102.855529] [98764] 165536 98764     1411       18       8       3        0             0 c++
[23102.855531] [98765] 165536 98765    86210    73304     169       3        0             0 cc1plus
[23102.855556] [102764] 165536 102764     9158      116      22       3        0             0 top
[23102.855567] Memory cgroup out of memory: Kill process 97896 (cc1plus) score 70 or sacrifice child
[23102.863076] Killed process 97896 (cc1plus) total-vm:347112kB, anon-rss:296704kB, file-rss:0kB

[siddontang]

if we have no cache, seem we need to build TiKV every time?

[Hoverbear]

Yes if we disable cache entirely it will require a full rebuild. I'm going to try to find a good balance here so we're not seeing 21 minutes of caching but it also doesn't blow up the build time.

[Hoverbear]

Entertainingly, disabling cache entirely doesn't slow down the build. :) https://circleci.com/gh/pingcap/tikv/10157

[Hoverbear]

Hm, I was trying to do docker based builds but rocksdb seems to get killed.

[breezewish]

Do we have any updates about this attempt?

[Hoverbear]

@breeswish I've put it on hold as we've discussed moving some of this over to Jenkins. CircleCI 2.0 is giving some issues with memory and queuing. TravisCI has some issues as well, but they can be resolved I think after talking to one of the engineers there I know.

@siddontang should work on this be continued?

[breezewish]

I looked through sccache's documentation, it seems that when using sccache, target cache is not needed any more, i.e. we choice either to cache target directory, or to use sccache. If sccache is faster than caching & restoring target directory, I guess we can have a try,

[breezewish]

Also I found out that, for a fresh build in fact the target directory is only about 3GB. Notice that in CI it reports a size of 15GB. This is because there are legacy built binaries and as we build more and more times the accumulated cache size becomes larger.

[breezewish]

I'm playing with my ideas on this PR #3247 :)

[Hoverbear]

Closing in favor of #3247