Evaluate build cache for Windows builds

[ChristianKuehnel]

Goal

Find a build cache tool that works with the Visual Studio compile on our build agents.

Context

The Windows build is quite slow at the moment (32 core, local SSD):

• ninja all: 27 min
• ninja check-all: 13 min (after ninja all)

On Linux ccache brings a major decrease in compile times: from ~20 min (clean ninja all) down to ~1 min (fully cached). It would be good, to have something similar for Windows as well.

You can use the existing benchmark script to measure the speedup.

Tools I found so far:

• https://stashed.io/
  • commercial, with discounts for "non-profits"
• https://github.com/frerich/clcache
  • OSS
  • seems to be unmaintained
  • see consider using sscache #38
• https://www.incredibuild.com/
  • commercial
• https://github.com/mozilla/sccache
  • OSS
  • not sure if it works with VisualStudio
  • we could also use it for Linux builds
• https://github.com/mbitsnbites/buildcache
  • OSS

Requirements

• works with Visual Studio
• works in Windows docker container(s)
• nice to have:
  • tool is free for LLVM (no hassle with licence payments)
  • supports distributed builds

Alternatives

• use more build machines instead of speeding up builds
• use gn/goma (see Try goma/RBE to speed up the builds #20)

[ChristianKuehnel]

Results so far:

• Goma/RBE
  • tied to google infrastructure
  • tricky to set up, not much documentation
• sccache (with local caching)
  • is easy to set up and speeds up builds
  • has problems with preprocessing the code (Pre-processor error in multiline string mozilla/sccache#720). This needs to be resolved before we can use it.
  • first measurements with ninja all :
    • clean build: 27 min
    • fully cached build: 14 min
    • size of cache: ~500 MB

[ChristianKuehnel]

The preprocessor with sccache on Windows seem to be a compiler bug: https://developercommunity.visualstudio.com/content/problem/67300/stringifying-raw-string-literal.html

Updating to VS2019 and trying again...

[ChristianKuehnel]

Preparing rollout to beta testers: #162
This is using local sccache, no distributed caching yet.

[ChristianKuehnel]

using sccache builds now for beta testers