Lots of cache misses building servo

[luser]

I did a full build of servo with sccache, and then ran ./mach clean and rebuilt, but it took a long time and the stats afterward showed:

Compile requests                                                                       283
Compile requests executed                                                              262
Cache hits                                                                             183
Cache misses                                                                            74

This should be able to get a 100% cache hit rate unless there's something funny going on. I suppose it's possible that the C/C++ dependencies that are being compiled are not reproducible, so they're causing some intermediate crates to be cache misses?

[luser]

After digging a bit more, it looks like the ordering of -L native=/path is not stable between compiles. cc @alexcrichton

[luser]

Additionally: the order of --cfg are not stable.

[luser]

I fixed the ordering of -L / --cfg in 580c393 . I also fixed a bug where sccache wouldn't cache generated dep-info if --extra-filename wasn't specified which was being tickled in the servo build: ca9d7f7 .

I'm still seeing a lot of cache misses, I need to dig in more.

[alexcrichton]

FWIW sorting --cfg is semantically ok, but sorting -L is semantically an error I believe b/c if there's two native libraries to be found then -L dictates which one is found (first one wins).

I believe this is a bug that needs to be fixed in Cargo (-L) and seems reasonable to fix --cfg while we're at it.

[luser]

This makes sense. FWIW I'm only sorting them for the purposes of hashing the commandline arguments. For executing the compiler we still use the original ordering.

[luser]

I added an even-more-verbose set of logs to write out all the cache inputs for a crate to a file, and I noticed a few things:

1. I think --extern arguments are not consistently ordered.
2. gl_generator generated bindings don't seem to be deterministic.

[luser]

Filed the gl_generator issue: brendanzab/gl-rs#414.

[luser]

I think --extern arguments are not consistently ordered.

So I already knew this, and have code to sort the --extern arguments when hashing the commandline arguments, but we also save the list of extern files and feed the hashes of those files as input to the hash, and those filenames don't get sorted, so we have an unstable ordering. Sorting externs at the end of parse_arguments is enough to fix that.

With that change + my change to gl_generator I can build webrender and get 100% cache hits! I'll try it on servo next.

[luser]

I fixed the handling of externs in 7ac81f2.

[luser]

Better!

Compile requests               303
Compile requests executed      280
Cache hits                     279
Cache misses                     1
<...>
Non-cacheable calls             23

It would be interesting to look at what those 23 non-cacheable calls are. The Servo build still spent a bunch of time compiling stuff like mozjs-sys, but I didn't have ccache enabled in my .servobuild, so I'll try using ccache=sccache there as well.

[luser]

In any event, I think this particular issue is fixed.

[luser]

Building servo (from servo/servo@61d64da) with the latest sccache, I now see:

Compile requests               930
Compile requests executed      735
Cache hits                     735
Cache misses                     0
Non-cacheable calls            163
Non-compilation calls           32

So that's pretty good! Of the non-cacheable calls, most of them are C/C++, and most of those are things autoconf is doing. I opened a few new issues on some simple things we could fix to support things I noticed in the logs, but they're not going to buy us much. The breakdown of the non-cacheable calls is:

$ rg CannotCache /tmp/sccache.log | sed -re 's/.*CannotCache\((.+)\).*/\1/' | sort | uniq -c
     58 crate-type
     36 multiple input files
     50 no output file
      1 stdin
     18 unknown source extension

So there are 58 rustc invocations we refused to cache because of the crate type. Of those, most of them are bin:

$ rg 'parse_arguments.*--crate-type' /tmp/sccache.log | sed -re 's/.*"--crate-type", "([^"]+)".*/\1/' | sort | uniq -c
     50 bin
      1 dylib
    257 lib
      7 proc-macro
      4 rlib

...and most of those are build scripts. There's probably not much more to be had here without doing a lot of hard work to cache linker invocations.

[metajack]

I recreated this today, and the results are improved. We're down to ~50 crate-types (proc-macro and binaries, mostly build.rs) and 1 stdin (not sure what this is) and 18 unknown extension, which are all conftest.C invocations from configure. However we have 32 cache misses, including all the expensive to compile crates. Right now sccache is only saving 4m on a 13m build for me.

[metajack]

"improved" was the wrong word there. The uncachable stuff is improved, but overall we are worse.

[luser]

@metajack Oh, so @glandium found another source of nondeterminism in webrender's build script recently: servo/webrender#1475 .

Did servo pick up that fix? If not, try that and see if it works. With that fix we get 100% cache hits on a Firefox stylo build.