stage2: introduce CacheMode

[andrewrk]

The two CacheMode values are whole and incremental.
incremental is what we had before; whole is new.
Whole cache mode uses everything as inputs to the cache hash;
and when a hit occurs it skips everything including linking.
This is ideal for when source files change rarely and for backends that
do not have good incremental compilation support, for example
compiler-rt or libc compiled with LLVM with optimizations on.
This is the main motivation for the additional mode, so that we can have
LLVM-optimized compiler-rt/libc builds, without waiting for the LLVM
backend every single time Zig is invoked.

Incremental cache mode hashes only the input file path and a few target
options, intentionally relying on collisions to locate already-existing
build artifacts which can then be incrementally updated.

The bespoke logic for caching stage1 backend build artifacts
is removed since we now have a global caching mechanism for
when we want to cache the entire compilation, including linking.
Previously we had to get "creative" with libs.txt and a special
byte in the hash id to communicate flags, so that when the cached
artifacts were re-linked, we had this information from stage1
even though we didn't actually run it. Now that CacheMode.whole
includes linking, this extra information does not need to be
preserved for cache hits. So although this changeset introduces
complexity, it also removes complexity.

The main trickiness here comes from the inherent differences between the
two modes: incremental wants a directory immediately to operate on,
while whole doesn't know the output directory until the compilation is
complete. This commit deals with this problem mostly inside update(),
where, on a cache miss, it replaces zig_cache_artifact_directory with a
temporary directory, and then renames it into place once the compilation is
complete.

Items remaining before this branch can be merged:

• make sure @import files make it into the cache manifest
• make sure @embedFile files make it into the cache manifest
• make sure C files make it into the cache manifest
• double check that the emit paths of other things besides the binary are working correctly.
• test -fno-emit-bin + -fstage1
• test -femit-bin=foo + -fstage1
• implib emit directory copies bin_file_emit directory in create() and needs to be adjusted to be overridden as well.
• make sure emit-h is handled correctly in the cache hash
• Cache: detect duplicate files added to the manifest
• macOS: object file paths in the binary need to point to the final renamed directory, not the tmp directory
• windows: object file paths in the binary need to point to the final renamed directory, not the tmp directory
• builtin.zig is getting the wrong value for zig_is_stage if you run stage2 behavior tests followed by stage1 behavior tests.

Some preliminary performance measurements of wall clock time and
peak RSS used:

stage1 behavior (1077 tests), llvm backend, release build:

• cold global cache: 4.6s, 1.1 GiB
• warm global cache: 3.4s, 980 MiB

stage2 master branch behavior (575 tests), llvm backend, release build:

• cold global cache: 0.62s, 191 MiB
• warm global cache: 0.40s, 128 MiB

stage2 this branch behavior (575 tests), llvm backend, release build:

• cold global cache: 0.62s, 179 MiB
• warm global cache: 0.27s, 90 MiB

[andrewrk]

The macOS failure looks like this:

Test 1/13 stack-trace return (Debug)...
========= Expected this output: =========
error: TheSkyIsFalling
source.zig:2:5: [address] in main (test)
    return error.TheSkyIsFalling;
    ^

================================================
error: TheSkyIsFalling
???:?:?: 0x107d80082 in _main.0 (???)

error: TestFailed

It can be reproduced with this test.zig file:

pub fn main() !void {
    return error.TheSkyIsFalling;
}

and running this command:

./zig build-exe test.zig --enable-cache
# then run the test binary from the printed directory

The problem is this code here:

zig/lib/std/debug.zig

Line 1424 in 7c1e17a

const o_file_path = mem.sliceTo(self.strings[symbol.ofile..], 0);

The o_file_path that it gets from the binary contains something like zig-cache/tmp/[random]/test.o but with these new CacheMode changes, in whole mode, that directory gets renamed to zig-cache/o/[digest] after linking. So the o_file_path has a wrong absolute path.

I'm not quite sure how the object file paths get set in the binary. Some different ideas to address this problem:

• After renaming the tmp dir to the final cache dir, tell the linker to update the object file paths in the binary.
• We can know the final cache dir before calling flush(); perhaps the file paths can be delayed to be written until flush().
• Adjust the object file paths to be relative to the binary rather than absolute.
• Leave the compilation/linking alone, and introduce logic in our stack trace dumping code to notice when a binary has been moved, and use the new file system destination as a "patch" to adjust object file paths. The downside here is that this only works for zig stack traces, but we want gdb/lldb and other system tools to work as well. So arguably we should explore the other ideas first.

One thing that would help unblock me here would be to have a chat with @kubkon about where exactly is the logic that decides how those object file paths get computed and populated. But that will have to wait until holiday festivities are over 🙂 Happy New Year! 🥳