introduce file system watching features to the zig build system

[andrewrk]

Feature Explanation

  --watch                      Continuously rebuild when source files are modified
  --debounce <ms>              Delay before rebuilding after changed file detected

Uses the build system's perfect knowledge of all file system inputs to the pipeline to keep the build runner alive after completion, watching the minimal number of directories in order to trigger re-running only the dirty steps from the graph.

Default debounce time is 50ms but this is configurable. It helps prevent wasted rebuilds when source files are changed in rapid succession, for example when saving with vim it does not do an atomic rename into place but actually deletes the destination file before writing it again, causing a brief period of invalid state, which would cause a build failure without debouncing (it would be followed by a successful build, but it's annoying to experience the temporary build failure regardless).

The purpose of this feature is to reduce latency between editing and debugging in the development cycle. In large projects, the cache system must call fstat on a large number of files even when it is a cache hit. File system watching allows more efficient detection of stale pipeline steps.

Mainly this is motivated by incremental compilation landing soon, so that we can keep the compiler running and responding to source code changes as fast as possible. In this case, also keeping the rest of the build pipeline up-to-date is table stakes.

This also paves the road towards #68. A Run step combined with --watch connects file system updates directly to new code inside an already-running executable. It takes steps closer to more advanced use cases as well:

• IDE plugin running zig build as a child process, speaking a compiler protocol (Implement a server in the compiler that serves information about the compilation #615) to learn about type information, perform refactors, request rebuilds, receive errors, etc. The protocol will multiplex between an arbitrary number of Compile steps, each with a running instance of the compiler.
• An application, compiled in debug mode, speaking the build runner protocol, able to poll on a file descriptor and learn when a hot swap is available, requesting it when it is convenient, or learning about new installed asset updates occurring.

Run step asciinema demo This demo only shows 1/2 terminals used, but in the other window I'm editing assembly files with vim and saving them.

Compile step asciinema demo - getting quick compile error feedback. Incremental compilation is not done yet, this is full compiler rebuilds, but skipping codegen.

unit test workflow asciinema demo

Migration Guide

If you were using WriteFile for its ability to update source files, that functionality has been extracted into a separate step called UpdateSourceFiles. Everything else is the same, so migration looks like this:

-    const copy_zig_h = b.addWriteFiles();
+    const copy_zig_h = b.addUpdateSourceFiles();

If you were using a RemoveDir step, it now takes a LazyPath instead of []const u8. Probably your migration looks like this:

-        const cleanup = b.addRemoveDirTree(tmp_path);
+        const cleanup = b.addRemoveDirTree(.{ .cwd_relative = tmp_path });

However, please consider not choosing a temporary path at configure time as this is somewhat brittle when it comes to running your build pipeline concurrently.

Follow-Up Work

• Build Runner: Implement File System Watching on Windows #20598
• Build Runner: Implement File System Watching for kqueue #20599
• build runner: make watched Compile steps keep the compiler alive #20600
• --watch does not pick up sub-Compilation file system inputs such as compiler_rt, glibc, or musl source files #20601
• make zig build --watch detect modifications to the build script and rerun itself #20602
• rework build system memory allocations  #20603
• open a communication channel between the build system and application #20604
• Implement a server in the compiler that serves information about the compilation #615
• ability to register file system input dependencies during the configure phase #20605
• zig build test-cases --watch does not pick up changes to source files #20606
• std.Build.Step.Run: avoid file system watches when already tracked by step dependency edges #20631
• emit the file_system_inputs message early so that even when the compiler crashes,
  we still know the set of files that trigger recompilation.
• build --watch panics on debian linux (name_to_handle_at returns .INVAL) #20720