HMake Details

What makes HMake more correct than Ninja?

Ninja does not record the dependency graph itself in its build log. Instead, it parses the current build.ninja file on each run and checks whether any input file is newer than its corresponding output file. As a result, if all dependencies are already built and you modify build.ninja to add or remove a dependency for a target, Ninja will not rebuild that target because the file timestamps remain unchanged.

HMake, by contrast, serializes the dependency graph itself into its build cache. When HMake detects a change in the serialized graph of a target (such as adding or removing a dependency), it automatically marks the target for rebuild.

Ninja is a file-based build system. What is HMake?

HMake is a BTarget-based build system. A BTarget is a custom abstraction that provides advanced scheduling, custom caching, and flexible extension capabilities. While Ninja specifies dependency relationships strictly between files and rebuilds outputs when they are outdated relative to inputs, HMake models dependencies between BTarget objects, allowing it to support complex rules beyond simple file-timestamp checks.

How do you extend BTarget?

You extend BTarget by inheriting from it and overriding its virtual methods to implement custom build behaviors. HMake includes several built-in subclasses:

• CppMod: Manages C++ module compilation.
• CppSrc: Manages ordinary source file compilation.
• LOAT: Handles linking and archiving (Linker Or Archiver Target).
• CppTarget: Orchestrates a group of compile/link targets.

Ninja records command hashes and file times in .ninja_log, and header dependencies in .ninja_deps. What does HMake record in the build cache?

HMake also records command hashes and timestamps, managed directly by the BTarget base class. In addition, HMake allows subclasses to store custom cache payload data. For example, CppSrc caches header file lists, and CppMod caches both header file lists and dynamically discovered dependencies.

In HMake:

• The command hash is part of the cumulativeHash (RealBTarget::cumulativeHash). For instance, CppSrc computes its cumulativeHash by hashing a combined array containing the compile command hash, the source file content hash, and the content hashes of all referenced header files.
• The execution timestamp is recorded as launchTime (RealBTarget::launchTime).

Does CppMod store both header files and dynamic dependencies? What is their storage format?

Yes. Both lists are serialized using the same format: a 4-byte size prefix followed by an array of 4-byte identifiers.

• Header files: The array contains the unique ID (Node::myId, linked at Node::myId) of each header file Node.
• Dynamic dependencies: The array contains the unique cache index (BTarget::cacheIndex, linked at BTarget::cacheIndex) of the dependent targets.

What is BTarget::cacheIndex?

BTarget::cacheIndex is a unique index used to identify a BTarget across multiple configurations and builds. It is mapped from a unique uint64_t hash, BTarget::cacheName, which is stored in the config cache.

At startup, HMake loads the config cache and populates a hash-map mapping cacheName to its corresponding cacheIndex. When a BTarget is constructed, its constructor consults this map using its cacheName to retrieve its stable cacheIndex. This ensures that even if you change the order of target declarations, or add, remove, and then re-add targets, they will consistently resolve to the same cache entries.

How does HMake ensure that each BTarget::cacheName is unique?

At configure-time, HMake validates all target names and errors out if two targets share the same cacheName. For example, in the "Hello World" sample, defining an executable named "app" causes HMake to instantiate:

1. A CppTarget using the hash of "app-cpp" as its cacheName.
2. A LOAT using the hash of "app" as its cacheName.

What is a Node? What is its ID?

A Node is an interned representation of a filesystem path. HMake uses a global hash-map to ensure only a single unique Node instance is created for any given path. Before interning, paths are normalized (and converted to lower-case on Windows to handle case insensitivity).

Each Node has a unique 32-bit ID (Node::myId, linked at Node::myId), which is assigned by incrementing a global counter during construction. This allows HMake to serialize path references compactly. Much like cacheIndex, a given path will consistently map to the same Node::myId across multiple configurations and builds.

How does HMake ensure unique cache names for CppSrc and CppMod?

HMake computes their cacheName by hashing a tuple containing:

1. The source file's Node::myId (Node::myId)
2. The parent target's CppTarget::cacheIndex (BTarget::cacheIndex)
3. The module compilation type (e.g., CppModType, linked at CppModType)

This layout handles several edge cases correctly:

• Multiple source files in one target: Two different source files in the same CppTarget will have different Node::myIds, producing unique cache indices.
• Shared source files across targets: The same source file compiled in two different targets will resolve to different parent cache indices, yielding two distinct CppSrc instances.
• Changing module interface roles: If a source file is changed from a standard module partition to an interface partition, its CppModType changes, generating a different cacheName and ensuring its cache entry does not conflict.

What is RealBTarget::updateStatus? And what is BTarget::setUpdateStatus()?

UpdateStatus is an enum with three values: UNCHECKED, UPDATE_NEEDED, and UPDATE_NOT_NEEDED.

Every RealBTarget's status is initialized to UpdateStatus::UNCHECKED. Calling BTarget::setUpdateStatus() resolves and sets this status to either UPDATE_NEEDED or UPDATE_NOT_NEEDED.

The base implementation of BTarget::setUpdateStatus() behaves as follows:

1. It ensures that setUpdateStatus() has been called recursively on all of the target's dependencies.
2. It checks if any dependency has its status set to UPDATE_NEEDED. If so, this target is also marked UPDATE_NEEDED.
3. It compares timestamps: this target's own launchTime must be newer than the launchTime of all its dependencies. If any dependency is newer, this target is marked UPDATE_NEEDED. This handles the case where a dependency was rebuilt during a previous partial build (e.g. when only a subset of targets was built), ensuring that downstream targets are correctly rebuilt on subsequent full runs.

What is launchTime? Is it similar to the timestamp that Ninja stores in .ninja_log?

Yes, they serve the same functional purpose. It is named launchTime (RealBTarget::launchTime) because HMake records this timestamp immediately before launching the build process.

This ordering is crucial: if a header file is edited after the compiler process starts but before it finishes, the header's modification time will be newer than the target's recorded launchTime. This guarantees that the target will be detected as outdated and rebuilt on the next build.

How is CppMod::setUpdateStatus() implemented?

CppMod::setUpdateStatus() implements incremental checks for C++ modules:

1. It verifies if the output BMI/PCM and .o files exist on disk. If any are missing, it sets the status to UPDATE_NEEDED and exits.
2. It iterates over all dynamically discovered dependencies (from cachedDeps) and ensures setUpdateStatus() has been evaluated for them.
3. If any dynamic dependency is marked UPDATE_NEEDED or has a launchTime newer than this target's own, this target is marked UPDATE_NEEDED and exits.
4. If no dynamic dependency triggers a rebuild, it computes its cumulativeHash (hashing the compile command, source file, and cached headers, similar to CppSrc::setUpdateStatus()).
5. Finally, it invokes ObjectFile::setUpdateStatus(), which delegates to the base BTarget::setUpdateStatus() to compare the cumulativeHash against the cached build footer and check static dependencies.

Why are dynamic module dependencies not added directly to the static build graph? Wouldn't that avoid duplicating check logic between CppMod::setUpdateStatus() and BTarget::setUpdateStatus()?

Dynamic dependencies are discovered during compilation and can change on any source edit. Modifying the static build graph with these dynamic dependencies would cause two major issues:

1. Stale waiting: If a user edits a source file to remove an import/dependency, a static graph using the old cached dependency would still force the builder to wait for that target, leading to unnecessary serialization or blocking.
2. Spurious cycles: Suppose Module A previously depended on Module B. If the user swaps their roles so that Module B now depends on Module A, adding this new edge to a static graph that still holds the old cached dependency would incorrectly report a cycle, blocking the build.

Evaluating dynamic dependencies manually inside CppMod::setUpdateStatus() ensures that HMake always acts on the correct, updated dependency relationships.

What is the format of the build cache for each BTarget?

Each BTarget serializes its build cache in the following order:

1. Build Graph: A list of BTarget::cacheIndex (BTarget::cacheIndex) values representing the target's static dependencies.
2. Custom Build Cache: A custom payload written by the subclass (e.g., cached headers for CppSrc, or headers and dynamic dependencies for CppMod).
3. Build Footer: A fixed 16-byte suffix containing cumulativeHash (8 bytes) and launchTime (8 bytes) for process-launching targets.