Total build graph#2551
Merged
Merged
Conversation
added 8 commits
December 8, 2025 15:49
In order to compile a project with dependencies (and in turn their dependencies) we used to invoke ourselves, i.e. start another actonc process to first compile the dependency projects (which in turn would invoke a sub-compiler in case they had dependencies). Once done, by carefully assembling a search-path, we could reach all the .ty files and in the final build step, the build.zig and generated C source code of all dependencies necessary in order to link the final binaries. We now perform all that work the proper way, staying within a single compiler instance, by first constructing a total build graph; a DAG that includes the modules of all projects. Just like before, we compile modules concurrently up to the maximum number of capabilities (Haskell speak for worker threads) and our critical-path cost approximation is still active, but can now track the cost along the whole path, from the cost of modules within dependencies of dependencies, which should yield a better total order potentially. It's also interesting to note that in dependency projects, we won't necessarily compile all modules - only those that have been chased down by an import. Pure test files for example, should not be needed and thus won't be compiled. Some of the output has been updated to reflect in which project a module is to make it easier to distinguish. I've left it at this point though - I think we should do a major revamp of this now that we do concurrent compilation across projects. It deserves some proper love!
If we find multiple different versions of a dependency, we now override it using the left-most, i.e. anything that we have found earlier which typically means that dependencies listed in the main project or higher up dependencies will override those of lower deps. This ensures a consistent build and avoids various build errors. A warning is printed when a conflict is overridden. Matching on name is really too naive in order to match up actual conflicts, but we don't have anything better for the time being and it's frequent enough of a problem that this is worth it right now.
actonc can now fetch dependencies. This is built into the normal build pipeline and is also exposed as `acton fetch`, mirroring `acton fetch`.
This shows package dependencies and their hashes, for example: ➜ sorespo git:(main) ✗ actonc pkg show Dependency tree (hash overrides shown): - actmf (hash=1220f8aaed03905011ac50f85a48a4c13174b2ac2d7117b1505e76df26d98b610935) - yang (hash=12201fe035e306af5ff4165a58ff3d4d6df4e52bf4c58d58434f5e8c2582c1c28dd8 overridden -> hash=12203f3f7bd27a8671772faf7dd95139460e2daea747a32399db07feba176ab9ffe9) - netconf (hash=1220fb3b2d2a8f69c06d8674bb774659219fbea001b802c1dcad32ab0b7bcdd31d0b) - orchestron (hash=122067a79c931a0b1c0bceba98ade4d838f017c3eda896b1ffe4148dbc5b6f17374a) - netconf (hash=1220fb3b2d2a8f69c06d8674bb774659219fbea001b802c1dcad32ab0b7bcdd31d0b) - yang (hash=12201fe035e306af5ff4165a58ff3d4d6df4e52bf4c58d58434f5e8c2582c1c28dd8 overridden -> hash=12203f3f7bd27a8671772faf7dd95139460e2daea747a32399db07feba176ab9ffe9) - rtrsecrets (hash=1220fc5ebc12b8ae7fee4dae6ba674e49e6ef9421ca0484b36c9a138eb5c428fb0eb) - yang (hash=12203f3f7bd27a8671772faf7dd95139460e2daea747a32399db07feba176ab9ffe9) ➜ sorespo git:(main) ✗
We statically bump up the size for the GC nursery / generation 0 to 64MB which yields quite a good performance improvement. We already did this when invoking actonc from acton, but now it's done statically so direct actonc calls are also faster. Also, the LSP-server gets the same plus threads are enabled since we want to start running tasks in threads or at the very least experiment with it.
We now use the new total build graph in actonc as the default way to build projects. We also have `acton build --old-build` for the old builder - not really sure why anyone would want that, but I guess we'll keep it for a bit before cleaning it away. The biggest change is probably that actonc properly overrides deps on conflicts so that we get a consistent set of dependencies across the build graph.
Just like acton supports using --dep to override a declared dependency with the path to a local dep, actonc now does the same!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In order to compile a project with dependencies (and in turn their dependencies) we used to invoke ourselves, i.e. start another actonc process to first compile the dependency projects (which in turn would invoke a sub-compiler in case they had dependencies). Once done, by carefully assembling a search-path, we could reach all the .ty files and in the final build step, the build.zig and generated C source code of all dependencies necessary in order to link the final binaries.
We now perform all that work the proper way, staying within a single compiler instance, by first constructing a total build graph; a DAG that includes the modules of all projects. Just like before, we compile modules concurrently up to the maximum number of capabilities (Haskell speak for worker threads) and our critical-path cost approximation is still active, but can now track the cost along the whole path, from the cost of modules within dependencies of dependencies, which should yield a better total order potentially. It's also interesting to note that in dependency projects, we won't necessarily compile all modules - only those that have been chased down by an import. Pure test files for example, should not be needed and thus won't be compiled. All the work is performed in actonc.
acton buildhas been redirected to invokeactonc build. It is still possible to get the old build viaacton build --old-buildSome of the output has been updated to reflect in which project a module is to make it easier to distinguish. I've left it at this point though - I think we should do a major revamp of this now that we do concurrent compilation across projects. It deserves some proper love!
If we find multiple different versions of a dependency, we now override it using the left-most, i.e. anything that we have found earlier which typically means that dependencies listed in the main project or higher up dependencies will override those of lower deps. This ensures a consistent build avoids various build errors. A warning is printed when a conflict is overridden.
Matching on name is really too naive in order to match up actual conflicts, but we don't have anything better for the time being and it's frequent enough of a problem that this is worth it right now.
actonc can also fetch dependencies and there is a new
actonc pkg showThe GC nursery / generation 0 is more consistently bumped up by specifying it in package.yaml(.in) and not just relying on the acton.act invocation.