The Hylo Group
Compiler CLI Design for Build System Integration #1892
Replies: 4 comments 8 replies
|
Thank you very much, this is a very nice document, it touches on a lot of aspects! Here is a first batch of comments/questions (I'm still just halfway).
(I didn't fully understand the role of these artifacts yet, so sorry if something doesn't make sense here. Please help me understand) I'm wondering if it's worth grouping the source summary artifacts per file or whether it should be more or less fine-grained. This is probably related to how we implement incremental compilation and cache invalidation, which I'm not sure about yet. If we wanted to compile source files in parallel, we could ask the compiler to compile the whole program but ignore type-checking the bodies of functions outside of a certain file. Everything should get type-checked in the source file in which they are declared. But during type-checking of a single source file, we might ask for a type that's declared in another source file, so we would duplicate work by type-checking whatever is needed there. Do you know how vala/swift solves this? E.g. type-checking a struct requires seeing whether it implements the required trait conformances.
A Make-style depfile or equivalent machine-readable dependency description. (I didn't fully understand the role of this artifact yet, please correct me if I made incorrect assumptions.) Do you mean this as in per-source dependencies on other sources? The problem is that often every source file depends on every other source file in the module. If there is a call to (Side note: maybe it would also be useful to preemptively detect the ambiguity at declaration site. My sense is that this is not possible in general) If my above assumptions are correct, we have almost a complete graph (full of dependencies between almost any two source files).
For IDEs, we develop a language server conforming to the LSP. This server uses the Hylo compiler as a Swift library, so we can provide interaction with low overhead compared to separate CLI process calls for each interaction. I think all Hylo-specific tooling could be written in a similar way, invoking the compiler as a library. Another direction is compiler plugins, when the Hylo compiler would load a shared library to customize/extend some behavior. For build systems, a CLI contract is nice because the majority of the build systems work with CLI contracts, which we should leverage to ease adoption. There are some build systems that are library-based, notably ll-build and ll-build2 that could be great to leverage in the future, if we find out that what we can achieve with existing build systems could be streamlined/optimized further with Hylo-specific knowledge. This presentation goes in depth into the motivations for compiler-specific build systems. For me, the most compelling argument was that if we group the compilation of multiple translation units into one compiler CLI call, we can reuse caches and we can avoid the serialization cost. But maybe this grouping can be approximated well enough with existing build systems by now...
Yess! That's a good call to think about where else should the module identity come from. In Swift, module names are kinda expected to be globally unique across the dependencies of a module. If we depend on FrontEnd from the Hylo compiler, we can't depend on some web-related package also exposing FrontEnd. Now changing these pacakges to expose non-conflicting module names is outside of my locus of control. I'd like to avoid this in one way or another, e.g. by post-hoc renaming of modules at build-system level or allowing to prefix module names by package names (although those have a name squatting problem on another level).
The driver must accept source files explicitly. I can make a PR for this, it should be straightforward. Currently, we scan a given directory or use a given file.
The Hylo toolchain is by default a cross-compiler, and we support any target that LLVM supports. These are currently inputs of the compiler:
The features and cpu can be either generic or more specialized. When building for development, we may want to pass the native CPUs information, but for distribution, we probably want to target a generic CPU and generic feature set. The standard library may be slightly different on different targets (this is often the case in other languages, at least). Specifically, C integer type sizes and operating-system dependent code may need to get generated (or conditionally included) during the configute phase. Even if the compiler supports one-step compile-and-link commands, a separate link step is strongly recommended for build-system integration.What value could we add to the picture on top of existing linkers? Is this only useful for Hylo-specific artifacts, like the module artifact, or do we need the Hylo compiler for linking its produced object files? I guess we could perform some "link-time" optimization on the module artifacts before lowering to LLVM. Another pass of LTO can be performed on the LLVM level. That would be especially useful if we wanted to optimize across language boundaries where all languages lower to LLVM. (I have no experience with this yet)
We could define different kinds of module artifacts. One includes the lowered and locally optimized IR, while the other is only for interfaces. |
|
As a concrete example of how source-summary artifacts and dependency manifests can work together, imagine a simplified dialect of Suppose we have a module Now let’s build this module using the same general approach proposed for Hylo. In the In the For The initial build rule for Now consider what happens during an incremental build. If a change is made only to the body of a function in If, on the other hand, the signature of a function in Of course, Hylo has to account for more than just function signatures, but the underlying idea is similar. It should be able to:
That is the kind of build behavior I have in mind. |
|
Thanks for the explanation! I believe my claim about all source files in a module depending on all other source files in the module still holds. There is no benefit of knowing which files ended up containing the resolved symbols, given that name resolution must look in all files. This is most likely a problem also in C++, but it's a bit hard to compare, since they have separate headers included in all source files explicitly. Imagine this scenario: // a.c
void c(float x) { }// b.c
void main() {
c(3); // overload resolves to void c(float)
}Next up, we add c.c: // c.c
void c(int x) { }Note previously no symbols were used from |
|
Damn, you are right. |
|
There might be some complications regarding source-file level parallelism for compilation, I made a separate thread for that here: https://github.com/orgs/hylo-lang/discussions/1893 |
|
You can record, for each file, a list of the symbols whose introduction/removal in another source file can change overload resolution, then consult that list to decide whether any real work needs to be done. This is essentially recording symbol dependencies as well as file dependencies. |
|
Swift has great documentation about how their incremental compilation works, and a great explanation of how the driver works:
The dependency analysis is something like what Dave mentioned. |
|
aIn the past we have put these documents in the first post of a discussion and revised them as the discussion went along, e.g. #1868. I'm just a little uncomfortable with the document's home being in a gist that's not connected to the project. Do I have the wrong idea here? |
|
I have put the contents into the post. But maybe it would be good to have in a place where multiple authors can edit. |
|
Thanks! The maintainers can edit it, so it's fine for now. Anyways, I think it's best if the original author makes the edits based on the comments, otherwise it might get chaotic |
|
Related old discussion: http://github.com/orgs/hylo-lang/discussions/730 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
On the topic of build system integration, I wrote my recommendations for a compiler CLI. This would give Hylo a simpler and more robust build story than C++20, Rust, or Swift.
RFC: Hylo Compiler CLI Design for Build System Integration
Status: Draft
Audience: Hylo compiler, driver, tooling, and build integration designers
Summary
This document proposes a public CLI contract for the Hylo compiler that enables clean integration with CMake, Ninja, Make, Meson, Bazel, IDEs, and remote or sandboxed build environments.
The central design principle is:
Hylo should avoid the C++ modules model in which the build system must inspect source files to discover which file provides which module. Instead, Hylo source files should be grouped into modules by the build graph, and the compiler should accept that grouping explicitly through its CLI.
The compiler should support both:
Motivation
Hylo's language model is already far more build-system-friendly than C++ modules if it keeps the following properties:
This separates concerns cleanly:
Foo, andThis design eliminates the need for source scanning to discover module ownership. However, if Hylo also wants fast incremental builds and strong parallelization, its compiler CLI must still expose enough structure for build systems to model compilation efficiently.
Why this matters
A good build-facing CLI should make the following possible:
Goals
The Hylo compiler CLI should:
Non-goals
This proposal does not require:
Background and assumptions
This proposal assumes the following Hylo language-level model:
This implies:
Detailed design
Public compiler model
The public CLI should be a stable driver interface, not a thin wrapper over internal compiler phases.
The driver may internally perform many steps, but its public contract should define:
Subcommand names are flexible, but the capabilities described below should exist.
Core artifact model
To support both simple and scalable builds, the compiler should distinguish at least the following artifact roles.
1. Source file
A Hylo source file belonging to some explicit module.
2. Source summary artifact
A per-source-file artifact that contains the declarations needed by peer files in the same module.
Purpose:
This is conceptually similar to Vala's
fast-vapi, but adapted to Hylo's module model.3. Module artifact
A per-module artifact consumed by external modules.
Purpose:
4. Object file
A native code artifact for linking or archiving.
5. Dependency manifest
A Make-style depfile or equivalent machine-readable dependency description.
Purpose:
6. Diagnostics artifact
Optional but strongly recommended.
Purpose:
The exact file extensions are not important. The artifact roles are.
Fundamental CLI requirements
Explicit module identity
The driver must accept the module identity explicitly.
For example, conceptually:
--module-name FooModule identity must not be inferred from source contents.
Explicit source lists
The driver must accept source files explicitly.
Possible forms include:
--source path,--sources-from file, orThis is how the build system defines module membership.
Explicit external module mapping
The driver should accept explicit mappings from imported module names to module artifacts.
For example, conceptually:
--import-module Bar=path/to/Bar.module--import-module Baz=path/to/Baz.moduleThis is preferable to correctness depending on ambient search paths.
Search paths may still exist as convenience features, but build systems should not need them for correctness.
Explicit output paths
The driver must allow the build system to specify output paths for every artifact it may emit.
At minimum:
Build systems should never need to guess filenames.
Response file support
The driver must support response files for:
This is required for portability and for large modules.
Content-stable outputs
The compiler should avoid rewriting outputs when semantic contents have not changed.
This is especially important for:
This prevents rebuild cascades and works well with Ninja's
restatbehavior.Hermetic semantics
Correctness should depend only on:
The compiler may use caches, but caches should be:
Required command capabilities
The exact subcommand names are illustrative. The important part is that the driver support equivalent capabilities.
toolchain-infoA machine-readable query command that reports compiler and target capabilities.
Recommended information includes:
JSON output is strongly recommended.
This is especially useful for CMake-style compiler detection.
emit-summaryGenerate a per-source summary artifact.
Inputs:
Outputs:
Recommendation
emit-summaryshould be as source-local as possible.Ideally it should not require:
That property enables maximum parallelism.
emit-module-artifactGenerate the externally importable artifact for a module.
Inputs:
Outputs:
This step should not require object code, so downstream module builds can start as early as possible.
compile-sourceCompile one source file within a module using peer summaries.
Inputs:
Outputs:
This command is the key to scalable builds.
Recommendation
The compiler should ideally record only the summaries and external module artifacts actually used by the source file. That gives better incremental precision than conservatively depending on everything.
compile-moduleCompile a whole module in one invocation.
Inputs:
Outputs:
This is the simple mode and should remain a first-class interface.
linkPerform final archive or link creation.
Inputs:
Outputs:
Even if the compiler supports one-step compile-and-link commands, a separate link step is strongly recommended for build-system integration.
Build modes
Mode A: simple whole-module mode
In this mode, one invocation compiles an entire module.
Advantages
Disadvantages
Recommended use cases
Mode B: scalable incremental mode
In this mode, the build pipeline is decomposed into explicit phases.
Recommended pipeline
Phase 1: emit summaries
For each source file of module
Foo:emit-summaryThis phase should be fully parallel.
Phase 2: emit module artifact
After summaries are ready:
emit-module-artifactThis should produce the artifact imported by downstream modules.
Phase 3: compile sources
For each source file of module
Foo:compile-sourceusing:This phase should also be parallel.
Phase 4: link/archive
linkAdvantages
Disadvantages
Dependency model
Inter-module dependencies
These should be declared by the build system.
For example:
Foodepends on modulesBarandBazThis is the natural level for CMake, Meson, Bazel, and similar systems.
The compiler should accept explicit
name=artifactmappings for such dependencies.Intra-module dependencies
These may be tracked through summaries and depfiles.
For example:
a.odepends onb.summary,a.odepends onc.summary,a.odepends onBar.module.The build system should not need to derive this by parsing Hylo source text.
No required ownership-discovery scan
This proposal explicitly rejects a required source-scanning phase to answer:
Foo, orThat information should already be known from the build graph.
What is still useful
Compiler-emitted dependency reporting is still valuable for incrementality, for example:
That is dependency reporting, not module ownership discovery.
Requirements for fast incremental builds
To support fast edit-build cycles, the compiler should provide the following properties.
Peer-visible summary stability
A source summary should change only when declarations visible to peer files change.
Implementation-only changes should not perturb peer summaries.
Public module artifact stability
A module artifact should change only when the externally visible interface changes.
Private implementation changes should not force downstream module rebuilds.
Fine-grained dependency reporting
Each
compile-sourcestep should ideally record only the summaries and external module artifacts it truly depended on.Deterministic byproducts
Side artifacts should be explicit, deterministic, and suitable for declaration in a build graph.
No mandatory hidden daemon state
A daemon mode may improve latency, but correctness must not depend on it.
Requirements for build parallelization
To support strong build parallelism, the compiler design should make the following true.
Summary generation is parallel
emit-summaryfor different source files should be independent.Module artifact generation is early
emit-module-artifactshould depend on semantic summaries rather than object code, so downstream modules can begin as early as possible.Source compilation is parallel
compile-sourceshould depend on peer summaries and imported module artifacts, not on peer full source files.Module graph is acyclic
Hylo should strongly prefer or require acyclic module import graphs.
Diagnostics and machine-readable output
The CLI should clearly separate human-facing and machine-facing output.
Recommendations
JSON output is strongly recommended for:
Build-system implications
CMake
The natural mapping is:
compile-module,emit-summary,emit-module-artifact,compile-source,link.CMake benefits most from:
Ninja
Ninja benefits most from:
restat-friendly unchanged outputs,Other build systems
Meson, Bazel, Buck, Pants, and remote execution systems benefit from:
Alternatives considered
Alternative 1: source files declare module ownership
This would make Hylo more like C++ modules and would force build systems to inspect source files or invoke scanning phases to discover module providers.
Rejected because
Alternative 2: whole-module-only compiler model
Hylo could support only one-step whole-module compilation.
Advantages
Rejected as sole model because
Whole-module mode remains valuable, but not as the only mode.
Alternative 3: implicit filesystem-only module lookup
The compiler could search for imported modules by name using search roots only.
Rejected as primary correctness model because
Search paths may remain a convenience feature, but should not be required for correctness in build-system-driven workflows.
Open questions
The following questions remain to be resolved by Hylo language and compiler design.
1. What exactly belongs in a source summary?
The summary should contain enough information for peer semantic analysis but should not change for irrelevant implementation edits.
2. How should public vs private module dependencies be represented?
If Hylo supports re-exported imports, the module artifact should record them explicitly.
3. Can
emit-summarybe entirely source-local?That is ideal for parallelism. If not, the compiler should minimize cross-file coupling.
4. What object and module artifact forms are canonical?
The exact file types do not matter much, but the public CLI must define them clearly enough for build tools.
5. How much of the dependency graph should be reported conservatively vs precisely?
A conservative first implementation may be acceptable, but precise dependency reporting is desirable long-term.
6. Should the compiler optionally expose a daemon mode?
If yes, the public driver contract should remain correct and sufficient without one.
Suggested command shapes
The following are illustrative only.
Whole-module mode
hylo compile-module --module-name Foo --source a.hylo --source b.hylo --import-module Bar=... --module-output Foo.module --object-dir obj/Summary generation
hylo emit-summary --module-name Foo --source a.hylo --summary-output a.summary --depfile a.summary.dModule artifact generation
hylo emit-module-artifact --module-name Foo --summary a.summary --summary b.summary --output Foo.module --depfile Foo.module.dPer-source compilation
hylo compile-source --module-name Foo --source a.hylo --peer-summary a.summary --peer-summary b.summary --import-module Bar=... --object-output a.o --depfile a.o.dLink
hylo link --kind executable --object a.o --object b.o --output Foo.exeAgain, the exact spellings are less important than the semantics.
Decision
This RFC recommends that Hylo define a stable compiler driver CLI with the following properties:
Rationale
This design gives Hylo:
Conclusion
Hylo is in a strong position to avoid the main build-system integration problems that affected C++ modules.
The most important decision is to keep module membership outside the source text and inside the build graph. Once that is fixed, the compiler CLI should expose enough structured phases and artifacts to support both simple and scalable builds.
In one sentence:
Appendix: checklist
Must-have
Strongly recommended
Avoid
Revision history
All reactions