Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
dyndep: dynamically discovered dependencies for Fortran and C++20 modules #1521
Fortran modules (and likely C++20 modules) require sources that provide modules to be compiled before sources that consume modules. Therefore the content of the source files defines dependencies. Unlike preprocessor dependencies (handled by
This PR includes the full implementation, documentation, and thorough tests. The changes have been used in practice by CMake users for years (via a Kitware-maintained branch of ninja). They have been used to build large Fortran projects with thousands of sources and hundreds of modules.
The feature works by providing a new
In this example the order-only dependencies ensure that
The "dyndep" file specifies additional inputs and outputs for the relevant build statements beyond those specified in the build manifest.
See documentation added by this PR for more details and examples.
Please forgive me if I'm overstepping here, it seems like a lot of good work has gone into this - but is it necessary to require
@mqudsi I did think about that while designing this but decided against having dependencies specified in
@dublet CMake 3.7 and higher already have support. It is activated by recognizing a special
-const char* kNinjaVersion = "1.9.0.git"; +const char* kNinjaVersion = "1.9.0.git.dyndep-1";
Of course once Ninja integrates this PR in a release then we can update CMake to check for that instead.
Add an 'err' string argument and return a boolean for success. Update call sites to pass an 'err' string argument and check the return value. This will be useful later for adding logic to these methods that may fail.
In order to later support dynamic updates to the build plan while building, the Plan will need access to its Builder. Since this access will be needed only for specific features we can avoid updating all Plan constructions in the test suite by making this access optional.
Create a Parser base class that holds parser functionality not specific to the build manifest file format. This will allow it to be re-used for other parsers later.
Move the logic to mark edges as wanted over to a Plan::EdgeWanted method so it can be re-used elsewhere later.
Define a file format suitable for specifying dynamically-discovered dependency information for build edges. Design a format inspired by the build manifest format and using the same lexer. Start with a required format version specification followed by "build" statements that add implicit inputs and outputs to existing edges.
Allow rules or build statements to specify one of the build statement inputs in a "dyndep" binding. This will later be used to load dependency information from the specified file.
Add a LoadDyndeps method to load a dyndep file and update the edges that name it in their dyndep binding.
The full readiness of a node that has a dyndep binding cannot be known until after the dyndep file is loaded. If a dyndep file is ready while constructing the build plan it can be loaded immediately so full information can be used to decide whether anything needs to be built. If a dyndep file is not ready while constructing the build plan then the edges naming it cannot be ready either because the dyndep file is one of their inputs. In this case we defer loading the dyndep file until the build plan is being executed.
After finishing an edge that produces a dyndep file, load the file and update the build graph structure. Recompute the dirty state of all its dependents and of newly reachable portions of the graph. Add edges to the build plan that are discovered to be wanted. Finally, schedule edges that are wanted and now ready to build.
Show a simple example of Fortran module dependencies (this use case motivated the entire dyndep feature). Also show an example of tarball extraction, a case that few other buildsystems can handle cleanly.
Replace our single active edge pointer with a vector and add a parameter that tests can set to limit the number of concurrent edges. Set the default to 1 to preserve the current behavior. Specific tests will be able to override it later to simulate concurrent builds.
Some outputs may not be known in the main build manifest and are instead discovered through a dyndep binding. Load dyndep files that are available during cleaning so that we can clean these outputs too.
`Cleaner` provides two constructors that are the same except that one constructs a "real" disk interface internally and the other takes a caller-provided disk interface. A real disk interface is already available at the only call site for the former constructor. Use it directly and drop the unnecessary constructor variant.
Teach the `-t graph` tool to load dyndep files because they are part of the build graph. Issue a warning when the dyndep file cannot be loaded cleanly. This will help users visualize the complete build graph.