diff --git a/Documentation/package-dependency-flow.md b/Documentation/package-dependency-flow.md new file mode 100644 index 0000000000..7a892d4ead --- /dev/null +++ b/Documentation/package-dependency-flow.md @@ -0,0 +1,161 @@ +# Package Dependency Flow + +This document describes how package dependencies are handled within source build. +It describes the mechanisms that exist to control which package versions are used. + +## Origins of Packages + +A source build must be self-contained, meaning the entire product must be built +from source in an offline environment. To achieve this, all packages dependencies must +be satisfied by one of the following: + +### Source-Build-Reference-Packages + +The first repo that is built as part of source build is +[source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages). +This repo contains all of the reference packages used to build the product. The repo +contains [tooling to generate new reference packages](https://github.com/dotnet/source-build-reference-packages?tab=readme-ov-file#adding-new-packages). + +### Current Source Built Packages + +This refers to all of the packages produced in the current build. The set of packages +available to each repo varies based on its build order. For example the msbuild repo +can take a dependency on the current version of Microsoft.CodeAnalysis from roslyn +because roslyn builds before msbuild. Conversely, since roslyn builds before msbuild, +roslyn cannot take a dependency on the current version of Microsoft.Build; it can only +take a dependency on a previously released version. + +### Previous Source Built Packages + +Because the .NET product uses itself to build, the .NET source build product must be +[bootstrapped](./bootstrapping-guidelines.md). This process allows the packages from the +previous source build release to be used to build the next version of the product. This +provides a means for breaking the product's circular dependencies. For example repos like +[arcade](https://github.com/dotnet/arcade) can self-reference its previous version to +produce the next version. + +When referencing previous source built packages, it is important to not leak these +previously built packages into the resulting packages/product. This is considered a +[poison leak](./leak-detection.md) and is not permitted during a source build as it +breaks the notion of building the product entirely from source. This hinders the +ability to service the product. + +## Package Versions + +Package dependencies that defined using +[Arcade's Darc patterns](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md) +will get lifted dynamically during a source build if the following conditions are met: + +1. The dependency is declared in the Version.Details.xml file. + + **Version.Details.xml** + + ```xml + ... + + https://github.com/dotnet/command-line-api + 02fe27cd6a9b001c8feb7938e6ef4b3799745759 + + ... + ``` + +1. A corresponding version property is defined in the Versions.props. + + **Versions.props** + + ```xml + ... + 2.0.0-beta4.24068.1 + ... + ``` + +1. A repository reference is defined in the + [VMR's project dependency graph](https://github.com/dotnet/dotnet/tree/main/repo-projects). +This reference does not have to be direct, it can be transitive. + + **{VMR repo project}.proj** + + ```xml + ... + + ... + ``` + +When these conditions are met during a source build, the infrastructure will scan +the Version.Details.xml file and dynamically create two new Versions.props files +containing updated version properties for all non-pinned dependencies. + +**PackageVersions.Previous.props:** This will contain version properties with the +package versions from the [previous release of source build](#previous-source-built-packages). +If a new package exists that has never been released before, it will not have a +version property defined. + +```xml +... + 2.0.0-beta3 +... +``` + +**PackageVersions.Current.props:** This will contain version properties with the +package versions from the [current source build](#current-source-built-packages). +If a package comes from a repo that has not been built yet, it will not have a version +property defined. + +```xml +... + 2.0.0-beta4 +... +``` + +These two version.props files get imported by the arcade source build infrastructure after +the repo's Version.props file. Therefore the repo's Versions.props property versions +get overridden by the source build versions. In the case of the `SystemCommandLineVersion` +example, the current source build version, 2.0.0-beta4, would win. All msbuild references +(e.g. project PackageReferences) to these Versions.props properties pick up the newer +versions. This is known as package version lifting since it lifts the originally defined +package version to the current source built version. This behavior only applies to source +build in the context of the [VMR](https://github.com/dotnet/dotnet) (see also +[Repo Level Source Builds](#repo-level-source-builds)). + +### Transitive Version Properties + +Transitive version properties in your Versions.props file may not work as intended with +source build. + +**Versions.props** + +```xml +... + 17.7.0-preview-23217-02 + $(MicrosoftBuildFrameworkVersion) +... +``` + +**Version.Details.xml** + +```xml +... + + https://github.com/dotnet/msbuild + 2cbc8b6aef648cf21c6a68a0dab7fe09a614e475 + + +... +``` + +In this case source build will override the `MicrosoftBuildFrameworkVersion` to the +latest version but the `MicrosoftBuildVersion` will remain set to `17.7.0-preview-23217-02` +because of the property evaluation order. If the desired behavior is for +`MicrosoftBuildVersion` to be set to the same value as `MicrosoftBuildFrameworkVersion` +for source build, then you either need to declare the Microsoft.Build dependency +in the Version.Details.xml file or move the `MicrosoftBuildVersion` assignment outside +of the Versions.props file. + + +### Repo Level Source Builds + +The source build package lifting mechanism is not applicable when building individual +repos in source build mode because it doesn't have the context of the other product +repos or previous source build release. In repo source build mode, the versions of the +packages declared in the Versions.props are used (see also +[backlog issue](https://github.com/dotnet/source-build/issues/3562)).