Unified Build design - Managing multiple SDK bands in the VMR

[premun]

Context

The dotnet/dotnet repository is and will continue to be the vehicle through which we release .NET as source for you to consume. Once each major version ships (e.g. .NET 8.0.100 SDK), it will move into servicing. When in servicing, there will be new SDK bands created later in the cycle (e.g. 8.0.2xx, 8.0.3xx, ...) and shipped together in each servicing release.

SDK bands are versions of the SDK that differ in their feature set and allow us to support new .NET capabilities in new Visual Studio releases before the next major version comes. More about the release cycle and rhythm is explained in the documents in this PR.

This PR

Adds a main document and two specific proposals that go against each other. The main document compares them.

The PR is a vehicle for discussion over the proposals and design so things are subject to change.

dotnet/arcade-services#2364
dotnet/arcade-services#2622

[mmitche]

Good start. Lots of comments and debate. IMO there is a pretty clear winner so far (branches).

[tmds]

For source-build, it's likely maintainers will only build one SDK feature branch, and it would be nice if they can just work with a single branch from the vmr for doing so. This requires each SDK branch to include all shared components so it is self-contained.

If a maintainer would like to provide all SDK feature branches, it would be preferable to only have to build the runtime once. This could be implemented through a top-level flag causing only the sdk to be built. This will effectively build the same things as this proposal has for building under the non 1xx branches.

The proposed solution for eliminating the shared components from the non 1xx branches seems to be for dealing with:

This will give us more flexibility such as locking down the version of the shared components in the preview band to the last released version.

I don't fully understand the issue mentioned here, but I assume it could be solved by adding the necessary things to dotnet/installer branches.

[premun]

This is good feedback, thanks. We still need to discuss the "Release" section in depth but my personal expectation is that with the product becoming a bit more complex, the process of how we release and deliver the sources might be affected as it will need to address these new options.
The solution to this, in my opinion, can range from having a "super VMR" which would reference the released commits of the VMR via submodules, through us preparing a special release branch in the VMR, to just compiling the release tarball for all bands. But I think we should discuss this further and propose options here.

I don't fully understand the issue mentioned here, but I assume it could be solved by adding the necessary things to dotnet/installer branches.

This is mentioned in the parent document, in the Band life cycle section:

For this setup, we'd say the 100th band is in servicing and the 200th band is in preview. It is important to also note that while a band is in preview, it uses the most recently released .NET runtime while the servicing band revs with the 7.0 channel.

But I can maybe pull this up into its own subsection to make it more visible.

[tmds]

My main point of feedback is that for source-build maintainers the UX is better if branches are self-contained and can be built directly.

A key feature of the vmr is that it bundles all sources to build .NET in a single place.
That is definitely a big improvement from how we were building previously. We should aim to preserve this model.

[tmds]

cc @dotnet/distro-maintainers

[tmds]

In dotnet/source-build#3524 (reply in thread) @mmitche said:

• Currently, there is no guarantee that you can use a 1xx SDK to build the newer SDK bands. It's likely that newer SDK bands take dependencies on new features that are introduced in those bands. However, barring unexpected bugs, The N SDK should always build with N-1 of the same SDK band. So 8.0.202 with 8.0.201.
• Currently there is no guarantee that you can use a 2xx+ SDK to build the runtime. I think it's likely that it will work most of the time, and should work with small modifications (e.g. disable a warning or analyzer), but we don't guarantee it today. The runtime will always build with N-1 of the 1xx band though. We always want to be building it with a supported toolset, and the 1xx band is supported as long as the runtime is in support.

This means that if a 2xx band were to include the runtime, the runtime would need a 1xx SDK band, and the sdk would need a 2xx band. These are conflicting build dependencies.

The current proposal respects the guarantees by having in each branch only the sources that are expected to build with their corresponding SDK band.

To be able to build a runtime, distro maintainers need to maintain a 1xx band.
It's not possible to only build and ship the latest band.

[mmitche]

@tmds The 2xx+ bands would not include the runtime. So to build the 2xx band, you'd build 1xx, which would build the 1xx SDK and the shared runtime. Then those source-built outputs would be fed into the 2xx, 3xx, etc. SDK builds. Each branch would not be independently buildable such that each included the shared components.

It is pretty important to not view the SDK and runtime as a single product unit. They're essentially separate products, where the SDK happens to redist the runtime. In addition, the shared bits must be built once for N SDK bands. Also remember that the SDK is not the only distribution mechanism of the runtime bits. The aspnetcore hosting bundle, the runtime installers/zips, etc. also redist the runtime.

To illustrate this more, let's say you made each SDK band independently buildable. If a distro decides to make available the 1xx and 2xx bands, they would check out the 1xx SDK branch, build the runtime and SDK, and then check out the 2xx branch, again building the runtime and SDK. Now you have two copies of the shared components, built from different sources. When you deploy the runtime package, you need to choose one. You make an arbitrary choice, let's say 1xx. But now for the 2xx sdk, the runtime redisted in the SDK layout (which would be used in certain build scenarios) is not the same as the one used for 1xx. This would cause at best confusion, and at worst, installation issues.

So basically we're stuck in a place where either at most one band must contain the runtime sources, or no bands contain the runtime sources, and the runtime and SDK are built separately. In either case, the output of the build that creates the runtime binaries serves as an input to other builds. We think that having the 1xx band as the home of the runtime provides the least amount of friction.

[premun]

Thank you all for the discussion and the feedback. We have decided to merge this PR but keep the associated ongoing discussion alive where we can discuss the impact on distro maintainers. However, none of the discussions seem to affect the decision made here that the SDK branches proposal is the one we're going to implement.