Experiment with enabling msbuild static graph build

[ViktorHofer]

The --graph / --graphBuild msbuild CLI option enables the static graph build feature. From discussion with @rainersigwald:

no, Arcade doesn't use it by default
specs at https://github.com/dotnet/msbuild/blob/master/documentation/specs/static-graph.md but they're not good for high-level/motivation

in brief:
MSBuild normally discovers dependencies just in time: a project is building and invokes the MSBuild task to build another project. That works fine, but means we can't do some things like advanced scheduling (because we don't know what's going to be built in advance) and caching at the project level. MSBuild static graph restricts the build to define a graph of ProjectReferences in advance and then builds based on that graph, rather than just-in-time. That can help some builds

it's mostly helpful when combined with a build system that can cache at the project level, like CloudBuild does internally
but it can have some benefits even without that

We should see if enabling this feature reduces our build times and even if not, a predictable static graph is preferred over a just-in-time hard to reason about graph so it might be a good option anyway.

@jaredpar this would be a pre-cursor for using CloudBuild in CI.

cc @dotnet/runtime-infrastructure

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

[jkoritzinsky]

The installer layer is likely not compatible with static graph builds today. We'll have to do some work to provide the right information to MSBuild to support static graph.

[ViktorHofer]

Agreed. We already disable static graph for some of the installer projects during restore:

runtime/Build.proj

Lines 40 to 45 in 171ef84

	<Target Name="RestoreWithoutStaticGraph"
	BeforeTargets="Restore">
	<MSBuild Projects="@(DepprojProjectToBuild);@(PkgprojProjectToBuild);@(BundleProjectToBuild)"
	Properties="MSBuildRestoreSessionId=$([System.Guid]::NewGuid());RestoreUseStaticGraphEvaluation=false"
	Targets="Restore" />
	</Target>

.

@jkoritzinsky you mentioned that this will change with your new SDK?

[jkoritzinsky]

So static graph restore will work with the new SDK, but I don't know if a full static graph build will. I'll need to figure out based on the static graph protocol how to represent the various <MSBuild /> calls in the new sdk.

[ViktorHofer]

That makes sense. We will likely hit similar problems during the libraries builds as we call into msbuild in couple places as well (in example in the TargetFramework.SDK that lives in Arcade).

[rainersigwald]

There are two levels of static graph:

1. -graph, which requires that projects have ProjectReferences to build the graph but can handle calls outside it
2. -isolate, which is required for CloudBuild, and which forbids project-to-project references not known to the graph.

So hopefully the installer stuff will work in -graph mode even if it would fail in -isolate.

[ViktorHofer]

Thanks for the clarification @rainersigwald but doesn't that mean that the SDK itself won't work with -isolate as we invoke ie GetTargetFrameworks via an msbuild task?

[jaredpar]

@rainersigwald

which requires that projects have ProjectReferences to build the graph but can handle calls outside it

Can u elaborate on what this means? Or maybe provide some examples of what would not meet this requirement?

Feel like I've asked this before, wonder if we should start a doc on static graph (cookbook style) that discusses what is / isn't legal here.

[ViktorHofer]

I agree with Jared that further details on what is allowed and what is prohibited in the two modes would help us. Sounds like we should be able to enable -graph?

[rainersigwald]

@ViktorHofer @jaredpar can I impose on you to check out https://github.com/dotnet/msbuild/pull/5741/files?short_path=570cf46#diff-570cf46aec4396d3da8763436f48defb? Feedback welcome!

doesn't that mean that the SDK itself won't work with -isolate as we invoke ie GetTargetFrameworks via an msbuild task?

We should account for all of the standard SDK and common.targets referencing "for free". Things should only fail to work for customizations on top of that.

📝 There are a lot of "should"s in that sentence. We have some known problems, for example with NuGet and especially pack operations. That's why we're not evangelizing this hard. Please let us know how it works for you!