Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support DotNetBuildPass property in Arcade's Build.proj #14660

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
+52 −87
Open

Support DotNetBuildPass property in Arcade's Build.proj #14660

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
52f8474
Support DotNetBuildPass property in Arcade's Build.proj
ViktorHofer Mar 27, 2024
ca489cc
Update src/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
ViktorHofer Mar 28, 2024
bde03ad
Update src/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
ViktorHofer Mar 28, 2024
ccbc0ee
Update Build.proj
ViktorHofer Mar 28, 2024
f745c6d
Update Build.proj
ViktorHofer Mar 28, 2024
75eefba
Update Build.proj
ViktorHofer Mar 28, 2024
6a723de
Update Build.proj
ViktorHofer Mar 28, 2024
b0de28b
Update SourceBuildArcadeBuild.targets
ViktorHofer Mar 28, 2024
26f17d0
Update Build.proj
ViktorHofer Mar 28, 2024
d2bdf1f
Update Build.proj
ViktorHofer Apr 2, 2024
f9f8a93
Updates to DotNetBuildPass
ViktorHofer Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 40 additions & 38 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
Optional parameters:
Configuration Build configuration: "Debug", "Release", etc.

DotNetBuildPass The build pass number. Set to a number other than '1' when performing a join build.
This makes ProjectToBuild items only be built when the `DotNetBuildPass` metadata matches.
DotNetBuildRepo Build the repo as part of the entire .NET stack.
DotNetBuildInnerRepo Build the repo as part of the entire .NET stack. This switch is used on the inner repo invocation and should not be passed by the user.
DotNetBuildOrchestrator Build the entire .NET stack.
Expand Down Expand Up @@ -58,27 +60,35 @@
<Import Project="$(RepositoryEngineeringDir)Build.props" Condition="Exists('$(RepositoryEngineeringDir)Build.props')" />

<PropertyGroup>
<_ProjectsPropertyWasUpdatedInBuildProps Condition="'$(_OriginalProjectsValue)' != '$(Projects)'">true</_ProjectsPropertyWasUpdatedInBuildProps>
<_DotNetBuildPassSet Condition="'$(DotNetBuildPass)' != '' and '$(DotNetBuildPass)' != '1'">true</_DotNetBuildPassSet>
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
</PropertyGroup>

<ItemGroup Condition="'$(Projects)' != ''">
<!-- Copy the original list so we can log diagnostics later. -->
<_OriginalProjectToBuild Include="@(ProjectToBuild)" />
<!-- Setting 'Projects' overrides the ProjectToBuild list. -->
<ProjectToBuild Remove="@(ProjectToBuild)" />
<ProjectToBuild Include="$(Projects)" />
</ItemGroup>

<!--
Default values.
-->
<!-- If Projects is unspecified and ProjectToBuild was not set via Build.props, fallback to building .sln files in the repo root. -->
<ItemGroup Condition="'@(ProjectToBuild)' == ''">
<ProjectToBuild Include="$(RepoRoot)*.sln" />
<!-- Don't glob and build solution files from the repo root by default when DotNetBuildPass is set to a value other than 1. -->
<EnableDefaultSolutionItems Condition="'$(EnableDefaultSolutionItems)' == '' and '$(_DotNetBuildPassSet)' == 'true'">false</EnableDefaultSolutionItems>
<EnableDefaultSolutionItems Condition="'$(EnableDefaultSolutionItems)' == ''">true</EnableDefaultSolutionItems>
<ProjectToBuild Include="$(RepoRoot)*.sln" Condition="'$(EnableDefaultSolutionItems)' == 'true'" />
</ItemGroup>

<Target Name="Execute">
<Error Text="No projects were found to build. Either the 'Projects' property or 'ProjectToBuild' item group must be specified." Condition="'@(ProjectToBuild)' == ''"/>
<!-- Only build the projects for the passed in build pass numer in when DotNetBuildPass is set to a value other than 1. -->
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
<ItemGroup Condition="'@(ProjectToBuild)' != '' and '$(_DotNetBuildPassSet)' == 'true'">
<_ProjectToBuildKeep Include="@(ProjectToBuild->WithMetadataValue('DotNetBuildPass', '$(DotNetBuildPass)'))" />
<_ProjectToBuildRemove Include="@(ProjectToBuild)" Exclude="@(_ProjectToBuildKeep)" />
<ProjectToBuild Remove="@(_ProjectToBuildRemove)" />
</ItemGroup>

<!-- Skip execution when DotNetBuildPass is set and ProjectToBuild is empty.
This can happen whenever a repository gets built as part of a VMR join vertical but already built
all components in the non-join build. -->
<Target Name="Execute"
Condition="'$(_DotNetBuildPassSet)' != 'true' or '@(ProjectToBuild)' != ''">
<Error Text="Property 'RepoRoot' must be specified" Condition="'$(RepoRoot)' == ''"/>
<Error Text="File 'global.json' must exist in directory specified by RepoRoot: '$(RepoRoot)'" Condition="'$(RepoRoot)' != '' and !Exists('$(RepoRoot)global.json')"/>

Expand Down Expand Up @@ -110,6 +120,9 @@
<_SolutionBuildTargets Include="PerformanceTest" Condition="'$(PerformanceTest)' == 'true'" />
</ItemGroup>

<Error Text="No projects were found to build. Either the 'Projects' property or 'ProjectToBuild' item group must be specified."
Condition="'@(ProjectToBuild)' == '' and '@(_SolutionBuildTargets)' != '' and '$(SkipProjectToBuildError)' != 'true'"/>

<PropertyGroup>
<_RemoveProps>Projects;Restore;Deploy;Sign;Publish;NETCORE_ENGINEERING_TELEMETRY;@(_SolutionBuildTargets)</_RemoveProps>
</PropertyGroup>
Expand Down Expand Up @@ -177,9 +190,7 @@
<_SolutionBuildProps Include="__ImportPackTargets=true" Condition="'$(Pack)' == 'true'" />
</ItemGroup>

<!--
Restore built-in tools.
-->
<!-- Restore built-in tools -->
<MSBuild Projects="Tools.proj"
Targets="Restore"
Properties="@(_RestoreToolsProps);_NETCORE_ENGINEERING_TELEMETRY=Restore"
Expand Down Expand Up @@ -226,7 +237,7 @@
Targets="_IsProjectRestoreSupported"
SkipNonexistentTargets="true"
BuildInParallel="true"
Condition="'$(RestoreUsingNuGetTargets)' != 'false' and '%(ProjectToBuild.Extension)' != '.sln' and '%(ProjectToBuild.Extension)' != '.slnf' and '$(RestoreUseStaticGraphEvaluation)' != 'true' and '$(Restore)' == 'true'">
Condition="'$(RestoreUsingNuGetTargets)' != 'false' and '@(ProjectToBuild)' != '' and '%(ProjectToBuild.Extension)' != '.sln' and '%(ProjectToBuild.Extension)' != '.slnf' and '$(RestoreUseStaticGraphEvaluation)' != 'true' and '$(Restore)' == 'true'">

<Output TaskParameter="TargetOutputs" ItemName="_ProjectToRestoreWithNuGet" />
</MSBuild>
Expand Down Expand Up @@ -259,26 +270,22 @@
Targets="Restore"
SkipNonexistentTargets="true"
BuildInParallel="%(_ProjectToRestore.RestoreInParallel)"
Condition="'$(Restore)' == 'true'"/>
Condition="'$(Restore)' == 'true' and '@(_ProjectToRestore)' != ''" />

<!--
Build solution.
-->
<!-- Build ProjectToBuild items -->
<MSBuild Projects="@(ProjectToBuild)"
Properties="@(_SolutionBuildProps);__BuildPhase=SolutionBuild;_NETCORE_ENGINEERING_TELEMETRY=Build"
RemoveProperties="$(_RemoveProps)"
Targets="@(_SolutionBuildTargets)"
BuildInParallel="%(ProjectToBuild.BuildInParallel)"
Condition="'@(_SolutionBuildTargets)' != ''" />
Condition="'@(_SolutionBuildTargets)' != '' and '@(ProjectToBuild)' != ''" />

<MSBuild Projects="AfterSolutionBuild.proj"
Properties="@(_CommonProps);_NETCORE_ENGINEERING_TELEMETRY=Build"
Targets="@(_SolutionBuildTargets)"
Condition="'@(_SolutionBuildTargets)' != ''" />

<!--
Sign artifacts.
-->
<!-- Sign artifacts -->
<MSBuild Projects="Sign.proj"
Properties="@(_CommonProps)"
Targets="Sign"
Expand All @@ -297,30 +304,25 @@
Properties="@(_CommonProps);_NETCORE_ENGINEERING_TELEMETRY=AfterSourceBuild"
Condition="'$(ArcadeBuildFromSource)' == 'true' or '$(DotNetBuildRepo)' == 'true'"/>

<!--
Publish artifacts. This should run in the following situations:
- Regular single repo builds
- When running in the outer build phase of a source-only repo build, to properly publish the source-build intermediate nupkg.
- VMR builds
- When running in the inner build phase

It shouldn't run:
- In an inner build if running from single repo build.
- In an outer build if running from the full product build.
-->

<PropertyGroup>
<!-- Publish artifacts -->
<PropertyGroup Condition="'$(Publish)' == 'true'">
<_VmrBuild Condition="'$(DotNetBuildFromSourceFlavor)' == 'Product' or '$(DotNetBuildOrchestrator)' == 'true'">true</_VmrBuild>

<_OuterRepoBuild Condition="'$(ArcadeInnerBuildFromSource)' != 'true' and '$(DotNetBuildPhase)' != 'InnerRepo'">true</_OuterRepoBuild>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need the legacy switches here. Arcade must pass the legacy switches to support repos that don't use the new switches within their repo code, but Arcade shouldn't need to use switches any longer. You can just have comparisons against DotNetBuildPhase (Repo, InnerRepo and Orchestrator)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that's true. A repository that is still on Arcade 8 wouldn't have DotNetBuildPhase set to anything in the outer repo non-VMR build.

That's because repos on the old Arcade and eng/common scripts don't pass DotNetBuildRepo in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if the repo is still on Arcade 8, it wouldn't be using this code at all in non-VMR builds.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about a repo like Roslyn that still uses Arcade 8 and eng/common scripts from Arcade 8 inside the VMR? In source-build mode they still use their eng/common scripts which don't pass the new properties in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The eng/common scripts aren't driving the parameters passed to the VMR-based builds, the VMR orchestrator does that. Anyways, we copy over those scripts, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyways, we copy over those scripts, right?

Not for source-build: https://github.com/dotnet/dotnet/blob/262762535e6023f91ed52f8650287a47a124faaa/repo-projects/Directory.Build.targets#L238

The eng/common scripts aren't driving the parameters passed to the VMR-based builds, the VMR orchestrator does that.

Good point.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyways, you can verify, but in the inner and outer builds for roslyn in the VMR, you should see all the new parameters.

<_InnerRepoBuild Condition="'$(ArcadeInnerBuildFromSource)' == 'true' or '$(DotNetBuildPhase)' == 'InnerRepo'">true</_InnerRepoBuild>
<_ShouldRunPublish Condition="'$(_InnerRepoBuild)' == 'true' and '$(_VmrBuild)' == 'true'">true</_ShouldRunPublish>
<_ShouldRunPublish Condition="'$(_InnerRepoBuild)' != 'true' and '$(_VmrBuild)' != 'true'">true</_ShouldRunPublish>

<!-- Don't publish inside the VMR in the outer repo build. -->
<_ShouldRunPublish Condition="'$(_VmrBuild)' == 'true' and '$(_OuterRepoBuild)' == 'true'">false</_ShouldRunPublish>
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
<!-- Don't publish outside of the VRM in an inner build (source-only). -->
<_ShouldRunPublish Condition="'$(_VmrBuild)' != 'true' and '$(_InnerRepoBuild)' == 'true'">false</_ShouldRunPublish>
<!-- Otherwise, publish -->
<_ShouldRunPublish Condition="'$(_ShouldRunPublish)' == ''">true</_ShouldRunPublish>
</PropertyGroup>

<!-- Make sure we always publish in VMR build - working around runtime repo which sets Publish to false. -->
<MSBuild Projects="Publish.proj"
Properties="@(_PublishProps);_NETCORE_ENGINEERING_TELEMETRY=Publish"
Targets="Publish"
Condition="'$(Publish)' == 'true' and '$(_ShouldRunPublish)' == 'true'"/>
Condition="'$(_ShouldRunPublish)' == 'true'"/>
</Target>

</Project>
17 changes: 12 additions & 5 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,6 @@
('$(DotNetBuildSourceOnly)' != 'true' or
('$(ArcadeInnerBuildFromSource)' == 'true' and '$(DotNetBuildFromSourceFlavor)' != 'Product'))">true</AutoGenerateSymbolPackages>

<AssetManifestOS Condition="'$(AssetManifestOS)' == ''">$(OS)</AssetManifestOS>

<AssetManifestFileName>$(AssetManifestOS)-$(PlatformName).xml</AssetManifestFileName>
<AssetManifestFilePath>$(ArtifactsLogDir)AssetManifest\$(AssetManifestFileName)</AssetManifestFilePath>

<SymbolPackagesDir>$(ArtifactsTmpDir)SymbolPackages\</SymbolPackagesDir>

<PublishDependsOnTargets Condition="$(PublishToSymbolServer)">$(PublishDependsOnTargets);PublishSymbols</PublishDependsOnTargets>
Expand All @@ -78,6 +73,18 @@
<PublishDependsOnTargets>BeforePublish;$(PublishDependsOnTargets)</PublishDependsOnTargets>
</PropertyGroup>

<PropertyGroup>
<!-- Prefer TargetOS when set over OS -->
<AssetManifestOS Condition="'$(AssetManifestOS)' == ''">$([MSBuild]::ValueOrDefault('$(TargetOS)', '$(OS)'))</AssetManifestOS>
<!-- Prefer TargetArchitecture when set over PlatformName -->
<AssetManifestArch Condition="'$(AssetManifestArch)' == ''">$([MSBuild]::ValueOrDefault('$(TargetArchitecture)', '$(PlatformName)'))</AssetManifestArch>
<!-- Add the build pass number when DotNetBuildPass is set to a value other than 1. -->
<AssetManifestPass Condition="'$(DotNetBuildPass)' != '' and '$(DotNetBuildPass)' != '1'">-Pass$(DotNetBuildPass)</AssetManifestPass>

<AssetManifestFileName>$(AssetManifestOS)-$(AssetManifestArch)$(AssetManifestPass).xml</AssetManifestFileName>
<AssetManifestFilePath>$(ArtifactsLogDir)AssetManifest\$(AssetManifestFileName)</AssetManifestFilePath>
</PropertyGroup>

<Import Project="$(NuGetPackageRoot)microsoft.dotnet.build.tasks.feed\$(MicrosoftDotNetBuildTasksFeedVersion)\build\Microsoft.DotNet.Build.Tasks.Feed.targets"/>
<Import Project="$(NuGetPackageRoot)microsoft.symboluploader.build.task\$(MicrosoftSymbolUploaderBuildTaskVersion)\build\PublishSymbols.targets" Condition="$(PublishToSymbolServer)"/>

Expand Down
14 changes: 0 additions & 14 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/Noop.proj
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ call an inner build after some setup. The targets work roughly like this:
* Create intermediate nupkg that contains the inner source-build's artifacts.
* MSBuild `Projects=SourceBuildIntermediate.proj Targets=Restore;Pack`
* Empty out the list of `ProjectToBuild`, because we already built them from source.
* Put `Noop.proj` in the list as a sentinel value.
* During **Outer Execute**:
* MSBuild `Projects=Tools.proj Targets=Restore`
* Does nothing interesting.
Expand Down
31 changes: 12 additions & 19 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,21 @@

<_DirSeparatorEscapedCharForExecArg Condition="'$(OS)' == 'Windows_NT'">\</_DirSeparatorEscapedCharForExecArg>
<_DirSeparatorEscapedCharForExecArg Condition="'$(OS)' != 'Windows_NT'" />

<RunInnerSourceBuildCommand Condition="('$(ArcadeBuildFromSource)' == 'true' and
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
'$(ArcadeInnerBuildFromSource)' != 'true')
or '$(DotNetBuildPhase)' == 'Repo'">true</RunInnerSourceBuildCommand>
<SkipProjectToBuildError Condition="'$(RunInnerSourceBuildCommand)' == 'true'">true</SkipProjectToBuildError>
</PropertyGroup>

<!-- Prevent projects from building in the outer build: they would use prebuilts. -->
<ItemGroup Condition="'$(RunInnerSourceBuildCommand)' == 'true'">
<ProjectToBuild Remove="@(ProjectToBuild)" />
</ItemGroup>

<Target Name="ExecuteWithSourceBuiltTooling"
DependsOnTargets="
GetSourceBuildCommandConfiguration;
RunInnerSourceBuildCommand"
Condition="
('$(ArcadeBuildFromSource)' == 'true' and '$(ArcadeInnerBuildFromSource)' != 'true') or
'$(DotNetBuildPhase)' == 'Repo'"
DependsOnTargets="GetSourceBuildCommandConfiguration;RunInnerSourceBuildCommand"
Condition="'$(RunInnerSourceBuildCommand)' == 'true'"
BeforeTargets="Execute" />

<!--
Expand Down Expand Up @@ -114,9 +120,6 @@
<Target Name="RunInnerSourceBuildCommand"
DependsOnTargets="PrepareInnerSourceBuildRepoRoot">
<PropertyGroup>
<!-- Prevent any projects from building in the outside build: they would use prebuilts. -->
<PreventPrebuiltBuild>true</PreventPrebuiltBuild>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dotnet/source-build-internal What happens if the outer build does build something and use pre-builts? Would they still get detected at repo-source build time?


<!--
Normally, the inner build should run using the original build command with some extra args
appended. Allow the repo to override this default behavior if the repo is e.g. not onboarded
Expand All @@ -132,14 +135,4 @@
EnvironmentVariables="@(InnerBuildEnv)" />
</Target>

<Target Name="PreventPrebuiltBuild"
DependsOnTargets="ExecuteWithSourceBuiltTooling"
Condition="'$(PreventPrebuiltBuild)' == 'true'"
BeforeTargets="Execute">
<ItemGroup>
<ProjectToBuild Remove="@(ProjectToBuild)" />
<ProjectToBuild Include="$(MSBuildThisFileDirectory)Noop.proj" />
</ItemGroup>
</Target>

</Project>
Loading