Add Android workload manifests to SB .NET SDK #42125

jonathanpeppers · 2024-07-12T15:37:35Z

If using a source-built .NET SDK, the Android workload manifests are
not included in the SDK. So, you won't see android workload when you
run dotnet workload search.

The Android workload, in general, contains private sources, but the
manifest files are completely public.

In the long run, we could consider onboarding dotnet/android to:

https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-repo.md

But the prerequisites would be to first have the Android workload
onboard to arcade:

https://github.com/dotnet/arcade/blob/main/Documentation/Onboarding.md

For now, we could include the Android workload in source-build by:

Add a step that copies the Android workload manifests source code
in-tree (to be used when updating manifests).
Commit the manifest sources.
Source-build will then include the Android workload manifests in
the SDK by using the source code, in-tree.

Context: https://github.com/dotnet/maui/releases/tag/9.0.0-preview.6.24327.7 Context: https://github.com/dotnet/android/releases/tag/34.99.0-preview.6.340 Context: https://github.com/xamarin/xamarin-macios/releases/tag/dotnet-9.0.1xx-preview6-9714 These released yesterday with .NET 9 Preview 6.

Context: dotnet/source-build#3242 If using a source-built .NET SDK, the Android workload manifests are not included in the SDK. So, you won't see `android` workload when you run `dotnet workload search`. The Android workload, in general, contains private sources, but the manifest files are completely public. In the long run, we could consider onboarding dotnet/android to: * https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-repo.md But the prerequisites would be to first have the Android workload onboard to arcade: * https://github.com/dotnet/arcade/blob/main/Documentation/Onboarding.md *For now*, we could include the Android workload in source-build by: * Add a step that copies the Android workload manifests source code in-tree (to be used when updating manifests). * Commit the manifest sources. * Source-build will then include the Android workload manifests in the SDK by using the source code, in-tree.

jonathanpeppers · 2024-07-12T15:38:56Z

src/Installer/redist-installer/targets/BundledManifests.targets

+    <!-- Do not copy for source build, should already be in-tree -->
+    <Copy Condition="'$(DotNetBuildSourceOnly)' != 'true'"
+          SourceFiles="@(SourceManifestContent)"
+          DestinationFolder="sdk-manifests/%(DestinationPath)"
+          SkipUnchangedFiles="true" />


This is for convenience of someone updating the manifests (likely me). We could put it behind a property like -p:UpdateManifestSources=true, if we are concerned about this running.

baronfel · 2024-07-22T18:47:38Z

cc @dotnet/source-build-internal this change updates the source-built SDK - does running the build script with -sb create an artifact that @jonathanpeppers can use to test the changes? That should all work from an e.g. Ubuntu WSL, right?

MichaelSimons · 2024-07-22T19:11:37Z

cc @dotnet/source-build-internal this change updates the source-built SDK - does running the build script with -sb create an artifact that @jonathanpeppers can use to test the changes? That should all work from an e.g. Ubuntu WSL, right?

No it won't. You need the sdk tarball that is produced from the sdk-source-build leg.

jonathanpeppers · 2024-07-22T20:44:46Z

Ok, it seems to be working, I tested in a GitHub codespace:

Get dotnet-sdk-9.0.100-preview.7.24366.1-centos.9-x64.tar.gz from build artifacts
mkdir test-sdk && cd test-sdk
tar -xvzf ../dotnet-sdk-9.0.100-preview.7.24366.1-centos.9-x64.tar.gz
$ ./dotnet workload search (this used to not even work)

Workload ID                 Description                                         
--------------------------------------------------------------------------------
android                     .NET SDK Workload for building Android applications.

$ ./dotnet workload install android

Skipping NuGet package signature verification.
...
Installing pack Microsoft.Android.Sdk.Linux version 34.99.0-preview.6.340...
...
Successfully installed workload(s) android.

Make a helloandroid folder somewhere and $ ../sdk/test-sdk/dotnet new android

The template "Android Application" was created successfully.

Now we need some Android dependencies:

$ ../sdk/test-sdk/dotnet build -t:InstallAndroidDependencies -p:AndroidSdkDirectory=/home/vscode/android-sdk/ -p:JavaSdkDirectory=/home/vscode/jdk/ -p:AcceptAndroidSDKLicenses=true

Build the app:

$ ../sdk/test-sdk/dotnet build -bl -p:AndroidSdkDirectory=/home/vscode/android-sdk/ -p:JavaSdkDirectory=/home/vscode/jdk/
Restore complete (0.5s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  helloandroid succeeded (17.4s) → bin/Debug/net9.0-android/helloandroid.dll

Build succeeded in 19.5s

$ ls -la bin/Debug/net9.0-android/*.apk
-rw-rw-rw- 1 vscode vscode 10224521 Jul 22 20:43 bin/Debug/net9.0-android/com.companyname.helloandroid.apk
-rw-r--rw- 1 vscode vscode 10237801 Jul 22 20:43 bin/Debug/net9.0-android/com.companyname.helloandroid-Signed.apk

jonathanpeppers · 2024-07-22T20:46:39Z

@MichaelSimons there is an error like:

/vmr/repo-projects/Directory.Build.targets(765,5): warning : 1 new packages used not in baseline! See report at /vmr/artifacts/prebuilt-report/baseline-comparison.xml for more information. Package IDs are: [/vmr/repo-projects/dotnet.proj]
/vmr/repo-projects/Directory.Build.targets(765,5): warning : Microsoft.NET.Sdk.Android.Manifest-9.0.100-preview.6.34.99.0-preview.6.340 [/vmr/repo-projects/dotnet.proj]
/vmr/repo-projects/Directory.Build.targets(765,5): warning : Prebuilt usages are different from the baseline. If detected changes are acceptable, update baseline with: [/vmr/repo-projects/dotnet.proj]
/vmr/repo-projects/Directory.Build.targets(765,5): warning : cp '/vmr/artifacts/prebuilt-report/generated-new-baseline.xml' '/vmr/eng/tools/prebuilt-baseline.xml' [/vmr/repo-projects/dotnet.proj]
  Packaged all symbols in '/vmr/artifacts/assets/Release/dotnet-symbols-all-9.0.100-preview.7.24366.1-centos.9-x64.tar.gz'
  Packaged sdk symbols in '/vmr/artifacts/assets/Release/dotnet-symbols-sdk-9.0.100-preview.7.24366.1-centos.9-x64.tar.gz'
  Found 1 files in prebuilt packages dir.
  Tarball '/vmr/artifacts/assets/Release/Private.SourceBuilt.Prebuilts.9.0.100-preview.7.24366.1.centos.9-x64.tar.gz' was successfully created from '/vmr/artifacts/prebuilt-report/prebuilt-packages/'
/vmr/eng/finish-source-only.proj(171,5): error : 1 Prebuilts Exist

Can that be fixed in this repo?

MichaelSimons · 2024-07-22T20:56:49Z

eng/SourceBuildPrebuiltBaseline.xml

@@ -7,6 +7,9 @@
    <UsagePattern IdentityGlob="Nuget.*/*" />
    <UsagePattern IdentityGlob="Microsoft.Build.NuGetSdkResolver/*" />

+    <!-- Sources for this package are included in-repo -->
+    <UsagePattern IdentityGlob="Microsoft.NET.Sdk.Android.Manifest-*/*" />


I don't think this should be added here. If it is in this repo, then it shouldn't be detected as a prebuilt. I suspect something isn't working as expected.

How does it "find" prebuilts? Would <PackageDownload Include="Foo" Version="[1.0.0]"/> trigger this?

The package is being included as a download dependency:

So in this case, yes, <PackageDownload Include="Microsoft.NET.Sdk.Android.Manifest" .../> is triggering the prebuilt.

More often than not, prebuilts are caused by package references & transitive dependencies. Technically, any package used in the build that is not built from source is a prebuilt, so it makes sense that a download dependency would cause a prebuilt, but this is the first time I've seen a package download cause a prebuilt.

src/Installer/redist-installer/targets/BundledManifests.targets

Co-authored-by: Michael Simons <msimons@microsoft.com>

MichaelSimons · 2024-07-23T14:24:55Z

src/Installer/redist-installer/targets/BundledManifests.targets

@@ -31,12 +31,14 @@
      <RestoredNupkgContentPath>$(NuGetPackageRoot)$([MSBuild]::ValueOrDefault('%(NupkgId)', '').ToLower())/$([MSBuild]::ValueOrDefault('%(Version)', '').ToLower())</RestoredNupkgContentPath>
      <MsiNupkgId>%(Identity).Manifest-%(FeatureBand).Msi.$(MsiArchitectureForWorkloadManifests)</MsiNupkgId>
      <RestoredMsiNupkgContentPath>$(NuGetPackageRoot)$([MSBuild]::ValueOrDefault('%(MsiNupkgId)', '').ToLower())/$([MSBuild]::ValueOrDefault('%(Version)', '').ToLower())</RestoredMsiNupkgContentPath>
+      <IncludedInSource Condition="'%(IncludedInSource)' == ''">false</IncludedInSource>
+      <PackageDownload Condition="'$(DotNetBuildSourceOnly)' != 'true' or '%(IncludedInSource)' != 'true'">true</PackageDownload>


Would it make sense to not specialize this for source-build? e.g. the MSFT build would also use the version from the SDK repo. It is best to avoid unnecessary differences between the two builds. Keeping them the same helps flush out issues.

I can rework this to use a property like -p:UpdateManifestSources=true, and then it wouldn't need to ever check $(DotNetBuildSourceOnly). Someone would only set this property when updating the manifest sources.

Is the build green (or need a rerun)? I don't think I've seen it fully green yet, so just wondering if it works before I refactor it. 😄

Ok, I think this got past the pre-built errors now, but some of the smoke tests run into:

System.InvalidOperationException : Failed to execute /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/extracted-sdk/dotnet test /bl:/vmr/artifacts/TestResults/Release/BasicScenarioTests_MSTest_CSharp-test.binlog Exit code: 1 /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: SDK Resolver Failure: "The SDK resolver "Microsoft.DotNet.MSBuildWorkloadSdkResolver" failed while attempting to resolve the SDK "MSTest.Sdk/3.4.3". Exception: "System.IO.FileNotFoundException: Workload manifest microsoft.net.sdk.android: 34.99.0-preview.6.340/9.0.100-preview.6 from workload version 9.0.100-preview.7.24373.1 was not installed. Running "dotnet workload repair" may resolve this. /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider.GetManifests() /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.LoadManifestsFromProvider(IWorkloadManifestProvider manifestProvider) /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests() /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.TryGetPackInfo(WorkloadPackId packId) /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, IWorkloadManifestProvider manifestProvider, IWorkloadResolver workloadResolver) /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, String dotnetRootPath, String sdkVersion, String userProfileDir, String globalJsonPath) /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.WorkloadSdkResolver.Resolve(SdkReference sdkReference, SdkResolverContext resolverContext, SdkResultFactory factory) /vmr/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/projects-202407231701323151/BasicScenarioTests_MSTest_CSharp/BasicScenarioTests_MSTest_CSharp.csproj : error MSB4242: at Microsoft.Build.BackEnd.SdkResolution.SdkResolverService.TryResolveSdkUsingSpecifiedResolvers(IReadOnlyList`1 resolvers, Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio, SdkResult& sdkResult, IEnumerable`1& errors, IEnumerable`1& warnings)""

I'll see if I can reproduce this on GitHub codespaces.

MichaelSimons · 2024-08-28T16:26:55Z

@jonathanpeppers, is this something that is still in scope for 9.0?

jonathanpeppers · 2024-08-28T16:30:29Z

@MichaelSimons yes, unfortunately I've had to focus on other things for a bit.

Linux support is just lower on the list than anything else, but we'd still like to complete this at some point.

Maybe if this doesn't make RC 2, we do it for .NET 10?

MichaelSimons · 2024-08-28T16:51:20Z

@MichaelSimons yes, unfortunately I've had to focus on other things for a bit.

Linux support is just lower on the list than anything else, but we'd still like to complete this at some point.

Maybe if this doesn't make RC 2, we do it for .NET 10?

Thanks, if you need help diagnosing source-build related failures, let me know and I can help.

jonathanpeppers added 2 commits July 12, 2024 09:57

dotnet-issue-labeler bot added Area-Workloads untriaged Request triage from a team member labels Jul 12, 2024

jonathanpeppers commented Jul 12, 2024

View reviewed changes

jonathanpeppers and others added 2 commits July 15, 2024 14:07

Merge branch 'main' into android-source-build

86de0c5

Update SourceBuildPrebuiltBaseline.xml

c00c168

build-analysis bot mentioned this pull request Jul 16, 2024

Roslyn analyzer throws error AD0001 NullReferenceException dotnet/dnceng#3305

Open

3 tasks

Merge branch 'main' into android-source-build

6ffb23a

jonathanpeppers force-pushed the android-source-build branch from 008916f to 6ffb23a Compare July 16, 2024 13:35

Update SourceBuildPrebuiltBaseline.xml

d3191c4

jonathanpeppers marked this pull request as ready for review July 22, 2024 20:45

jonathanpeppers requested a review from a team as a code owner July 22, 2024 20:45

jonathanpeppers requested a review from MichaelSimons July 22, 2024 20:46

MichaelSimons reviewed Jul 22, 2024

View reviewed changes

jonathanpeppers and others added 2 commits July 22, 2024 16:06

Merge branch 'main' into android-source-build

c0883a0

Conditionally add to @(PackageDownload)

641d9a3

MichaelSimons reviewed Jul 23, 2024

View reviewed changes

src/Installer/redist-installer/targets/BundledManifests.targets Outdated Show resolved Hide resolved

Update src/Installer/redist-installer/targets/BundledManifests.targets

13223ae

Co-authored-by: Michael Simons <msimons@microsoft.com>

MichaelSimons reviewed Jul 23, 2024

View reviewed changes

build-analysis bot mentioned this pull request Jul 23, 2024

msbuild crashes with "MSB0001: Internal MSBuild Error: must be valid" dotnet/dnceng#3304

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Android workload manifests to SB .NET SDK #42125

Add Android workload manifests to SB .NET SDK #42125

jonathanpeppers commented Jul 12, 2024 •

edited

Loading

jonathanpeppers Jul 12, 2024

baronfel commented Jul 22, 2024

MichaelSimons commented Jul 22, 2024

jonathanpeppers commented Jul 22, 2024 •

edited

Loading

jonathanpeppers commented Jul 22, 2024

MichaelSimons Jul 22, 2024

jonathanpeppers Jul 22, 2024

ellahathaway Jul 22, 2024

ellahathaway Jul 22, 2024 •

edited

Loading

MichaelSimons Jul 23, 2024

jonathanpeppers Jul 23, 2024

jonathanpeppers Jul 23, 2024

MichaelSimons commented Aug 28, 2024

jonathanpeppers commented Aug 28, 2024

MichaelSimons commented Aug 28, 2024

Add Android workload manifests to SB .NET SDK #42125

Are you sure you want to change the base?

Add Android workload manifests to SB .NET SDK #42125

Conversation

jonathanpeppers commented Jul 12, 2024 • edited Loading

jonathanpeppers Jul 12, 2024

Choose a reason for hiding this comment

baronfel commented Jul 22, 2024

MichaelSimons commented Jul 22, 2024

jonathanpeppers commented Jul 22, 2024 • edited Loading

jonathanpeppers commented Jul 22, 2024

MichaelSimons Jul 22, 2024

Choose a reason for hiding this comment

jonathanpeppers Jul 22, 2024

Choose a reason for hiding this comment

ellahathaway Jul 22, 2024

Choose a reason for hiding this comment

ellahathaway Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

MichaelSimons Jul 23, 2024

Choose a reason for hiding this comment

jonathanpeppers Jul 23, 2024

Choose a reason for hiding this comment

jonathanpeppers Jul 23, 2024

Choose a reason for hiding this comment

MichaelSimons commented Aug 28, 2024

jonathanpeppers commented Aug 28, 2024

MichaelSimons commented Aug 28, 2024

jonathanpeppers commented Jul 12, 2024 •

edited

Loading

jonathanpeppers commented Jul 22, 2024 •

edited

Loading

ellahathaway Jul 22, 2024 •

edited

Loading