Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breakage with workload installation in SDK 6.0.400 #27109

Open
filipnavara opened this issue Aug 11, 2022 · 20 comments
Open

Breakage with workload installation in SDK 6.0.400 #27109

filipnavara opened this issue Aug 11, 2022 · 20 comments
Labels
Area-Workloads untriaged Request triage from a team member
Milestone

Comments

@filipnavara
Copy link
Member

filipnavara commented Aug 11, 2022

I originally reported our broken scenario in #25651. We use Mono MSBuild to compile projects that reference multi-target libraries. These referenced libraries often include something like net6.0-macos in the TFM along with netstandard2.1. SDK 6.0.300 band broke the builds as described in the linked issue. The workaround was to install also the 6.0.200 band which was picked up by the desktop Mono MSBuild and worked fine. This required installation of workloads through dotnet workload install --sdk-version 6.0.203 macos. Since 6.0.400 SDK was published the workaround no longer works. The MSBuild complains about missing workloads even though it references the 6.0.203 SDK in the paths in log:

"/Users/teamcity/actions-runner/_work/emclient/emclient/MailClient.Mobile/MailClient.Mobile.iOS/MailClient.Mobile.iOS.csproj" (Restore target) (1) ->
"/Users/teamcity/actions-runner/_work/emclient/emclient/Dependencies/Mac/MacApi/MacApi.csproj" (_GenerateProjectRestoreGraph target) (56:6) ->
"/Users/teamcity/actions-runner/_work/emclient/emclient/Dependencies/Mac/MacApi/MacApi.csproj" (_GenerateProjectRestoreGraphPerFramework target) (56:8) ->
  /Users/teamcity/.dotnet/sdk/6.0.203/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.ImportWorkloads.targets(38,5): error NETSDK1147: To build this project, the following workloads must be installed: macos [/Users/teamcity/actions-runner/_work/emclient/emclient/Dependencies/Mac/MacApi/MacApi.csproj]
Error: /Users/teamcity/.dotnet/sdk/6.0.203/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.ImportWorkloads.targets(38,5): error NETSDK1147: To install these workloads, run the following command: dotnet workload install macos [/Users/teamcity/actions-runner/_work/emclient/emclient/Dependencies/Mac/MacApi/MacApi.csproj]

The workload installation log:

>  dotnet workload install --sdk-version 6.0.[2](https://github.com/emclient/emclient/runs/7781705689?
>  env:
>    DOTNET_ROOT: /Users/teamcity/.dotnet
Workload(s) 'macos ios' are already installed.
Skipping NuGet package signature verification.
Skipping NuGet package signature verification.
Installing pack Microsoft.macOS.Sdk version 12.3.[4](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:4)47...
Pack Microsoft.macOS.Sdk version 12.3.447 is already installed.
Writing workload pack installation record for Microsoft.macOS.Sdk version 12.3.447...
Installing pack Microsoft.macOS.Ref version 12.3.447...
Pack Microsoft.macOS.Ref version 12.3.447 is already installed.
Writing workload pack installation record for Microsoft.macOS.Ref version 12.3.447...
Installing pack Microsoft.macOS.Runtime.osx-arm64 version 12.3.447...
Pack Microsoft.macOS.Runtime.osx-arm64 version 12.3.447 is already installed.
Writing workload pack installation record for Microsoft.macOS.Runtime.osx-arm64 version 12.3.447...
Installing pack Microsoft.macOS.Runtime.osx-x64 version 12.3.447...
Pack Microsoft.macOS.Runtime.osx-x64 version 12.3.447 is already installed.
Writing workload pack installation record for Microsoft.macOS.Runtime.osx-x64 version 12.3.447...
Installing pack Microsoft.macOS.Templates version 12.3.447...
Pack Microsoft.macOS.Templates version 12.3.447 is already installed.
Writing workload pack installation record for Microsoft.macOS.Templates version 12.3.447...
Installing pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.8...
Pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.8...
Installing pack Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.8...
Pack Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.8...
Installing pack Microsoft.iOS.Sdk version 1[5](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:5).4.447...
Pack Microsoft.iOS.Sdk version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Sdk version 15.4.447...
Installing pack Microsoft.iOS.Ref version 15.4.447...
Pack Microsoft.iOS.Ref version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Ref version 15.4.447...
Installing pack Microsoft.iOS.Runtime.ios-arm version 15.4.447...
Pack Microsoft.iOS.Runtime.ios-arm version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Runtime.ios-arm version 15.4.447...
Installing pack Microsoft.iOS.Runtime.ios-arm[6](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:7)4 version 15.4.44[7](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:8)...
Pack Microsoft.iOS.Runtime.ios-arm64 version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Runtime.ios-arm64 version 15.4.447...
Installing pack Microsoft.iOS.Runtime.iossimulator-x[8](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:9)6 version [15](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:16).4.447...
Pack Microsoft.iOS.Runtime.iossimulator-x86 version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Runtime.iossimulator-x86 version 15.4.447...
Installing pack Microsoft.iOS.Runtime.iossimulator-x64 version 15.4.447...
Pack Microsoft.iOS.Runtime.iossimulator-x64 version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Runtime.iossimulator-x64 version 15.4.447...
Installing pack Microsoft.iOS.Runtime.iossimulator-arm64 version 15.4.447...
Pack Microsoft.iOS.Runtime.iossimulator-arm64 version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Runtime.iossimulator-arm64 version 15.4.447...
Installing pack Microsoft.iOS.Templates version 15.4.447...
Pack Microsoft.iOS.Templates version 15.4.447 is already installed.
Writing workload pack installation record for Microsoft.iOS.Templates version 15.4.447...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm64 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-arm64 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-arm64 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.iossimulator-arm64 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x64 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x64 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.iossimulator-x64 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x86 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x86 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.iossimulator-x86 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.Mono.ios-arm version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.Mono.ios-arm version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.ios-arm version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-arm64 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-arm64 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.iossimulator-arm64 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 version 6.0.8...
Installing pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-x86 version 6.0.8...
Pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-x86 version 6.0.8 is already installed.
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.iossimulator-x86 version 6.0.8...
Garbage collecting for SDK feature band(s) 6.0.400 6.0.[20](https://github.com/emclient/emclient/runs/7781705689?check_suite_focus=true#step:4:21)0...

Successfully installed workload(s) macos ios.

I originally didn't want to open separate issue but it looks like other people started running into similar issues on Azure Devops when the new 400-band SDK was released (taken from Twitter in the dotnet community):
image

This is heads up that something likely got broken even though I cannot quite nail down what. The common thing seems to be that both of us use some sort of CI environment. In our case (GitHub Actions w/ self-hosted runners) the .NET installation is local in the $HOME/.dotnet directory, installed through actions/setup-dotnet. I will try to direct people to this issue to share more details.

@dotnet-issue-labeler dotnet-issue-labeler bot added Area-Workloads untriaged Request triage from a team member labels Aug 11, 2022
@filipnavara
Copy link
Member Author

filipnavara commented Aug 11, 2022

I can reproduce it:

  • Create a new project with dotnet new console.
  • Add global.json with dotnet new globaljson.
  • Edit the .csproj TFM to <TargetFramework>net6.0-macos</TargetFramework>
  • Edit global.json to pin to 6.0.203 SDK: "version": "6.0.203"
  • Install 6.0.203 and 6.0.400 SDKs. I used this:
    • ./dotnet-install.sh -v 6.0.203 --install-dir ~/.dotnet
    • ./dotnet-install.sh -v 6.0.400 --install-dir ~/.dotnet
  • Install macos workload for 6.0.203 SDK: ~/.dotnet/dotnet workload install --sdk-version 6.0.203 macos
  • Add global.json with dotnet new globaljson.
  • Edit global.json to pin to 6.0.203 SDK: "version": "6.0.203".
  • Run build with dotnet build

Output:

/Users/filipnavara/.dotnet/sdk/6.0.203/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.ImportWorkloads.targets(38,5): error NETSDK1147: To build this project, the following workloads must be installed: macos [/Users/filipnavara/test/test.csproj]
/Users/filipnavara/.dotnet/sdk/6.0.203/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.ImportWorkloads.targets(38,5): error NETSDK1147: To install these workloads, run the following command: dotnet workload install macos [/Users/filipnavara/test/test.csproj]

@filipnavara
Copy link
Member Author

cc @dsplaisted

@filipnavara filipnavara changed the title Possible breakage with workload resolution/installation in SDK 6.0.400 Breakage with workload resolution/installation in SDK 6.0.400 Aug 11, 2022
@dsplaisted
Copy link
Member

I tried the repro steps on Windows and it worked OK.

@marcpopMSFT, could someone with a Mac try reproing this?

@filipnavara

This comment was marked as outdated.

@filipnavara

This comment was marked as outdated.

@filipnavara filipnavara changed the title Breakage with workload resolution/installation in SDK 6.0.400 Breakage with workload installation in SDK 6.0.400 Aug 11, 2022
@filipnavara

This comment was marked as outdated.

@filipnavara
Copy link
Member Author

filipnavara commented Aug 11, 2022

Ah, I figured it out... I reversed some of the steps in the instructions. I can reproduce it again. The key is to run ~/.dotnet/dotnet workload install --sdk-version 6.0.203 macos before global.json exists. Still cannot reproduce it on Windows though.

@filipnavara
Copy link
Member Author

Reduced repro on macOS. Install SDK 6.0.400 and run dotnet workload install --sdk-version 6.0.203 macos --print-download-link-only.

Observe that the --sdk-version parameter is ignored:

Resolving package URLs for workload(s) macos...
Skipping NuGet package signature verification.
==allPackageLinksJsonOutputStart==
[
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.sdk.android.manifest-6.0.400/32.0.448/microsoft.net.sdk.android.manifest-6.0.400.32.0.448.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.sdk.ios.manifest-6.0.400/15.4.447/microsoft.net.sdk.ios.manifest-6.0.400.15.4.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.sdk.maccatalyst.manifest-6.0.400/15.4.447/microsoft.net.sdk.maccatalyst.manifest-6.0.400.15.4.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.sdk.macos.manifest-6.0.400/12.3.447/microsoft.net.sdk.macos.manifest-6.0.400.12.3.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.sdk.maui.manifest-6.0.400/6.0.486/microsoft.net.sdk.maui.manifest-6.0.400.6.0.486.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.sdk.tvos.manifest-6.0.400/15.4.447/microsoft.net.sdk.tvos.manifest-6.0.400.15.4.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.workload.mono.toolchain.manifest-6.0.300/6.0.8/microsoft.net.workload.mono.toolchain.manifest-6.0.300.6.0.8.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.workload.emscripten.manifest-6.0.300/6.0.4/microsoft.net.workload.emscripten.manifest-6.0.300.6.0.4.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.macos.sdk/12.3.447/microsoft.macos.sdk.12.3.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.macos.ref/12.3.447/microsoft.macos.ref.12.3.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.macos.runtime.osx-arm64/12.3.447/microsoft.macos.runtime.osx-arm64.12.3.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.macos.runtime.osx-x64/12.3.447/microsoft.macos.runtime.osx-x64.12.3.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.macos.templates/12.3.447/microsoft.macos.templates.12.3.447.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.runtime.monoaotcompiler.task/6.0.8/microsoft.net.runtime.monoaotcompiler.task.6.0.8.nupkg",
  "https://api.nuget.org/v3-flatcontainer/microsoft.net.runtime.monotargets.sdk/6.0.8/microsoft.net.runtime.monotargets.sdk.6.0.8.nupkg"
]
==allPackageLinksJsonOutputEnd==

@dsplaisted
Copy link
Member

Yes, --sdk-version may well be broken. It is broken for printing the download links, and #27064 should fix that, though I'm not sure it will fix it for actual installation.

Instead of using --sdk-version can you run the dotnet workload install after creating the global.json file which specifies 6.0.203? That should cause that version of the SDK to be the one that actually installs the workload instead of trying to do a cross-version install, which is a lot more tricky.

@filipnavara
Copy link
Member Author

Instead of using --sdk-version can you run the dotnet workload install after creating the global.json file which specifies 6.0.203?

The original issue was observed on CI machines where the .NET SDK and workload setup was done as part of the workflow/pipeline. That's why it didn't use the global.json file and that worked in 6.0.3xx SDKs. Specifically in our case we needed it only in some cases where we didn't even want to force the SDK version but it happened to be necessary to workaround other bug.

@marcpopMSFT
Copy link
Member

--sdk-version was really never meant to be used as your using it I don't think. I think we created it to enable VSMac scenarios and that's why it's hidden. I definitely recommend using global.json instead.

Out of curiosity, are you using sdk-version to install workloads into a different SDK (which global.json would be better for) or install different workloads into the current sdk (which rollback would be better for)?

@filipnavara
Copy link
Member Author

Out of curiosity, are you using sdk-version to install workloads into a different SDK (which global.json would be better for) or install different workloads into the current sdk (which rollback would be better for)?

To install workloads for specific SDK band.

I cannot use global.json for our use case. We have a mix of old style xamarin.ios TFM projects that reference multi-TFM shared library .csproj. Since legacy Xamarin projects have to run with "desktop" MSBuild I get into an untested scenario that was broken since 6.0.300 band (and reported in #25651). When we actually build the net6.0-Xxx projects in the same repository we want to get the latest SDK.

The scenario with setting up CI machines without having access to global.json is quite common and creating it only to install the correct workloads seems like quite a hassle given that there was a command line option (which actually worked before!).

@filipnavara
Copy link
Member Author

Also, it apparently behaves differently on different systems (Windows/macOS) which seems weird.

@marcpopMSFT
Copy link
Member

Can you not use a global.json, install workloads, and then delete the global.json? Without a global.json, am I understanding correctly that it's not that you use the 6.0.200 sdk but rather you use the workloads from 6.0.200? Could you use a rollback file to solve this?

My change might fix this but it's really hard to tell as that switch was not meant for this scenario unfortunately. It was originally fairly limited in scope (hence why it's hidden). I pinged the mono folks on the linked items to see if they can investigate as it looks like a mono issue.

@filipnavara
Copy link
Member Author

filipnavara commented Aug 12, 2022

Can you not use a global.json, install workloads, and then delete the global.json?

Well, that's one of the workarounds. It's just regression from 6.0.300 and few extra steps in the CI pipelines.

Without a global.json, am I understanding correctly that it's not that you use the 6.0.200 sdk but rather you use the workloads from 6.0.200?

The desktop MSBuild picks the 6.0.2xx SDK (if both 6.0.400 and 6.0.2xx are installed) along with the workloads for that band. Since the installation now installs the workloads for the latest band instead of the old - explicitly specified - one I end up with non-working scenario.

Could you use a rollback file to solve this?

As far as I can tell, no. Don't rollback files still install the workloads for the latest SDK band? Even if they didn't then I would need to be explicit about the versions of each package. I prefer not to. I want to get the latest servicing for a given SDK band.

@marcpopMSFT marcpopMSFT added this to the Backlog milestone Aug 23, 2022
@marcpopMSFT
Copy link
Member

Checking back on this, is there still a need to use one SDK to install workloads into a different SDK? That was never part of the design and not really was --sdk-version was for. Using each SDk to install for itself and using rollback files is the guidance. Are you unblocked by moving forward to a newer sdk/vs combo?

@filipnavara
Copy link
Member Author

filipnavara commented Oct 12, 2022

Checking back on this, is there still a need to use one SDK to install workloads into a different SDK?

Yes, there is, until #25651 is fixed.

@marcpopMSFT
Copy link
Member

That issue explains why you need to install older workloads into an older SDK but not why you need --sdk-version. I'm still not clear why you can't use a global.json to pin to the older SDK, then install workloads at that time with a rollback file if you need a specific version, and build that way.

I pinged Larry on the other issue as likely on the runtime side.

@filipnavara
Copy link
Member Author

filipnavara commented Oct 13, 2022

That issue explains why you need to install older workloads into an older SDK but not why you need --sdk-version.

The workload installation is done by the CI job at the beginning of the process. It is done independently of checking out the source code.

I'm still not clear why you can't use a global.json to pin to the older SDK, then install workloads at that time with a rollback file if you need a specific version, and build that way.

Most projects in the solution actually want the latest SDK and workloads since they target .NET 6. I need to install the workloads in both the current SDK and the old one to allow compilation of the legacy Xamarin projects. global.json doesn't help here because I would need more than one to accomplish this and that's quite clunky.

I pinged Larry on the other issue as likely on the runtime side.

It's on SDK/MSBuild side, it was already moved between Xamarin, dotnet/runtime and dotnet/sdk repositories in the past.

@marcpopMSFT
Copy link
Member

I appreciate the explanation. That helps me better understand why you need it as it particular to the ordering of your CI build. Note that we --sdk-version was not intended for this nor is your scenario intended to work at this time.

As for the other bug, I wouldn't expect it to be sdk related. I would have thought it'd be something in the workload targets. That being said, it looks like you're using mono. I don't think we've tested or support workloads with mono. How old are the mono bits at this point?

CC @Redth @lewing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Workloads untriaged Request triage from a team member
Projects
None yet
Development

No branches or pull requests

3 participants