Upgrade Windows 2019 image Visual Studio 16.8

[robertmclaws]

Hey folks! Just wanted to give you folks a heads-up that a breaking change was introduced into .NET 5 that required mixed Framework / Core projects to use a version of MSBuild that only ships with the latest release. Since the "VS Preview" image was eliminated, there hasn't been a way to build our mixed projects in Azure DevOps for several months.

I'm raising this ticket to make sure VS2019 gets upgraded in a timely manner, so those of us that can't build with dotnet build can still enjoy the magic that is .NET 5. You're awesome, thanks so much! 👊🏻

[Darleev]

Hello @robertmclaws,

We keep the visual studio version always up to date. However, deploying images can sometimes take a week(or two).

I will let you know right after the next image rollout.

[tsigereda]

We are also waiting for this, we couldn't build our projects (Azure devops) that we upgraded to .Net 5 and waiting eagerly for the VS 16.8 to be available in the images.

[kvpt]

While waiting for the image to be updated a workaround is to update Visual Studio directly within the script.

This can be done like mentioned here.

I added these steps to my build pipeline (it add 5/10 minutes to the build but the build pass now).

- bash: dotnet tool update -g dotnet-vs
- bash: vs install preview -sku:enterprise --quiet +Microsoft.VisualStudio.Component.ManagedDesktop.Core +Microsoft.NetCore.Component.DevelopmentTools
- bash: echo "##vso[task.prependpath]$(vs where preview --prop=InstallationPath)\MSBuild\Current\Bin"

No need to use the Net installer to install Net5 in this case as it is included with Visual Studio.
I tried to use the vs update command, but it was not working so I choose to install the preview version like in the blog post.

[maxim-lobanov]

Hello everyone,
We have some delay with delivering Visual Studio 16.8. This version contains a bunch of breaking changes and could break customers who use Nuget < 5.8. See details in https://developercommunity.visualstudio.com/content/problem/1248649/error-netsdk1005-assets-file-projectassetsjson-doe.html
We are working to mitigate it and trying to reduce impact.
Thank you for your patience

[alepauly]

Hi @jackbond, we weren't oblivious but we missed one piece of the puzzle. We are obviously taking it as something to do better next time and planning the work to get it right. As for a timeline, we are targeting early next week. I'll update here if that slips.

Sorry all for the delay!

[0ChristopherCampbell]

How long are we talking here? Also, I'm more than happy to be called a dick, but seriously, it seems that certain factions within Microsoft were COMPLETELY OBLIVIOUS to the fact that .NET5 was under development. I mean, what's the point of having previews if certain Microsoft infrastructure teams simply have the attitude, "well gee, we'll cross that bridge when we get to it." All the work you are doing to fix this should have been done PRIOR to Tuesday.

Anyone surprised at the amount of breaking changes, terrible documentation (plenty of pages where they just change the version number but don't actually check if the documentation is accurate. I dare you to try and implement custom storage providers for .net core identity.. The documentation is a F***ing joke) and general frustration that comes along with EVERY release of .NET Core to date really hasn't been paying attention. It's been an absolute shit show from the very start. Ever since they went 'open source'..

EF Core Many->Many relationships anyone? Waited forever and they couldn't even implement it without some hacky work around syntax.

Microsoft wake up, you're falling further and further behind.

[robertmclaws]

I found myself not wanting to wait 2 weeks for a new release, and also not wanting to wait an additional 10 minutes for already long builds. So I posted a simple solution here: dotnet/core#5120 (comment)

Short version: Use Chocolatey to add the latest VS Build Tools to the server, and compile using that.

To the Azure DevOps team: it would make these situations easier if the MSBuild task let you specify alternate msbuild.exe locations.

HTH!

[p2pbsh]

I moved my project over to .NET5 when the first RC dropped, which was advertised as 'production ready and supported', however as my CI build requires Visual Studio my Azure pipeline has effectively been unusable for over two months now.

Looking ahead to .NET6, I think it would be highly desirable to add back support for preview builds of VS on the Windows 2019 image to eliminate this pain point of early adopters and testers of .NET previews.

[robertmclaws]

To the Azure DevOps team's credit, this process is mostly automated, so they push new software up into the image as it is released.

I think the core issue here is 1) the VS / MSBuild team pushed a requirement for a new version of MSBuild that maybe didn't understand the full ramifications of that decision, b) the consolidation of images meant that there is no longer a "VS Preview" image that has the very latest preview bits, and c) the tasks available to the Pipelines don't make it easy to specify different MSBuild executables (mostly because that hasn't been necessary for a long time).

All small problems by themselves, but in the context of trying to "Go Live" with an RC, turned out preventing us from taking advantage of that.

Hopefully my code helps out for now, and then we can get a Preview image that always has the latest stuff.

Thanks!

[saddamhossain]

Hi,
I am facing troubles when I am doing CI/CD with Azure DevOps with dot net 5.0. It's said to build failed. It worked for asp.net core 3.1 but It's not working in .NET 5.0. I am using VSTS. I am trying it since the release of .NET 5.0

The error I face is
C:\Program Files\dotnet\sdk\3.1.403\Microsoft.Common.CurrentVersion.targets(1177,5): Error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks

[robertmclaws]

@saddamhossain I posted how to fix this here: dotnet/core#5120 (comment)

[saddamhossain]

@robertmclaws Thank you so much. I found the solution.

[maxim-lobanov]

Hello everyone!
Just an update, image with VS 16.8 is rolling out right now. Some machines are already updated.
Deployment should be finished in a few days

[JamesDSch]

That's great news @maxim-lobanov ! To set expectations, is the ongoing rollout (and the "few days" duration) applicable to both Azure Devops and Github Actions?

[maxim-lobanov]

@JamesDSch , Yes, we deploy both Azure DevOps and GitHub Actions in parallel.

[p2pbsh]

@maxim-lobanov Thanks for the update. Out of interest, does this image have 16.8 and the .NET5 SDK, or is the plan to still add the SDK later?

[maxim-lobanov]

@p2pbsh This image contains only Visual Studio 16.8 without separate .NET 5 installation. But VS itself contains .NET 5 if you use VS to build apps.
We are going to install .NET 5 on images in a few weeks, see #1891 (comment)
Until that, it could be installed in runtime via setup-dotnet task (GitHub Actions) or Use .NET Core task (Azure DevOps)

[dahlbyk]

We have some delay with delivering Visual Studio 16.8. This version contains a bunch of breaking changes and could break customers who use Nuget < 5.8. See details in https://developercommunity.visualstudio.com/content/problem/1248649/error-netsdk1005-assets-file-projectassetsjson-doe.html

Just documenting that I ran into this error starting today with the VS 16.8 upgrade in Windows Server 2019 (20201116 update) #2076:

error NETSDK1005: Assets file 'D:\a[redacted][redacted]\src[redacted]\obj\project.assets.json' doesn't have a target for 'netstandard1.6'. Ensure that restore has run and that you have included 'netstandard1.6' in the TargetFrameworks for your project.
error NETSDK1005: Assets file 'D:\a[redacted][redacted]\src[redacted]\obj\project.assets.json' doesn't have a target for 'net472'. Ensure that restore has run and that you have included 'net472' in the TargetFrameworks for your project.

Update: NuGet 5.8.0 is now ReleasedAndBlessed, so this fix should no longer be necessary.

---

Fix iswas to use NuGet 5.8, which nuget/setup-nuget doesn't recognize as latest yet (not ReleasedAndBlessed yet at https://dist.nuget.org/tools.json):

     - uses: nuget/setup-nuget@v1
+      with:
+        nuget-version: '5.8.x'

[patdaman]

Our builds broke today due to this, right when we had to release Hotfixes to Production! If these were breaking, Azure DevOps should have rolled back, or at least introduced another version with the previous image. It's crap to wake up to nasty messages from developers and customers who cannot get a fix out on time.

[AlenaSviridenko]

Hi @patdaman,
based on our image testing, VS 16.8 itself doesn't contain breaking changes, but it's having issues when working with Nuget < 5.8 https://developercommunity.visualstudio.com/content/problem/1248649/error-netsdk1005-assets-file-projectassetsjson-doe.html We are working on adding backward compatibility with older Nuget versions.

Could you please check if you are using latest Nuget?

If the issue persists with latest Nuget, please, file an issue in our repo with minimal repro steps, and we will investigate it.
Unfortunately, we are not covering all possible use cases with our tests, but we are constantly working on increasing their coverage.

[Naninani]

I think this change was rolled out yesterday to our ado instance and the majority of our builds running there started failing. Could you roll this back? I'd be great if you could send announcements with the breaking changes and what actions needs to be taken by the customers.
In addition there are tasks that rely on older versions that customers do not own, so more thorough testing would be appreciated.

##[error]C:\Program Files\dotnet\sdk\5.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): Error NETSDK1005: Assets file 'D:\a\1\s\Source\Common\Notifications\obj\project.assets.json' doesn't have a target for 'netcoreapp3.1'. Ensure that restore has run and that you have included 'netcoreapp3.1' in the TargetFrameworks for your project.

The framework 'Microsoft.NETCore.App', version '2.1.0' was not found.
- The following frameworks were found:
3.1.10 at [C:\hostedtoolcache\windows\dotnet\shared\Microsoft.NETCore.App]
You can resolve the problem by installing the specified framework and/or SDK.
The specified framework can be found at:
- https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=2.1.0&arch=x64&rid=win10-x64
##[error]The process 'C:\hostedtoolcache\windows\dotnet\dotnet.exe' failed with exit code 2147516566

[smkanadl]

Seen the exact same on my hosted agents this evening. I have the latest NuGet installed and was using the DotNetCoreCli Task and NO UseDotNet task. But the logs reported that the Net5 ask was used in the build.

[LanceMcCarthy]

That error is because .NET Core 2.1 is deprecated, it was removed in VS 2019 16.8 (I dont remember where I read the warning, but Msft was clear that it was being removed).

You can however easily fix it by explicitly adding a Use .NET Core task.

Azure DevOps

For example in Azure DevOps classic pipeline

or a YAML pipeline:

    - task: UseDotNet@2
      inputs:
        packageType: 'sdk'
        version: '2.1.0'

GitHub Actions workflow

GitHub worflows use the setup-dotnet Action https://github.com/marketplace/actions/setup-net-core-sdk

    - name: Install .NET Core
      uses: actions/setup-dotnet@v1.7.2
      with:
        dotnet-version: '2.1.0'

[p2pbsh]

On the flip side, 16.8 image in combo with NuGet 5.8 and UseDotNet task got all my broken NET5 projects building again, so please don't roll back :)

[Naninani]

The issue is more complex as it could affect internal ADO tasks that are central and would block many MS orgs.
I think a more reasonable approach would be to roll back and make sure this change is communicated internally to the task owners teams with a timeframe to fix the dependencies.
I don't think its reasonable to ask customers to update 1000+ (our case) build pipelines that are affected by this change.

[LanceMcCarthy]

What I think would be a nice feature is to be able to pass in a specific image release number and the VM would spin up for that one (instead of from a hot pool of windows-latest).

Like a demand, for example:

• releases/win19/20201021

Yes, you know it is going to take more time to run the build because you need to wait for VM init, but you know exactly what you are getting.

In the meantime, if you cannot add tasks to a pipeline, a faster fix may be to just switch to a self-hosted pool.

[robertmclaws]

I think the thing that sticks out at me from this .NET 5 release from a DevOps perspective is that Microsoft has data on every task that is being run, and the settings for each task. Microsoft should be able to know telemetry on the makeup of the build processes of their customers, and be able to warn specific customers when upcoming VM image default changes will break builds. They should also be able to have one-click mitigation options, like for example when .NET Core 2.X becomes unsupported, clicking the "Fix Me" button should add the .NET Core 2.1 task in.

At the VERY least, there should be a build log analyzer that knows common failures in the build pipeline and recommends fixes.

[robertmclaws]

I totally get where you're coming from @jackbond, and in a previous life, I would wholeheartedly agree with you. However, Azure DevOps pipelines are basically just highly customizable Powershell commands, and you can actually do quite a bit yourself (including installing different Visual Studio instances).

If you look at my earlier posts in this thread, I was able to get .NET 5 & the new MSBuild installed and compiling my builds in a couple hours over a weekend, with nothing more than Googlefu + trial and error. Yeah, breaking the builds sucks, but you need to be in a mindset that your build pipeline cannot be static, it will always be adapting.

I think the biggest issue overall is that the VS team underestimated the impact of requiring updated MSBuild versions would have on customers. That disconnect comes from Microsoft assuming that projects are homogenous (pure .NET 5 that can be built from the command line) vs realizing that different parts of an architecture can and do run different frameworks.

[tejasd1990]

@LanceMcCarthy dotnet 2.1 is not deprecated, its eol date is in August 2021. Both 2.1 sdk and 2.1 runtimes are present on hosted agents. You can check that by running dotnet --list-sdks \ dotnet --list-runtimes through a script task.

@Naninani , the error message ##[error]C:\Program Files\dotnet\sdk\5.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): Error NETSDK1005: Assets file 'D:\a\1\s\Source\Common\Notifications\obj\project.assets.json' doesn't have a target for 'netcoreapp3.1'. Ensure that restore has run and that you have included 'netcoreapp3.1' in the TargetFrameworks for your project. is because of incompatibility of dotnet 5 with older nuget versions(<=5.7). So the more appropriate workaround is to upgrade nuget to 5.8 in your pipeline as suggested by @dahlbyk, rather than downgrading dotnet
Also, can you tell which task is giving this other error message The framework 'Microsoft.NETCore.App', version '2.1.0' was not found., and share the logs?

[dahlbyk]

The ship has sailed, of course, but new MSBuild being incompatible with restores from older NuGet versions seems like something that could have at least been a warning in the build log. I don't suppose NuGet leaves any evidence in packages or obj/ of what version was used to restore?

[bishal-pdMSFT]

@dahlbyk we have infact added warning logs to DotNetCoreCLI task and started rolling out on Friday. We are continuing with further rollout.

[miketimofeev]

Visual studio 16.8 was deployed last week.
I'm going to close the issue.
Thanks, everyone for your thougths\suggestions!