[WebToolsE2E] [Unstable] When running Aspire app, we sometimes see that the run session fails to start with error context deadline exceeded

[balachir]

INSTALL STEPS

1. Clean machine: Win11 x64 23h2 ENU
2. Install VS 17.10 P3, Web workload
   • Latest build rel.d17.10-34804.81, Preview channel
   • Includes Aspire 8.0.0-preview.5.24201.12
   • Includes .NET SDK SDK 8.0.300-preview.24203.14 and runtime 8.0.2
   • Apply feeds
3. Install latest daily azd build
   • version 1.8.0-beta.1-daily.3669344

REPRO STEPS

1. New Aspire Starter app > Use redis option is unchecked > AspireStarterApp1
2. Build and run (Ctrl+F5) > works fine and launches dashboard in browser
3. Close dashboard and debug console
4. Right-click AppHost project > Publish to ACA - create a new publish environment
5. Publish to ACA - works fine
6. Make a minor change in AppHost\Program.cs
7. Build and run (Ctrl+F5)

ACTUAL
Debug console shows errors and Dashboard shows that the services failed to start

Note: This repro is not very stable. The Repro Steps are what I did when I saw the issue reproduce.

Workaround
Give it a minute, close the windows and try again - it works fine.

EXPECTED
Dashboard launches the services successfully

[JohnGalt1717]

We're seeing this with Beta 5 right now. Just hitting run in VS.net causes this randomly and blocks dev. We're going to have to rollback to Beta 4.

[rmja]

Is there a workaround here?

[karolz-ms]

The workaround is to use the latest daily build of Aspire https://github.com/dotnet/aspire/blob/main/docs/using-latest-daily.md

Sorry, the fix is included in Aspire P6 but it did not make it to Aspire P5.

[JohnGalt1717]

This is a pretty big blocker that makes using aspire basically impossible except for in the most basic of cases... I'd suggest a point roll to fix this because you can't go back either because vs.net preview doesn't work with B4 properly so you have to revert all of your visual studio to a version that you can't even get at directly.

[karolz-ms]

@JohnGalt1717 not sure what makes you say this. I have been testing/developing Aspire with VS public preview laterly, no problems. What issues have you encountered?

[rmja]

@karolz-ms I am seeing something similar. I was on aspire preview3 and it worked just fine. I then updated vs to 17.10.0 preview 3 yesterday, and the aspire apphost project suddenly could not run anymore. I then decided to update the aspire dependencies to the most recent preview, but then I get this issue with projects being in "Failed to start" state.

[karolz-ms]

@rmja understood, and sorry again.

My recommendation to get rid of this issue would be to switch to the latest (main branch) Aspire bits, at least till P6 ships. I have been using public previews of VS (running VS 17.10.0 P3 as of today, like you do) and it works with latest Aspire, with the caveat that you need to use the .NET 8 NuGet source like https://github.com/dotnet/aspire/blob/main/docs/using-latest-daily.md says.

[rmja]

@karolz-ms Yes, that seems to fix issue for now - thanks.

[JohnGalt1717]

@JohnGalt1717 not sure what makes you say this. I have been testing/developing Aspire with VS public preview laterly, no problems. What issues have you encountered?

This issue literally happens 9/10 times that I try and run the host. It makes it completely unsuable and others are going to try out Aspire and have the same experience. And worse I only have a Yarp proxy and 5 microservices. Others will have much more than that which will basically guarantee failure and have to come looking for this and hack their environment and wait at least a month for a fix and hope nothing else gets broken in the mean time.

[rmja]

I was able to work around this issue by removing code from Program.cs in my services that could take time. For example by deferring connecting to referenced services that may not be online (e.g. redis). This apparently made the time from the start of Program.cs until app.Run() fast enough in the services that the aspire apphost does not timeout.

[dbreshears]

Another way to get this fix would be to run https://github.com/dotnet/aspire/blob/main/eng/installLatestFromReleaseBranch.ps1 which will install the preview 6 build.

[JohnGalt1717]

This is what I get when I run the script or do it by hand:

Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net7 doesn't exist.
Updated advertising manifest microsoft.net.sdk.maccatalyst.
Installing Microsoft.Android.Sdk.Windows.Msi.x64 ........ Done
Downloading Microsoft.Android.Sdk.Windows.Msi.x64 (33.0.95)
Workload installation failed. Rolling back installed packs...
Removing Microsoft.Android.Sdk.Windows.Msi.x64 ...... Done
Workload update failed: One or more errors occurred. (Version 33.0.95 of package microsoft.android.sdk.windows.msi.x64 is not found in NuGet feeds https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json".)

It appears that Aspire.StackExchange.Redis is required by DistributedCaching and OutputCaching but the right version of that package isn't in the dotnet8 nuget registry for preview 6 and thus can never work because it tries to use the version for .net 9 (that itself is not in the dotnet8 nuget package library)

Hence there's no way to use the preview nuget package library to work around this problem.

Which brings me back to: We need a down port to preview 5 that fixes this show stopper.

[joperezr]

Can you try running dotnet workload update without passing in any additional feeds to see if that fixes the issue? Make sure to run it from a directory where there are no NuGet.configs so that you use nuget.org as the feed

[JohnGalt1717]

Workload installation failed. Rolling back installed packs...
Installation rollback failed: Could not find a part of the path 'C:\Users\xxx.dotnet\sdk-advertising\8.0.300-preview.24203\microsoft.net.workloads\workloadVersion.txt'.
Workload update failed: Could not find a part of the path 'C:\Users\xxx.dotnet\sdk-advertising\8.0.300-preview.24203\microsoft.net.workloads\workloadVersion.txt'.

[joperezr]

cc @dsplaisted @marcpopMSFT Can you folks help here? For context, we suggested folks to run our installLatest ps1 script which uses rollback files to update the version of Aspire they use. From the above, looks like perhaps when fetching for advertising manifest sometime before running the script a feed was used which found new manifests but not sure what feed was used there. Is there a way to get @JohnGalt1717 back into a working state?

[JohnGalt1717]

Ok, it looks like I got it working. Wow, that was a process:

1. Leave your ServiceDefaults alone. Don't touch the P5 versions, there aren't any new ones for P6 yet.
2. Add the nuget reference for the beta nuget feed per the instructions.
3. Use the script to update your workloads.
4. In your host, you need these package references:

		<PackageReference Include="Aspire.Hosting" Version="8.0.0-preview.6.24178.5" />
		<PackageReference Include="Aspire.Hosting.AppHost" Version="8.0.0-preview.6.24178.5" />
		<PackageReference Include="Aspire.Hosting.Redis" Version="8.0.0-preview.6.24178.5" />
		<PackageReference Include="Grpc.AspNetCore" Version="2.62.0" />
		<PackageReference Include="Polly" Version="8.3.1" />
		<PackageReference Include="KubernetesClient" Version="13.0.26"/>

Failure to have exactly this will cause the project to not build.
5. You must disable Source Mapping or it blows up.
6. You'll still get errors in the console long on run, but it works as it should from there.

[dsplaisted]

Workload installation failed. Rolling back installed packs...
Installation rollback failed: Could not find a part of the path 'C:\Users\xxx.dotnet\sdk-advertising\8.0.300-preview.24203\microsoft.net.workloads\workloadVersion.txt'.
Workload update failed: Could not find a part of the path 'C:\Users\xxx.dotnet\sdk-advertising\8.0.300-preview.24203\microsoft.net.workloads\workloadVersion.txt'.

Workload installation failed. Rolling back installed packs... Installation rollback failed: Could not find a part of the path 'C:\Users\xxx.dotnet\sdk-advertising\8.0.300-preview.24203\microsoft.net.workloads\workloadVersion.txt'. Workload update failed: Could not find a part of the path 'C:\Users\xxx.dotnet\sdk-advertising\8.0.300-preview.24203\microsoft.net.workloads\workloadVersion.txt'.

That is an error we currently have with workload sets, which we are still working on. But you have to switch to workload set mode to hit that issue, and I'm surprised that anyone would be in that mode since it's not something that works yet. Do you know how that mode was enabled?

Anyway, you should be able to switch out of workload set mode by running dotnet workload update --mode loosemanifest, and then you shouldn't get that error when you try to update. Note that we've changed how you change the mode recently, in newer builds it will be dotnet workload config --update-mode manifests.

[JohnGalt1717]

That fixed the pull it appears. It was in that mode because the instructions say to do this if you run it manually instead of in the script.

[JohnGalt1717]

I take both my work around and this back.

The work around compiles and everything starts, but then service discovery doesn't work with yarp etc.

And the workload update failed: (just on a different package)

Workload update failed: One or more errors occurred. (Version 16.4.7142 of package microsoft.ios.sdk.msi.x64 is not found in NuGet feeds https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json".)

Still back to we need a patch of P5 of Aspire that fixes this show stopper blocker bug so that it can be used and you don't have to endure this mess.

[gabrewer]

I am having the same issue. I was able to eventually get the daily build installed but I need to add

Aspire.Hosting.Azure.ApplicationInsights
Aspire.Hosting.Azure.Storage

and now I get these errors.

error NU1102: Unable to find package Azure.Provisioning with version (>= 1.0.0-alpha.20240315.2)
error NU1102: - Found 4 version(s) in nuget.org [ Nearest version: 0.2.0-beta.2 ]
error NU1102: - Found 0 version(s) in Microsoft Visual Studio Offline Packages
error NU1102: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
error NU1102: - Found 0 version(s) in dotnet8
error NU1102: Unable to find package Azure.ResourceManager.Authorization with version (>= 1.2.0-alpha.20240309.1)
error NU1102: - Found 7 version(s) in nuget.org [ Nearest version: 1.1.1 ]
error NU1102: - Found 0 version(s) in Microsoft Visual Studio Offline Packages
error NU1102: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
error NU1102: - Found 0 version(s) in dotnet8
error NU1102: Unable to find package Azure.ResourceManager.KeyVault with version (>= 1.3.0-alpha.20240310.2)
error NU1102: - Found 15 version(s) in nuget.org [ Nearest version: 1.2.1 ]
error NU1102: - Found 0 version(s) in Microsoft Visual Studio Offline Packages
error NU1102: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
error NU1102: - Found 0 version(s) in dotnet8
error NU1102: Unable to find package Azure.ResourceManager.Resources with version (>= 1.8.0-alpha.20240309.1)
error NU1102: - Found 19 version(s) in nuget.org [ Nearest version: 1.7.1 ]
error NU1102: - Found 0 version(s) in Microsoft Visual Studio Offline Packages
error NU1102: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
error NU1102: - Found 0 version(s) in dotnet8

[mitchdenny]

@gabrewer at first glance your issue looks more to be around straight forward package restoration. Could you create a new issue for that so we don't get these two issues confused? It would be great if you could also post a small repro of the issue in a GitHub repository so we can pull it down on a clean machine with Aspire preview 5 installed and verify that things are working as they should.

I tried preview 5 on a clean machine specifically with Azure Storage and it restored, built and ran fine.

[mitchdenny]

@JohnGalt1717 I just want to confirm that the issue you are seeing if you are running Aspire P5 is what is described in the original report above (i.e. context deadline exceeded errors?).

As I read through the thread more I can see the above is the case. So the brief summary of your adventures so far is that you were using Aspire P4, updated to Aspire P5 and started seeing the context deadline exceeded problems.

@karolz-ms suggested using the latest P6 build which hasn't worked for you.

We don't release patches for preview builds. We've been working this past week to finalize P6 and so it makes more sense to understand what the blocking issues are there and get them addressed.

[thimobuchheister]

@gabrewer at first glance your issue looks more to be around straight forward package restoration. Could you create a new issue for that so we don't get these two issues confused? It would be great if you could also post a small repro of the issue in a GitHub repository so we can pull it down on a clean machine with Aspire preview 5 installed and verify that things are working as they should.

I tried preview 5 on a clean machine specifically with Azure Storage and it restored, built and ran fine.

I created a repo which showcases the problem @gabrewer mentioned. I ran into the same problem when trying to use Preview 6 daily builds.

[DamianEdwards]

Can this be closed now it's marked Done?

[dbreshears]

Yes. It is fixed in Preview 6.