Framework Not Detecting On Linux When Executing "dotnet" Command

[sangeethnandakumar]

Description

I installed .NET 7.0.7 SDK + Runtime on my Linux machine.

On which, I deployed nearly 5 ASP.NET Core MVC apps made in .NET 7.0.0

1. After deployment and setting an Apache reverse proxy, 4 out of 5 projects started. One didn't start
2. It gives 503 SERVICE UNAVAILABLE
3. To troubleshooting the non working app, I ran dotnet <DLL NAME>.dll, But this give me the following error

4. So, I executed dotnet --info, But that gave me

5. Then I realized, ### If I run any of my websites using dotnet <NAME OF DLL>.dll command, I get the above framework not found error

Means, I can't explicitly run dotnet <NAME OF DLL>.dll command on any of my 5 projects deployed

7. However, Once served via Apache, I can hit 4 URLs and get the site in a browser, That's strange

What is happening here? Why it's telling me no framework found?

Reproduction Steps

Install .NET 7 on a Linux VM
Publish a dotnet app to it
Try running dotnet <NAME OF DLL>.dll

Expected behavior

Should be able to run the program on Linux by running dotnet <NAME OF DLL>.dll command.

Actual behavior

It's saying the framework is not installed. That's not true

Regression?

Worked till a few days ago on the same server on the same .NET version.

Known Workarounds

People are suggesting .NET ROOT path issue can cause this

Configuration

.NET 7.0.0 (apps)
.NET 7.0.3 (sdk)

Other information

No response

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I installed .NET 7.0.3 on my Linux machine.

On which, I deployed nearly 5 ASP.NET Core MVC apps made in .NET 7.0.0

1. After deployment and setting an Apache reverse proxy, 4 out of 5 projects started
2. To troubleshoot, I ran dotnet <DLL NAME>.dll, But this give me the following error

3. I executed dotnet --info, But that gave me

4. If I run any of my websites using dotnet <NAME OF DLL>.dll command, I get the above framework not found error
5. However, Once served via Apache, I can hit 4 URLs and get the site in a browser, That's strange

What is happening here? Why it's telling me no framework found?

Reproduction Steps

Install .NET 7 on a Linux VM
Publish a dotnet app to it
Try running dotnet <NAME OF DLL>.dll

Expected behavior

Should be able to run the program on Linux by running dotnet <NAME OF DLL>.dll command.

Actual behavior

It's saying the framework is not installed. That's not true

Regression?

Worked till a few days ago on the same server on the same .NET version.

Known Workarounds

People are suggesting .NET ROOT path issue can cause this

Configuration

.NET 7.0.0 (apps)
.NET 7.0.3 (sdk)

Other information

No response

Author:	sangeethnandakumar
Assignees:	-
Labels:	area-Host
Milestone:	-

[danmoseley]

did you try this?

[sangeethnandakumar]

It's the same stuff I installed. The ONly difference I saw was my apps are in .NET 7.0.0 while my SDK and runtime are 7.0.7.
My understanding is minor version upgrades are fully backward compatible in .NET. Is that understanding correct?

[agocke]

Yes, patch versions should be compatible. How did you install .NET originally? Also, what distro+version?

[vitek-karas]

You should have both Microsoft.NETCore.App and Microsoft.AspNetCore.App "runtimes" installed. The fact that you only have the ASP.NET one is weird. It means that the package install didn't work as it should. Could you please try also installing the Microsoft.NETCore.App?

@joeloff who might know which package behaves like this - that is installing only the ASP.NET runtime without its dependencies.

[joeloff]

@NikolaMilosavljevic as an FYI. The SDK packages should include all the runtimes. We do build packages that only contain ASP.NET, but AFAIK, that's only for Windows.

I installed dotnet-sdk-7.0 on my WSL instance and it listed bringing in all the runtime packages too for 7.0.304. I used the Microsft package feed (https://packages.microsoft.com/config/ubuntu/)

Which package feed was used to install the SDK where it only brought in the runtime?

[sangeethnandakumar]

I think the issue is different

1. I deployed 5 ASP.NET Core MVC websites behind Apache reverse proxy
2. 4 out of 5 are working. When accessed via URL from the internet
3. If runtime is not there how can they work??

Now the real issue

1. Because 1 was not working out of 5
2. I thought about starting it using the dotnet <DLL NAME>.dll command
3. That's when I got the above error

So I tried executing the same command on already working and deployed apps just to see

1. Strange enough any dlls show the same error
2. Then how do 4 out of 5 currently work behind Apache???

You can see ASP.NET Core runtime is installed with dotnet --info

Why can't I explicitly start any app using dotnet <DLL NAME>.dll

I use Microsoft feed for .NET installation

Doesn't install .NET SDK won't install .NET runtime in Linux?

[sangeethnandakumar]

Attaching a video demonstration of the issue mentioned above for brevity & more clarity

YouTube: https://www.youtube.com/watch?v=yJkIasUPWzU

[danmoseley]

Is it possible that you published the 4 working apps as standalone apps (they contain the runtime)? If so, combined with a global install that is somehow bad, it would explain things.

[agocke]

@sangeethnandakumar Your dotnet install is definitely broken/corrupted. How did you install it?

[sangeethnandakumar]

@danmoseley No brother. All 5 apps are deployed 'Framework Dependant'.

It was working fine and started behaving like this one random day.
I'm worried that any Ubuntu update is causing some chaos. It's a managed VM from AWS.

Any chance my issue is related to this : #79237

[vitek-karas]

Any chance my issue is related to this : #79237

It doesn't seem likely - it would have broken all apps, not just 1 out of 5.

Can you go to your /user/share/dotnet/shared and see if there is a Microsoft.NETCore.App directory there? If not then the install is somehow corrupted.

Why does it work from Apache (at least some of them)

There might be another install of .NET on your machine and the processes under Apache might be using that one. Can you please look for /etc/dotnet/install_location and see what's in that file? There might also be /etc/dotnet/install_location_x64 file so look into that one as well.

Look in your environment - especially in the Apache server if there's a DOTNET_ROOT env. variable set.

Try to run the app via the executable from command line - so in one of your examples just ./Twileloop - and see if that works.

Currently I'm thinking these might be two separate problems:

• One of your apps doesn't work in Apache - for this it would be great to collect the stdout and stderr of the failing app. If it was a .NET install problem the app would have printed something into the output.
• The dotnet command in your path is somehow corrupted. For that please confirm that it is corrupted (looking at the directory mentioned above). And the best solution is to probable just clean and reinstall .NET.

[sangeethnandakumar]

Thanks @vitek-karas for the detailed info.
I checked the respective directories and found the installations seem to be corrupted in some way.

As you suggested, I uninstalled the entire SDKs and runtimes and did a fresh install from scratch and all 5 apps starts to work.
This costs me a few minutes of downtime but it's fine since it worked anyway.

Now although it works now, I'm concerned with a few things
This was a production server and no maintenance has been done from my side, One random day it corrupted the .NET installation, and I'm still checking the logs and traces to see what happened. Since the VM is fully managed by AWS is it a Ubuntu update issue or not is also a concern for me. If so, this might happen again. Apart from that all works now

Closing the issue. Thanks for everyone helped

[vitek-karas]

@sangeethnandakumar thanks for letting us know that it worked.
I share your concern about the corruption of the install. That said if it was something regular, we would have seen more reports about it... which I'm not aware of.
I'm still curious how come 4 apps worked and 1 didn't, the missing MIcrosoft.NETCore.App is the runtime, so without it, nothing should run. But I totally understand you don't want to mess with a production sever more than necessary.

[NikolaMilosavljevic]

@sangeethnandakumar which Ubuntu version are you using?

On newer Ubuntu releases, .NET is present in distro-native package feeds. On those systems, users should not configure Microsoft feed (packages.microsoft.com), unless required for specific scenarios. Adding Microsoft feed would cause issues with missing SDKs or runtimes - similar to what you encountered.

Here is the issue describing various scenarios and solutions for misconfigured systems: dotnet/core#7699

[sangeethnandakumar]

@NikolaMiloavljevic I'm using Ubuntu 22.04 LTS. I believe it's the latest, also I can't control the OS version since the VM is a managed instance from AWS.

I faced issues using the native distro feed from Ubuntu while reinstalling this time.
Thus I pointed to the Microsoft feed and disabled the Ubuntu feed for packages matching 'dotnet.*' on my machine.

This forces the package to be installed only from the Microsoft feed. My runtime installations are right now from the Microsoft feed.
So far it's working fine, I may prefer upgrading to .NET 8 preview runtime by the end of this July

I'd set up everything including a backup server connected to a load balancer to redirect traffic in case the issue came back for some time.
I also developed & deployed a health check endpoint in all of my APIs and a monitoring and reporting system to let me know at the event of a potential future failure.