Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.NET runtime not found on Ubuntu #79237

Closed
ptr727 opened this issue Dec 5, 2022 · 24 comments
Closed

.NET runtime not found on Ubuntu #79237

ptr727 opened this issue Dec 5, 2022 · 24 comments

Comments

@ptr727
Copy link

ptr727 commented Dec 5, 2022

Description

My weekly GitHub actions build started failing.
I suspect some change in Ubuntu .NET 6 results in the apps no longer running, and complaining no runtime found.

Docker:

# Build externally
# dotnet publish ./PlexCleaner/PlexCleaner.csproj --runtime linux-x64 --self-contained false --output ./Docker/PlexCleaner
COPY PlexCleaner /PlexCleaner
RUN dotnet --info \
    && /PlexCleaner/PlexCleaner --version

Actions error:

#15 [10/10] RUN dotnet --info     && /PlexCleaner/PlexCleaner --version
#15 0.222 .NET SDK (reflecting any global.json):
#15 0.222  Version:   6.0.111
#15 0.222  Commit:    b3bb659a9d
#15 0.222 
#15 0.222 Runtime Environment:
#15 0.224  OS Name:     ubuntu
#15 0.224  OS Version:  22.04
#15 0.227  OS Platform: Linux
#15 0.251  RID:         ubuntu.22.04-x64
#15 0.251  Base Path:   /usr/lib/dotnet/sdk/6.0.111/
#15 0.254 
#15 0.254 global.json file:
#15 0.254   Not found
#15 0.254 
#15 0.254 Host:
#15 0.254   Version:      6.0.11
#15 0.254   Architecture: x64
#15 0.254   Commit:       943474ca16
#15 0.254 
#15 0.254 .NET SDKs installed:
#15 0.254   6.0.111 [/usr/lib/dotnet/sdk]
#15 0.254 
#15 0.254 .NET runtimes installed:
#15 0.254   Microsoft.AspNetCore.App 6.0.11 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
#15 0.254   Microsoft.NETCore.App 6.0.11 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
#15 0.254 
#15 0.254 Download .NET:
#15 0.254   https://aka.ms/dotnet-download
#15 0.254 
#15 0.254 Learn about .NET Runtimes and SDKs:
#15 0.254   https://aka.ms/dotnet/runtimes-sdk-info
#15 0.262 You must install .NET to run this application.
#15 0.262 
#15 0.262 App: /PlexCleaner/PlexCleaner
#15 0.262 Architecture: x64
#15 0.262 App host version: 6.0.11
#15 0.262 .NET location: Not found
#15 0.262 
#15 0.262 Learn about runtime installation:
#15 0.262 https://aka.ms/dotnet/app-launch-failed
#15 0.262 
#15 0.262 Download the .NET runtime:
#15 0.262 https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=ubuntu.22.04-x64&apphost_version=6.0.11
#15 ERROR: process "/bin/sh -c dotnet --info     && /PlexCleaner/PlexCleaner --version" did not complete successfully: exit code: 131

Full error: https://github.com/ptr727/PlexCleaner/actions/runs/3622494725/jobs/6107272543

Reproduction Steps

Install .NET 6 on ubuntu:latest.
Compile .NET core app in github actions step, copy .NET app to docker
ditnet --info works.
Run .NET app, complains no .NET found?

Expected behavior

Work as before.

Actual behavior

.NET location: Not found

Regression?

Yes, this worked in previous ubuntu:latest release, no code changes, github actions build just started failing.

Known Workarounds

No response

Configuration

https://github.com/ptr727/PlexCleaner/actions/runs/3616398576

Other information

No response

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Dec 5, 2022
@ghost
Copy link

ghost commented Dec 5, 2022

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

My weekly GitHub actions build started failing.
I suspect some change in Ubuntu .NET 6 results in the apps no longer running, and complaining no runtime found.

Docker:

# Build externally
# dotnet publish ./PlexCleaner/PlexCleaner.csproj --runtime linux-x64 --self-contained false --output ./Docker/PlexCleaner
COPY PlexCleaner /PlexCleaner
RUN dotnet --info \
    && /PlexCleaner/PlexCleaner --version

Actions error:

#15 [10/10] RUN dotnet --info     && /PlexCleaner/PlexCleaner --version
#15 0.222 .NET SDK (reflecting any global.json):
#15 0.222  Version:   6.0.111
#15 0.222  Commit:    b3bb659a9d
#15 0.222 
#15 0.222 Runtime Environment:
#15 0.224  OS Name:     ubuntu
#15 0.224  OS Version:  22.04
#15 0.227  OS Platform: Linux
#15 0.251  RID:         ubuntu.22.04-x64
#15 0.251  Base Path:   /usr/lib/dotnet/sdk/6.0.111/
#15 0.254 
#15 0.254 global.json file:
#15 0.254   Not found
#15 0.254 
#15 0.254 Host:
#15 0.254   Version:      6.0.11
#15 0.254   Architecture: x64
#15 0.254   Commit:       943474ca16
#15 0.254 
#15 0.254 .NET SDKs installed:
#15 0.254   6.0.111 [/usr/lib/dotnet/sdk]
#15 0.254 
#15 0.254 .NET runtimes installed:
#15 0.254   Microsoft.AspNetCore.App 6.0.11 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
#15 0.254   Microsoft.NETCore.App 6.0.11 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
#15 0.254 
#15 0.254 Download .NET:
#15 0.254   https://aka.ms/dotnet-download
#15 0.254 
#15 0.254 Learn about .NET Runtimes and SDKs:
#15 0.254   https://aka.ms/dotnet/runtimes-sdk-info
#15 0.262 You must install .NET to run this application.
#15 0.262 
#15 0.262 App: /PlexCleaner/PlexCleaner
#15 0.262 Architecture: x64
#15 0.262 App host version: 6.0.11
#15 0.262 .NET location: Not found
#15 0.262 
#15 0.262 Learn about runtime installation:
#15 0.262 https://aka.ms/dotnet/app-launch-failed
#15 0.262 
#15 0.262 Download the .NET runtime:
#15 0.262 https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=ubuntu.22.04-x64&apphost_version=6.0.11
#15 ERROR: process "/bin/sh -c dotnet --info     && /PlexCleaner/PlexCleaner --version" did not complete successfully: exit code: 131

Full error: https://github.com/ptr727/PlexCleaner/actions/runs/3622494725/jobs/6107272543

Reproduction Steps

Install .NET 6 on ubuntu:latest.
Compile .NET core app in github actions step, copy .NET app to docker
ditnet --info works.
Run .NET app, complains no .NET found?

Expected behavior

Work as before.

Actual behavior

.NET location: Not found

Regression?

Yes, this worked in previous ubuntu:latest release, no code changes, github actions build just started failing.

Known Workarounds

No response

Configuration

https://github.com/ptr727/PlexCleaner/actions/runs/3616398576

Other information

No response

Author: ptr727
Assignees: -
Labels:

area-Host, untriaged

Milestone: -

@Clockwork-Muse
Copy link
Contributor

Install .NET 6 on ubuntu:latest.

Side note: Is there some reason you're doing this yourself, instead of using one of the pre-built .net images?

@ptr727
Copy link
Author

ptr727 commented Dec 5, 2022

Install .NET 6 on ubuntu:latest.

Side note: Is there some reason you're doing this yourself, instead of using one of the pre-built .net images?

Pre-built images did not exist when I started my project circa .NET Core 3 (at least I don't think they existed?).
I'll try switching when I switch my project to .NET 7.

As for this issue, I found several other reports of the .11 .NET build that is included in Ubuntu being broken, and to switch to installing hardcoded .8 builds.

As for me, I just reverted to installing using PMC and not relying on Ubuntu native support.

# Install .NET runtime using PMC
# Ubuntu 22.04 / Jammy includes .NET, set correct order when using packages-microsoft-prod.deb
# https://github.com/dotnet/core/issues/7699
RUN wget https://packages.microsoft.com/config/ubuntu/$(lsb_release -sr)/packages-microsoft-prod.deb \
    && dpkg -i packages-microsoft-prod.deb \
    && rm packages-microsoft-prod.deb \
    && touch /etc/apt/preferences \
    && echo "Package: *" >> /etc/apt/preferences \
    && echo "Pin: origin \"packages.microsoft.com\"" >> /etc/apt/preferences \
    && echo Pin-Priority: 1001 >> /etc/apt/preferences \
    && cat /etc/apt/preferences \
    && cat /etc/apt/sources.list.d/microsoft-prod.list \
    && apt-get update \
    && apt-get install -y dotnet-runtime-6.0 \
    && dotnet --info

@agocke
Copy link
Member

agocke commented Dec 5, 2022

This combo looks suspicious:

#15 0.254 .NET SDKs installed:
#15 0.254 6.0.111 [/usr/lib/dotnet/sdk]

#15 0.262 App: /PlexCleaner/PlexCleaner
#15 0.262 Architecture: x64
#15 0.262 App host version: 6.0.11
#15 0.262 .NET location: Not found

The default dotnet install location on Linux is /usr/share/dotnet, but this is /usr/lib/dotnet.

You can change this by putting a different path in /etc/dotnet/install_location, but I'm not sure if that was created by this installer.

@vitek-karas
Copy link
Member

As far as I can tell this is likely due to wrong installation. The fact that dotnet --info works means that the installation itself is fine and can run (that command runs managed code, so runtime works). But then it fails when running PlexCleaner executable, so apphost. So I would guess that:

  • The dotnet location is on PATH and thus dotnet --info works.
  • The installation (whatever it was) didn't register the runtime in the /etc/dotnet/install_location - this is the primary way how the executable finds the installed runtime
  • There's no DOTNET_ROOT defined in the environment - this is the secondary way to find the runtime

If the installation of the runtime is supposed to be global, then it needs to be registered in /etc/dotnet/install_location. It would also work if it was in the default location which is /usr/share/dotnet, but since it's not there it needs to be registered.

If the installation is supposed to be only for running the app and not global to the machine, then it needs to set DOTNET_ROOT environment variable - since otherwise the executable has no way of finding the runtime.

If you want to be 100% about this, you can set COREHOST_TRACE=1 before running the app, it should print out detailed log of where it looked for the runtime and why.

The key question is - how did you install the SDK/runtime?

@ptr727
Copy link
Author

ptr727 commented Dec 5, 2022

The key question is - how did you install the SDK/runtime?

Using apt-get install -y dotnet-sdk-6.0 on a vanilla Ubuntu image, see

@Clockwork-Muse
Copy link
Contributor

Pre-built images did not exist when I started my project circa .NET Core 3 (at least I don't think they existed?).
I'll try switching when I switch my project to .NET 7.

3.1 images exist, although I'm not sure when they were first created.

@ghost
Copy link

ghost commented Dec 6, 2022

DOTNET_ROOT=/usr/lib/dotnet solved it for me. I was setting DOTNET_ROOT to something else in my bashrc.

@vitek-karas
Copy link
Member

@markhiveit thanks - DOTNET_ROOT will override any other configuration, so if it points to the wrong location, the app will simply not work at all.

@ghost
Copy link

ghost commented Dec 6, 2022

@markhiveit thanks - DOTNET_ROOT will override any other configuration, so if it points to the wrong location, the app will simply not work at all.

The odd thing was that it was working up until very recently with DOTNET_ROOT set in my bashrc - it was pointing to ${HOME}/.config/dotnet. It's possible I changed something else that stopped it working, but then others seem to be bumping into the same issue at the same time. I'm also on Ubuntu 22.04, so wonder if it's something specific to that -- an apt upgrade deleted a symlink or something?

Anyway, sounds like my config was wrong in the first place and needed correcting.

@agocke
Copy link
Member

agocke commented Dec 6, 2022

Confirmed: the default dotnet install location is [/usr/lib/dotnet/shared/Microsoft.NETCore.App] for ubuntu:latest, but it doesn't have the /etc/dotnet/install_location file. So this deb package is broken.

Not sure who owns creating deb packages. @jkoritzinsky would you happen to know?

@Daan4
Copy link

Daan4 commented Dec 8, 2022

I experienced the same issue on ubuntu 22.04 going from 6.0.9 to 6.0.11. Resolved it by doing mv /lib/dotnet/install_location /etc/dotnet/install_location

@agocke
Copy link
Member

agocke commented Dec 8, 2022

@MichaelSimons do you know who the contact would be for the Ubuntu packages?

@MichaelSimons
Copy link
Member

@mirespace, is the primary dev of the source-built .NET packages in the Ubuntu feed. @mirespace, can you help out with this issue? TIA

@mirespace
Copy link
Contributor

Hi all,

Thanks a lot Michael for the mention! We detected that DOTNET_ROOT was not properly updated when upgrading from previous versions to latest 6.0.111 (we did a change in the install layout as some of you could notice), affecting only to amd64 architectures:

https://bugs.launchpad.net/ubuntu/+source/dotnet6/+bug/1997746

We were monitoring any possible issues related to this, but we didn't get any on Launchpad so we couldn't estimate if this was causing a real impact or not. @ptr727 or anyone else affected, whenever you face an issue with the package, feel free to file a issue about dotnet package in ubuntu at https://bugs.launchpad.net/ubuntu/+source/dotnet6/+filebug ... thankyou!. I'll track this to the above bug already opened. Anyway, happy to help here, of course!

The fix is prepared for next version .112 next week, sorry for the inconvenience.

The issue arises in the file /etc/profile.d/dotnet.sh, where the DOTNET_ROOT variable is set:

# Set location for AppHost lookup
[ -z "$DOTNET_ROOT" ] && export DOTNET_ROOT=/usr/lib/dotnet

There are a couple of workarounds in the meantime (I already saw some of you applying it, thanks!):

  • the DOTNET_ROOT can be exported manually
  • eliminate the condition in the above file to force the export of the variable (sourcing the file again).
  • set it in some json files for configuring the solution in the sln folder (*.runtimeconfig.json )
  • call the executable directly prepending the DOTNET_ROOT variable with the correct folder (in this case DOTNET_ROOT=/usr/lib/dotnet, i.e. DOTNET_ROOT=/usr/lib/dotnet MyApp).

We didn't notice the missing location files! Thanks @agocke for the reporting here and @Daan4 for the workaround. I already filed a bug for that:

https://bugs.launchpad.net/ubuntu/+source/dotnet6/+bug/1999266

I'll update this issue when the fix would be released. Thanks in advance for your patience.

@agocke
Copy link
Member

agocke commented Dec 9, 2022

@mirespace Just to give a little more context where this matters:

If you use dotnet directly, you probably won't notice much difference. In these cases the runtime is always next to the dotnet launcher so there's extra to find.

The scenario that's broken is what we call the "apphost" scenario, where a native exe is bundled with the user application. In that case the apphost needs to launch the runtime, but doesn't necessarily know where to look. The behavior is defined as either

  1. the default location in /usr/share
  2. the location as specified in /etc/dotnet/install_location

The fixes for this would generally be to either use the default install location, or place the install_location file in the expected /etc/dotnet location. No real preference on either choice.

darakeon added a commit to darakeon/server that referenced this issue Dec 12, 2022
.NET 6.0.11 calls from built sites is not working without this

dotnet/runtime#79237 (comment)
@vitek-karas
Copy link
Member

The fixes for this would generally be to either use the default install location, or place the install_location file in the expected /etc/dotnet location. No real preference on either choice.

I would say we prefer to always write the install_location file, even if the install is into the default location.

The design docs for this area are here:
https://github.com/dotnet/designs/blob/main/accepted/2020/install-locations.md
https://github.com/dotnet/designs/blob/main/accepted/2021/install-location-per-architecture.md

Ideally the installer would also write the install_location_<arch> file, also pointing to the same path.

@mirespace
Copy link
Contributor

mirespace commented Dec 13, 2022

Hi! To clarify, both fixes will be introduced in the next version:

  • files install_location and install_location_<arch> will be deployed at /etc/dotnet
  • DOTNET_ROOT will be set to /usr/lib/dotnet

@coc0a
Copy link

coc0a commented Dec 14, 2022

FWIW, was having this issue on a freshly spun up arm64 (NOT amd64) VPS instance, and before I found this issue, I tried rebooting the VPS, after which it worked fine. Investigating further, I found /etc/dotnet to be empty but DOTNET_ROOT set to the correct value of /usr/lib/dotnet, no idea if it was set before the reboot as this is my first time using .NET and didn't know about it before hand. No idea what happened after the reboot.

@Clockwork-Muse
Copy link
Contributor

FWIW, was having this issue on a freshly spun up arm64 (NOT amd64) VPS instance, and before I found this issue, I tried rebooting the VPS, after which it worked fine. Investigating further, I found /etc/dotnet to be empty but DOTNET_ROOT set to the correct value of /usr/lib/dotnet, no idea if it was set before the reboot as this is my first time using .NET and didn't know about it before hand. No idea what happened after the reboot.

Environment variables usually require session refresh (the common case being PATH), although there's various mechanisms to manually do so.

@masto
Copy link

masto commented Jan 4, 2023

FYI: I'm using zsh instead of bash, and as a result /etc/profile is not read and DOTNET_ROOT remains unset. Obviously I can work around this in any number of ways now that I know what's happening, but I'm dropping a comment here in case there's a better answer to make it work out of the box for non-bash users.

I don't know what that answer is. TBH I only learned today that zsh doesn't read /etc/profile. It seems like a bummer. On my system, there are 10 scripts in /etc/profile.d and I'm now wondering if any of them contain important things I've been missing.

Realizing now how much hander the generic problem of "we need to modify your shell environment on installation" is than I knew, I won't feel bad if the decision here is "you're on your own if you don't use the default shell". That said, I think zsh and fish are pretty popular these days. I think zsh is even the default on macOS so lots more people are aware of it.

@jkotas
Copy link
Member

jkotas commented Jan 4, 2023

The proper way is for the package installer to update /etc/dotnet/install_location with the path. This step is missing in the recent Ubuntu package installer. It is a bug in the Ubuntu package installer.

Setting DOTNET_ROOT is a quick workaround. It comes with number of issues as you have pointed out.

@mirespace
Copy link
Contributor

Hi all! Happy New year btw.

(OFYI) Sorry for the delay in commenting here: the fixes were released this week with version 6.0.113 (maybe some of you noticed them also in the 112 version that was in proposed pocket in December). I hope this help, and please, let me know if something else needs to be changed for you.

@masto Thanks for bringing up the zsh user experience.... would you mind submitting a bug at https://bugs.launchpad.net/ubuntu/+source/dotnet6/+filebug ? I will mark it as Wishlist at first, but it will help us to be aware of it. We will work on it as soon as possible when time and resources allow.

@elinor-fung
Copy link
Member

Thanks for the update, @mirespace! I'm going to close this issue since the fix has been released.

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Mar 16, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Apr 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests