Microsoft and source-built Linux packages don't mix well

[NikolaMilosavljevic]

Source-built .NET is in Fedora and CentOS. There are numerous reports of people hitting issues when some packages on their machines come from the Microsoft repos and others from the distro archive. These packages don't work well together, i.e. they don't mix well. They have different dotnet roots, /usr/share/dotnet vs. /usr/lib64/dotnet. Here's one recent report: dotnet/sdk#15476

Additionally, packages from different distro repositories, i.e. Fedora and RHEL could cause similar issues. Packages are never meant to be mixed, only a single repository should ever be used.

We need to find a way to help customers avoid this issue. Better documentation could help, but is not the preferred or only solution. Other packages and application vendors could have found themselves in the same situation. We need to find ways to leverage distro and community knowledge to come up with a proper mitigation.

[hoyosjs]

cc: @omajid @tmds

[wfurt]

It seems like DOTNET_MULTILEVEL_LOOKUP=0 as meant for that e.g. restrict dotnet to be "smart" and look outside of given location.

[tmds]

I ran into this just now myself.

I needed to run a powershell script so I installed powershell using instructions from: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7.1#fedora.

Now I get a mix of packages, and installation fails:

 sudo dnf install dotnet-sdk-3.1
Last metadata expiration check: 3:48:52 ago on Wed 27 Jan 2021 05:44:11 AM CET.
Dependencies resolved.
==============================================================================================================================================================
 Package                                         Architecture             Version                         Repository                                     Size
==============================================================================================================================================================
Installing:
 dotnet-sdk-3.1                                  x86_64                   3.1.405-1                       packages-microsoft-com-prod                    71 M
Installing dependencies:
 aspnetcore-runtime-3.1                          x86_64                   3.1.11-1.fc32                   updates                                       6.2 M
 aspnetcore-targeting-pack-3.1                   x86_64                   3.1.11-1.fc32                   updates                                       1.0 M
 dotnet-apphost-pack-3.1                         x86_64                   3.1.11-1.fc32                   updates                                        70 k
 dotnet-hostfxr-3.1                              x86_64                   3.1.11-1.fc32                   updates                                       164 k
 dotnet-runtime-3.1                              x86_64                   3.1.11-1.fc32                   updates                                        27 M
 dotnet-targeting-pack-3.1                       x86_64                   3.1.11-1.fc32                   updates                                       1.8 M

Transaction Summary
==============================================================================================================================================================
Install  7 Packages
...
Failed:
  dotnet-sdk-3.1-3.1.405-1.x86_64         
...

Can we move this issue to source-build repo?

cc @dagood

[omajid]

Yum repository files have a priority option that lets dnf/yum modify how important a repository is treated. We can adjust it to make dnf/yum prefer packages from other repositories.

The default repo priority is 99, and if a repo increases the value of the priority, it is less prioritized for fetching packages. (Yes, this seems a bit counter-intuitive that smaller priority numbers actually mean a higher priority).

Here's a test on Fedora 33:

$ sudo dnf remove '*dotnet*'                                                                                                            
No match for argument: *dotnet*                                                                                                                               
No packages marked for removal.                                                                                                                               
Dependencies resolved.
Nothing to do.                                                                                                                                                
Complete!               
$ rpm -qa | grep dotnet

$ wget https://packages.microsoft.com/config/fedora/33/prod.repo 
$ sudo cp prod.repo /etc/yum.repos.d/microsoft.repo
$ sudo dnf install dotnet-sdk-3.1                                     
packages-microsoft-com-prod                                                                                                   279 kB/s | 182 kB     00:00    
Dependencies resolved.                                                                                                                                        
========================================================================================================
 Package                          Architecture   Version           Repository                      Size
========================================================================================================
Installing:                                                                                             
 dotnet-sdk-3.1                   x86_64         3.1.405-1         packages-microsoft-com-prod     71 M
Installing dependencies:                                                                                
 aspnetcore-runtime-3.1           x86_64         3.1.11-1.fc33     updates                        6.2 M
 aspnetcore-targeting-pack-3.1    x86_64         3.1.11-1.fc33     updates                        1.0 M
 dotnet-apphost-pack-3.1          x86_64         3.1.11-1.fc33     updates                         71 k
 dotnet-host                      x86_64         5.0.2-1           packages-microsoft-com-prod     64 k
 dotnet-hostfxr-3.1               x86_64         3.1.11-1.fc33     updates                        166 k
 dotnet-runtime-3.1               x86_64         3.1.11-1.fc33     updates                         27 M
 dotnet-targeting-pack-3.1        x86_64         3.1.11-1.fc33     updates                        1.8 M
 netstandard-targeting-pack-2.1   x86_64         3.1.111-1.fc33    updates                        1.3 M
                                                                                                        
Transaction Summary                                                                                     
========================================================================================================
Install  9 Packages                                                                                                                                           

Total download size: 108 M
Installed size: 344 M                                                                                                                                         
Is this ok [y/N]: n 
Operation aborted.

# Reduce the priority of the Microsoft repository
$ sudo dnf config-manager --save --setopt="*microsoft-*.priority=100"

$ sudo dnf install dotnet-sdk-3.1                         
Dependencies resolved.
==========================================================================================
 Package                          Architecture   Version            Repository       Size
==========================================================================================
Installing:
 dotnet-sdk-3.1                   x86_64         3.1.111-1.fc33     updates          40 M
Installing dependencies:
 aspnetcore-runtime-3.1           x86_64         3.1.11-1.fc33      updates         6.2 M
 aspnetcore-targeting-pack-3.1    x86_64         3.1.11-1.fc33      updates         1.0 M
 dotnet-apphost-pack-3.1          x86_64         3.1.11-1.fc33      updates          71 k
 dotnet-host                      x86_64         3.1.11-1.fc33      updates         105 k
 dotnet-hostfxr-3.1               x86_64         3.1.11-1.fc33      updates         166 k
 dotnet-runtime-3.1               x86_64         3.1.11-1.fc33      updates          27 M
 dotnet-targeting-pack-3.1        x86_64         3.1.11-1.fc33      updates         1.8 M
 dotnet-templates-3.1             x86_64         3.1.111-1.fc33     updates         1.8 M
 netstandard-targeting-pack-2.1   x86_64         3.1.111-1.fc33     updates         1.3 M

Transaction Summary
==========================================================================================
Install  10 Packages

Total download size: 80 M
Installed size: 300 M
Is this ok [y/N]: n
Operation aborted.

If we can insert this option directly in the repo file, that might be good enough. We can do this by adding this line directly in the repo file: priority=100. Otherwise, we can ask users to run sudo dnf config-manager --save --setopt="*microsoft-*.priority=100". The command is slightly different with yum and I haven't tested it.

[omajid]

The proposed priority=100 option wont help in the case where the user wants to install dotnet-sdk-5.0 (not available in Fedora repos yet) with dotnet-sdk-3.1 (available in Fedora repos). That would still break with repo prioritization because dotnet-sdk-5.0 will be fetched from the MIcrosoft repo and placed at /usr/share/dotnet, while the host will probably try and use /usr/lib64/dotnet instead.

[dagood]

Oh wow, when I saw priority mentioned in one of these issues earlier (I've been collecting the ones I see in dotnet/docs#18347), based on the name, I thought it would only affect what happens when the same package with the same version is available in two repos. I'm surprised that it actually prevents upgrade even when a newer version is available on the less prioritized feed. It certainly makes sense that it works this way though, or it would be a pretty weak feature. 😛

Yeah, to give a little extra context why it doesn't help with 5.0: the mix of packages with major.minor in the name (dotnet-sdk-5.0) and no version in the name (dotnet-host) means that if you try to get dotnet-sdk-5.0 from a fresh setup, you'll end up with Microsoft's dotnet-sdk-5.0 and Fedora's dotnet-host, and dotnet commands will still fail: you end up with a partial install of 3.1 from Fedora and partial install of 5.0 from Microsoft.

(On the other side, if you want to get .NET from packages.microsoft.com, it seems that prioritizing packages.microsoft.com would work, because it's always a superset as far as .NET packages are concerned.)

If we could have fake temporary dotnet-*-5.0 packages in Fedora that actually just print a warning or something, that seems like it would at least make sure every single package is coming from the one repo with higher priority, preventing the current hostfxr errors that don't really point people at a solution. But I have no idea if that's a reasonable thing to do. Thoughts?

---

I needed to run a powershell script so I installed powershell using instructions from: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7.1#fedora.

This means to me that we have to have packages.microsoft.com themselves actually put the deprioritization in the central repo config files, because the powershell instructions (and more?) just download and install that packages.microsoft.com repo file.

Can we move this issue to source-build repo?

It certainly affects the use of distros with source-built SDKs. (/cc @dseefeld @crummel.) But I'm not sure what we can do about this on the source-build side--dotnet/runtime area-Setup is about the Microsoft packaging side of things, so I think it's correct to be here.

[omajid]

If we could have fake temporary dotnet-*-5.0 packages in Fedora that actually just print a warning or something, that seems like it would at least make sure every single package is coming from the one repo with higher priority, preventing the current hostfxr errors that don't really point people at a solution. But I have no idea if that's a reasonable thing to do. Thoughts?

For .NET 5.0, at least, I think it would be faster to just review and build the real packages. They are ready from my side and I am waiting for a formal review: https://bugzilla.redhat.com/show_bug.cgi?id=1919037

[dagood]

It looks like this problem also shows up for feature bands that Fedora doesn't build--I suppose priority doesn't block an install that includes the exact version. Installing dotnet-sdk-3.1 or dotnet-sdk-3.1-3.1.111 pulls from Fedora, but dotnet-sdk-3.1.405 still pulls from Microsoft.

# cat /etc/yum.repos.d/microsoft-prod.repo

[packages-microsoft-com-prod]
name=packages-microsoft-com-prod
baseurl=https://packages.microsoft.com/fedora/33/prod/
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc
priority=100

# dnf install dotnet-sdk-3.1-3.1.405
...
==============================================================================================
 Package                         Arch    Version           Repository                    Size
==============================================================================================
Installing:
 dotnet-sdk-3.1                  x86_64  3.1.405-1         packages-microsoft-com-prod   71 M
Installing dependencies:
 aspnetcore-runtime-3.1          x86_64  3.1.11-1.fc33     updates                      6.2 M
 aspnetcore-targeting-pack-3.1   x86_64  3.1.11-1.fc33     updates                      1.0 M
 dotnet-apphost-pack-3.1         x86_64  3.1.11-1.fc33     updates                       71 k
 dotnet-host                     x86_64  3.1.11-1.fc33     updates                      105 k
...

(The result is that dotnet finds no SDKs because they're off in some unknown location.)

[tmds]

The proposed solution is to lower the priority on the repo files from Microsoft in order to give preference to the dotnet packages that come with the distro.

With this known issue: in case a .NET version is not yet available from distro packages, pulling that version from the Microsoft repo will cause some/all versions not to work.

Can the priority can be added into the repo file at https://packages.microsoft.com/config/rhel/7/prod.repo?
Are there expected issues from including it?

[dagood]

Can the priority can be added into the repo file at https://packages.microsoft.com/config/rhel/7/prod.repo?

It certainly helps the install experience. My understanding is this is waiting for review from the area-Setup folks before asking the packages.microsoft.com team to do these changes.

A few more things to think about:

• How does this affect the upgrade experience for those who already have .NET installed from packages.microsoft.com with the current priority?
  • Lowest release band, 3.1.111, available in the distro?
  • Higher release band, 3.1.405, not available in the distro?
• What if we prioritize packages.microsoft.com instead?
  • I hope we don't have to, but it seems worth evaluating even if just to document how it leaves everyone worse off.
• Does all of this behave the same for Debian-based distros? (Future: Alpine?)

---

More generally, I want to point out that nothing proposed so far seems to fix all problems--we still need documentation to help people fix up their installs and get the bits installed that they actually want. I think holding off on dotnet/docs#18347 is not good.

[NikolaMilosavljevic]

[Triage] We need to break this into specific scenarios and prioritize them. Please consider Previews and future distro support.

[dleeapho]

/cc @rconard

[MichaelSimons]

Does all of this behave the same for Debian-based distros?

FWIW, I found this answer on askubuntu to be a helpful read on this topic. The short answer is similar mechanism exist to control which package source is used/prioritized but may have the same potential issue of not having a single solution for all scenarios. Furthermore these settings are at the machine level, not the feed level.