Cannot update workload manifests from authenticated AzDO feeds on macOS #35912

DamianEdwards · 2023-10-06T20:25:53Z

I'm not able to get any dotnet workload related commands to work when one of the workload manifests has updates on an authenticated AzDO artifacts feed on macOS. I've installed the nuget credential provider and dotnet restore from that feed works just fine.

I've tried relying on on-demand token acquisition (via device flow and passing --interactive), configuring a PAT in the NuGet.config file, passing the source directly, etc. Every time, sudo dotnet workload update fails with the following error message:

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://pkgs.dev.azure.com/<feed-name>/nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.aspire: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.ios: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.maccatalyst: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.macos: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.maui: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.tvos: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.current: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.current: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.net6: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.net7: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net6: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net7: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json.

Output of dotnet --info:

.NET SDK:
 Version:   8.0.100-rc.2.23472.8
 Commit:    e13fd5c19e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.4
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.100-rc.2.23472.8/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-rc.2.23466.4
  Architecture: arm64
  Commit:       287c10d253

.NET SDKs installed:
  6.0.402 [/usr/local/share/dotnet/sdk]
  8.0.100-rc.2.23472.8 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-rtm.23468.24 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-rc.2.23466.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  /Users/damian/src/MyProject/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

The text was updated successfully, but these errors were encountered:

marcpopMSFT · 2023-10-10T18:21:02Z

Chet believes the code in question might be here but it is correctly passing the interactive flag through:

sdk/src/Cli/dotnet/commands/dotnet-workload/WorkloadCommandNuGetRestoreActionConfigOptions.cs

Lines 55 to 58 in 6483a69

    
           return new RestoreActionConfig(DisableParallel: parseResult.GetValue(DisableParallelOption), 
        
               NoCache: parseResult.GetValue(NoCacheOption), 
        
               IgnoreFailedSources: parseResult.GetValue(IgnoreFailedSourcesOption), 
        
               Interactive: parseResult.GetValue(InteractiveRestoreOption));

DamianEdwards · 2023-10-11T00:35:06Z

I didn't have this issue on Ubuntu FWIW, and others have hit this on macOS too, but I appreciate it might be hard to repro without a workload manifest update ready to go on a private feed.

rmarinho · 2023-10-12T13:33:34Z

@DamianEdwards did you tried --configfile pointing to the global Nuget.config with the PAT ?

dsplaisted · 2023-10-12T14:54:39Z

Thanks a bunch to @rmarinho and @mattleibow for helping investigate this.

The problem is caused by running under su/sudo. When running a workload command under sudo, we override the HOME directory to point to a temporary directory. This is so that we don't end up writing files owned by root (for example first run flags or other things stored in the <HOME>\.dotnet folder) which would then not be accessible to the .NET CLI when running under normal permissions.

This means that when run under sudo, the credential provider isn't found by NuGet (since the path to search for it is in the HOME directory). This is also probably why putting the PATs in the NuGet.config file didn't work. (We do try to copy the NuGet.config to the new temporary home directory, but maybe this is failing).

We will discuss to see what we can do to improve this. For now, putting PATs in a NuGet.config file and specifying that file via the --configfile option to workload commands should work as a workaround.

DamianEdwards · 2023-10-12T17:04:00Z

That will explain why it always runs the first-run experience too I guess.

I'm still unable to get it to work with the --configfile option using the nuget.config in the repo, modified to contain a PAT in plaintext. It does report it's trying to access the correct feed though, so it must be reading at least that from the nuget.config, but not the PAT for some reason.

DamianEdwards · 2023-10-12T17:14:53Z

That said, I'd expect an authorization failure to be logged exactly as that, but what I'm seeing is "NotFound" results. I'll share the repo details and logs internally to see if that can aid in diagnosing what's going on.

dsplaisted · 2023-10-12T22:50:43Z

The output is pretty verbose, and it's checking for packages on all the different feeds that are set up. So you get a lot of NotFound log messages before you get the message related to the authorization failure, which looks something like:

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://pkgs.dev.azure.com/<feed-name>/nuget/v3/index.json..

We might be able to detect the authorization error specifically and generate a more specific error message.

It might also be a good idea to generate a warning if --interactive is used under sudo, since we now know that doesn't work.

DamianEdwards · 2023-10-13T00:07:24Z

Any idea why I'm unable to make it work with a PAT and the --configfile option?

dsplaisted · 2023-10-13T13:45:15Z

Any idea why I'm unable to make it work with a PAT and the --configfile option?

Not really. Are you sure the syntax in the config file for the PAT is correct?

DamianEdwards · 2023-10-13T14:10:23Z

Any idea why I'm unable to make it work with a PAT and the --configfile option?

Not really. Are you sure the syntax in the config file for the PAT is correct?

Not sure. I'm just working from documentation. The fact it's trying the right feed suggests it is picking up the nuget.config, and there are no parsing errors. I'm not sure what else to try.

Eilon · 2023-10-17T23:24:50Z

BTW I think I'm hitting the same issue on Windows (@DamianEdwards had it on macOS). I have Azure Cred Provider, but it doesn't seem to get used when I restore workloads. If I pass in --interactive then I get the devicelogin workflow and everything works.

marcpopMSFT · 2023-10-23T19:00:26Z

BTW I think I'm hitting the same issue on Windows (@DamianEdwards had it on macOS). I have Azure Cred Provider, but it doesn't seem to get used when I restore workloads. If I pass in --interactive then I get the devicelogin workflow and everything works.

We are discussing making --interactive default in some situations but the behavior you report is expected. Having the cred provider installed today is not sufficient to trigger the device login details as interactive is needed as well.

baronfel · 2023-10-23T19:07:05Z

@DamianEdwards per this doc the nuget plugin discovery is extensible via environment variables. Can you try:

setting the NUGET_PLUGIN_PATHS variable in your terminal profiles to somewhere that both sudo and non-sudo users have access to, and
copying the contents of ~/.nuget/plugins to that path, and
trying the workload install/update again?

baronfel · 2023-10-23T20:05:54Z

Another thing to try - the azure feed provider puts session tokens in a directory that's configurable via env var, and NuGet will also look in this location. Consider setting the location per these docs to somewhere outside of $HOME and trying that as well.

baronfel · 2023-10-23T20:06:19Z

cc @kartheekp-ms for visibility per Jon's suggestion

kartheekp-ms · 2024-03-18T17:26:22Z

As pointed out in #35912 (comment), the following environment variable might be helpful to work around this issue:

NUGET_NETCORE_PLUGIN_PATHS - Defines the plugins that will be used by the .NET Core-based tooling (dotnet.exe).

I guess we don't have to set NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED to false because the credential provider (plugin) anyway writes to the $HOME/.local/share/MicrosoftCredentialProvider/ location on Mac and Linux. From what I understood in this thread, the $HOME environment variable will be pointing to a temporary location that is created while running the sudo dotnet workload install command.

In case it helps, this document contains environment variables that can be set to manage NuGet global packages and cache folders.

gtbuchanan · 2024-07-25T19:01:44Z

~~Would this issue prevent the latest version of workloads from being installed?~~

I started encountering a MAUI iOS build issue that only seems to occur on Azure Pipelines. After investigating, I found that we were encountering this authentication error at the dotnet workload install maui step and the workload being installed on the agent was only v8.0.3 (whereas the latest on my Mac is v8.0.61).

We don't actually load workloads from authenticated sources, so I'll try passing a NuGet.Config without our internal source to --configfile and see what happens.

EDIT:

Yep, that was the problem and the solution worked. Here's the PowerShell step I used in my pipeline:

$NuGetConfig = Join-Path $env:AGENT_TEMPDIRECTORY "NuGet.Config"
Write-Host "Writing temporary NuGet.Config to $NuGetConfig..."
New-Item $NuGetConfig
Set-Content $NuGetConfig '<configuration><packageSources><add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" /></packageSources></configuration>'
dotnet workload install maui --configfile $NuGetConfig
dotnet workload list

After applying the above workaround, the newest version of the workload was successfully installed.

guyulmaz · 2024-08-14T21:39:01Z

Hi, I have same problem, cannot install workload to my macbook for sudo needed ones. Getting error=

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://api.nuget.org/v3/index.json..

I tried adding a nuget.config myself and use --configfile, but it did not work
how can I install android , wasm workloads to my mac ?
I'm trying to setup avalonia cross platform vscode template at my macbook, that will cover all desktop, web and mobile
Can please help

% dotnet --info
.NET SDK:
Version: 8.0.400
Commit: 36fe6dda56
Workload version: 8.0.400-manifests.56cd0383
MSBuild version: 17.11.3+0c8610977

Runtime Environment:
OS Name: Mac OS X
OS Version: 14.6
OS Platform: Darwin
RID: osx-arm64
Base Path: /usr/local/share/dotnet/sdk/8.0.400/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
[ios]
Installation Source: SDK 8.0.400
Manifest Version: 17.5.8020/8.0.100
Manifest Path: /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.ios/17.5.8020/WorkloadManifest.json
Install Type: FileBased

Host:
Version: 8.0.8
Architecture: arm64
Commit: 08338fcaa5

.NET SDKs installed:
8.0.300 [/usr/local/share/dotnet/sdk]
8.0.303 [/usr/local/share/dotnet/sdk]
8.0.400 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
None

Environment variables:
DOTNET_ROOT [/usr/local/share/dotnet]

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

petrkoutnycz · 2024-09-13T07:49:56Z

Tried the hack with setting NUGET_NETCORE_PLUGIN_PATHS env variable but with no avail :-/

rmarinho · 2024-10-02T18:01:54Z

Yeah this seems to only work for me if using the --configFile, sudo commands will not work without that .

Eliza-Huang · 2024-10-05T00:19:23Z

Even when I add the configfile option, it still wants me to run in sudo which doesn't work based on previous comments

dotnet-issue-labeler bot added Area-Workloads untriaged Request triage from a team member labels Oct 6, 2023

MiYanni mentioned this issue Oct 12, 2023

[NETE2ESDK] With .NET 8 RC2 SDK pkg/deb/rpm installed, "An issue was encountered verifying workloads. For more information, run "dotnet workload update"." displayed when running the first dotnet command. #35128

Open

DamianEdwards mentioned this issue Oct 13, 2023

[WebToolsE2E][Aspire][MacOS]Install aspire template failed with error "Workload ID aspire is not recognized." dotnet/aspire#281

Closed

marcpopMSFT mentioned this issue Jan 17, 2024

Workloads distribution and management is understandable and seamless for developers #30008

Open

17 tasks

nkolev92 mentioned this issue Apr 2, 2024

Support for NuGet authentication plugins deployed via .NET tools NuGet/Home#13242

Merged

embetten mentioned this issue Jul 18, 2024

nuget not prompting for credentials when querying Azure Artifacts feed on WSL microsoft/artifacts-credprovider#501

Open

baronfel added the Partner request requests from partners label Nov 1, 2024

baronfel self-assigned this Nov 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot update workload manifests from authenticated AzDO feeds on macOS #35912

Cannot update workload manifests from authenticated AzDO feeds on macOS #35912

DamianEdwards commented Oct 6, 2023

marcpopMSFT commented Oct 10, 2023

DamianEdwards commented Oct 11, 2023

rmarinho commented Oct 12, 2023 •

edited

Loading

dsplaisted commented Oct 12, 2023

DamianEdwards commented Oct 12, 2023

DamianEdwards commented Oct 12, 2023

dsplaisted commented Oct 12, 2023

DamianEdwards commented Oct 13, 2023

dsplaisted commented Oct 13, 2023

DamianEdwards commented Oct 13, 2023

Eilon commented Oct 17, 2023

marcpopMSFT commented Oct 23, 2023

baronfel commented Oct 23, 2023

baronfel commented Oct 23, 2023

baronfel commented Oct 23, 2023

kartheekp-ms commented Mar 18, 2024 •

edited

Loading

gtbuchanan commented Jul 25, 2024 •

edited

Loading

guyulmaz commented Aug 14, 2024

petrkoutnycz commented Sep 13, 2024

rmarinho commented Oct 2, 2024

Eliza-Huang commented Oct 5, 2024

Cannot update workload manifests from authenticated AzDO feeds on macOS #35912

Cannot update workload manifests from authenticated AzDO feeds on macOS #35912

Comments

DamianEdwards commented Oct 6, 2023

marcpopMSFT commented Oct 10, 2023

DamianEdwards commented Oct 11, 2023

rmarinho commented Oct 12, 2023 • edited Loading

dsplaisted commented Oct 12, 2023

DamianEdwards commented Oct 12, 2023

DamianEdwards commented Oct 12, 2023

dsplaisted commented Oct 12, 2023

DamianEdwards commented Oct 13, 2023

dsplaisted commented Oct 13, 2023

DamianEdwards commented Oct 13, 2023

Eilon commented Oct 17, 2023

marcpopMSFT commented Oct 23, 2023

baronfel commented Oct 23, 2023

baronfel commented Oct 23, 2023

baronfel commented Oct 23, 2023

kartheekp-ms commented Mar 18, 2024 • edited Loading

gtbuchanan commented Jul 25, 2024 • edited Loading

guyulmaz commented Aug 14, 2024

petrkoutnycz commented Sep 13, 2024

rmarinho commented Oct 2, 2024

Eliza-Huang commented Oct 5, 2024

rmarinho commented Oct 12, 2023 •

edited

Loading

kartheekp-ms commented Mar 18, 2024 •

edited

Loading

gtbuchanan commented Jul 25, 2024 •

edited

Loading