NuGet.exe restore does not use basic auth credentials from config #456

Closed
roryprimrose opened this Issue Apr 19, 2015 · 53 comments
@roryprimrose

I'm trying to set up a private NuGet feed that uses an API key for pushing packages and a simple basic auth IHttpModule to protect reads. The system fails on the build server when the build attempts to do a NuGet restore from the private feed.

The nuget.exe is invoked with the following powershell:

$ScriptDir = Split-Path -parent $MyInvocation.MyCommand.Path

Write-Host "Restoring NuGet packages"

& "$ScriptDir\nuget.exe" restore "$ScriptDir\..\MySolution.sln" -PackagesDirectory "$ScriptDir\..\Packages" -ConfigFile "$ScriptDir\NuGet.config" -NonInteractive -Verbosity detailed

The referenced config file contains the following (with sensitive info removed):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageRestore>
        <add key="enabled" value="True" />
        <add key="automatic" value="True" />
    </packageRestore>
  <activePackageSource>
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
  <packageSources>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
    <add key="Custom" value="https://nuget.myserver.com/nuget/" />
  </packageSources>
  <disabledPackageSources />
  <packageSourceCredentials>
    <Custom>
      <add key="Username" value="[UserName]" />
      <add key="ClearTextPassword" value="[Password]" />
    </Custom>
  </packageSourceCredentials>
</configuration>

The powershell script logs the following for the restore:

Restoring NuGet packages
GET https://www.nuget.org/api/v2/Packages(Id='MySystem.Core',Version='1.0.1')
GET https://www.nuget.org/api/v2/Packages(Id='OtherSystem.Client',Version='2.0.2')
GET https://www.nuget.org/api/v2/Packages(Id='MySystem.Core',Version='1.0.1.0')
GET https://www.nuget.org/api/v2/Packages(Id='OtherSystem.Client',Version='2.0.2.0')
Using credentials from config. UserName: [Username]
GET https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2')
GET https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1')
Installing 'MySystem.Core 1.0.1'.
Installing 'OtherSystem.Client 2.0.2'.
GET https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1
GET https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2
Please provide credentials for: https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2
UserName: Please provide credentials for: https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1
nuget.exe : System.InvalidOperationException: Cannot prompt for input in non-interactive mode.
At C:\Temp\MySystem\Tools\RestorePackages.ps1:5 char:1
+ & "$ScriptDir\nuget.exe" restore "$ScriptDir\..\MySolution.sln" -PackagesDi ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (System.InvalidO...teractive mode.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

   at NuGet.Common.Console.EnsureInteractive()
   at NuGet.Common.Console.ReadLine()
   at NuGet.ConsoleCredentialProvider.GetCredentials(Uri uri, IWebProxy proxy, Creden
tialType credentialType, Boolean retrying)
   at NuGet.SettingsCredentialProvider.GetCredentials(Uri uri, IWebProxy proxy, CredentialType credentialType, Boolean retrying)
   at NuGet.RequestHelper.SetCredentialsOnAuthorizationError(HttpWebRequest reques
t)
   at NuGet.RequestHelper.ConfigureRequest(HttpWebRequest request)
   at NuGet.RequestHelper.GetResponse()
   at NuGet.HttpClient.GetResponse()
   at NuGet.HttpClient.DownloadData(Stream targetStream)
   at NuGet.PackageDownloader.DownloadPackage(I
HttpClient downloadClient, IPackageName package, Stream targetStream)
   at NuGet.PackageDownloader.DownloadPackage(Uri uri, IPackageMetadata package, Stream targetStream)
   at NuGet.DataServicePackage.<EnsurePackage>b__0(Stream stream)
   at NuGet.Mac
hineCache.<>c__DisplayClass32.<InvokeOnPackage>b__31()
   at NuGet.MachineCache.TryAct(Func`1 action, String path)
   at NuGet.MachineCache.InvokeOnPackage(String packageId, SemanticVersion version, Action`1 action)
   at NuGet.DataServicePackage.Ensure
Package(IPackageCacheRepository cacheRepository)
   at NuGet.DataServicePackage.GetFiles()
   at NuGet.PackageManager.ExpandFiles(IPackage package)
   at NuGet.PackageManager.OnExpandFiles(PackageOperationEventArgs e)
   at NuGet.PackageManager.Execute
Install(IPackage package)
   at NuGet.PackageManager.Execute(PackageOperation operation)
   at NuGet.PackageManager.Execute(IPackage package, IPackageOperationResolver resolver)
   at NuGet.PackageManager.InstallPackage(IPackage package, FrameworkName t
argetFramework, Boolean ignoreDependencies, Boolean allowPrereleaseVersions, Boolean ignoreWalkInfo)
   at NuGet.PackageManager.InstallPackage(IPackage package, Boolean ignoreDependencies, Boolean allowPrereleaseVersions, Boolean ignoreWalkInfo)
   at Nu
Get.Common.PackageExtractor.<>c__DisplayClass1.<InstallPackage>b__0()
   at NuGet.Common.PackageExtractor.ExecuteLocked(String name, Action action)
   at NuGet.Common.PackageExtractor.InstallPackage(IPackageManager packageManager, IPackage package)
   a
t NuGet.Commands.RestoreCommand.RestorePackage(IFileSystem packagesFolderFileSystem, String packageId, SemanticVersion version, Boolean packageRestoreConsent, 
ConcurrentQueue`1 satellitePackages)
   at NuGet.Commands.RestoreCommand.<>c__DisplayClass5.<>c_
_DisplayClass7.<ExecuteInParallel>b__2()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
UserName: 

I put some rudimentary logging into the HttpModule and it definitely gets the credentials from config to correctly read the package feed and identify the packages. However it looks like it doesn't use these credentials for actually downloading the package.

I even added in cookie support for caching the basic auth credential in the hope that nuget.exe would use the same CookieContainer but it doesn't.

This is the log from the module:

4/19/2015 7:19:07 AM +00:00: https://nuget.myserver.com/nuget/ - Request for https://nuget.myserver.com/nuget/ - GET
4/19/2015 7:19:07 AM +00:00: https://nuget.myserver.com/nuget/ - No authorization header found
4/19/2015 7:19:07 AM +00:00: https://nuget.myserver.com/nuget/ - Request denied for /nuget/
4/19/2015 7:19:07 AM +00:00: https://nuget.myserver.com/nuget/ - Request for https://nuget.myserver.com/nuget/ - GET
4/19/2015 7:19:08 AM +00:00: https://nuget.myserver.com/nuget/ - No authorization header found
4/19/2015 7:19:08 AM +00:00: https://nuget.myserver.com/nuget/ - Request denied for /nuget/
4/19/2015 7:19:09 AM +00:00: https://nuget.myserver.com/nuget/ - Request for https://nuget.myserver.com/nuget/ - GET
4/19/2015 7:19:10 AM +00:00: https://nuget.myserver.com/nuget/ - Password valid - setting the principal
4/19/2015 7:19:10 AM +00:00: https://nuget.myserver.com/nuget/ - Auth cookie set
4/19/2015 7:19:11 AM +00:00: https://nuget.myserver.com/nuget/ - User authenticated, request is allowed
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Request for https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - GET
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - Request for https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - GET
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - No authorization header found
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - Request denied for /nuget/
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - No authorization header found
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - Request for https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - GET
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - Password valid - setting the principal
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - Auth cookie set
4/19/2015 7:19:13 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='MySystem.Core',Version='1.0.1') - User authenticated, request is allowed
4/19/2015 7:19:14 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Request denied for /nuget/
4/19/2015 7:19:14 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Request for https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - GET
4/19/2015 7:19:14 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Password valid - setting the principal
4/19/2015 7:19:14 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Auth cookie set
4/19/2015 7:19:14 AM +00:00: https://nuget.myserver.com/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - User authenticated, request is allowed
4/19/2015 7:19:18 AM +00:00: https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2 - Request for https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2 - GET
4/19/2015 7:19:18 AM +00:00: https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2 - No authorization header found
4/19/2015 7:19:18 AM +00:00: https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2 - Request denied for /api/
4/19/2015 7:19:19 AM +00:00: https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1 - Request for https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1 - GET
4/19/2015 7:19:19 AM +00:00: https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1 - No authorization header found
4/19/2015 7:19:19 AM +00:00: https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1 - Request denied for /api/

You can see here that for each set of requests, auth fails and then the auth credentials are read from config and provided, the cookie is set and the repeated requests is successful. This continues all the way until nuget.exe attempts to download the packages. At this point the auth credentials from config are not provided which results in a hard fail with no repeat with credentials.

@deepakaravindr deepakaravindr added this to the 3.0.0-RTM milestone Apr 20, 2015
@bhuvak
bhuvak commented Apr 23, 2015

@roryprimrose, do you have the credentials set in %AppData%NuGet config file ? Looks like that's the one being picked up while restoring during build

@roryprimrose

No, I point nuget.exe to a config in the same folder. It is here that the credentials get picked up as seen in the log line "Using credentials from config. UserName: [Username]".

I also tried ditching my config file and using powershell to add the credential via the command line before executing the restore. Exactly the same thing happened though.

@yishaigalatzer

@roryprimrose are you able to use the credentials from %appdata% Nuget.config as a workaround for now?

@roryprimrose

See previous two comments. This question has already been addressed.

@yishaigalatzer

Not really. The question was around if you can use the above suggestion as a workaround? If not we can look for another workaround until this gets fixed and a nuget.exe released

@roryprimrose

I think I have already covered that and found the same result. I tried using the command line to add the credentials as part of the build. This would put the credentials into %appdata% nuget.config. The creds were used to query the packages but not used to download them.

@yishaigalatzer

Can you verify the content of your global nuget.config. This is rather odd since this is a mainline scenario and I'm very surprised by it. We can obviously take a look first thing in the morning.

@roryprimrose

Not really. This is running in a VSO build so it would depend on that config. As you can see in the powershell though, I am getting nuget.exe to point to my specific config.

If it is any help in proving this yourself. I have a web project that has the nuget server package installed and then the following module (supports basic auth with a single config user):

namespace MyServer.NuGet
{
    using System;
    using System.Diagnostics;
    using System.IO;
    using System.Net.Http.Headers;
    using System.Security.Principal;
    using System.Text;
    using System.Threading;
    using System.Web;
    using System.Web.Configuration;
    using System.Web.Security;

    public class BasicAuthenticationModule : IHttpModule
    {
        private const string AuthCookieKey = "2327385720594389A013024C00F09EB6";

        private const string Realm = "NuGet";

        private static readonly object _syncLock = new object();

        public void Dispose()
        {
        }

        public void Init(HttpApplication context)
        {
            // Register event handlers
            context.AuthorizeRequest += OnAuthorizeRequest;
            context.AuthenticateRequest += OnApplicationAuthenticateRequest;
            context.EndRequest += OnApplicationEndRequest;
        }

        private static bool AuthenticateUser(HttpContext context, string credentials)
        {
            try
            {
                var encoding = Encoding.GetEncoding("iso-8859-1");
                credentials = encoding.GetString(Convert.FromBase64String(credentials));

                var separator = credentials.IndexOf(':');
                var name = credentials.Substring(0, separator);
                var password = credentials.Substring(separator + 1);

                if (CheckPassword(name, password))
                {
                    LogMessage("Password valid - setting the principal");

                    SetPrincipal(context, name);

                    return true;
                }
            }
            catch (FormatException)
            {
                // Credentials were not formatted correctly.
            }

            return false;
        }

        private static bool CheckPassword(string username, string password)
        {
            var expectedUserName = WebConfigurationManager.AppSettings["ReaderUserName"];
            var expectedPassword = WebConfigurationManager.AppSettings["ReaderPassword"];

            return username == expectedUserName && password == expectedPassword;
        }

        private static GenericPrincipal CreatePrincipal(string name)
        {
            var identity = new GenericIdentity(name);

            return new GenericPrincipal(identity, null);
        }

        private static IPrincipal GetCookieIdentity(HttpRequest request)
        {
            var authCookie = request.Cookies[AuthCookieKey];

            if (authCookie == null)
            {
                return null;
            }

            var ticket = FormsAuthentication.Decrypt(authCookie.Value);

            if (ticket == null)
            {
                return null;
            }

            if (ticket.Expired)
            {
                LogMessage("Auth cookie expired");

                request.Cookies.Remove(AuthCookieKey);

                return null;
            }

            LogMessage("Auth cookie found");

            var principal = CreatePrincipal(ticket.Name);

            return principal;
        }

        [Conditional("DEBUG")]
        private static void LogMessage(string message)
        {
            var context = HttpContext.Current;
            var path = context.Server.MapPath("~/App_Data") + @"\messages.log";

            lock (_syncLock)
            {
                File.AppendAllText(
                    path, 
                    DateTimeOffset.UtcNow + ": " + context.Request.Url + " - " + message + Environment.NewLine);
            }
        }

        private static void OnApplicationAuthenticateRequest(object sender, EventArgs e)
        {
            var application = (HttpApplication)sender;
            var context = application.Context;
            var request = context.Request;

            LogMessage("Request for " + request.Url + " - " + request.HttpMethod);

            Debug.WriteLine(request.Url + " - " + request.HttpMethod);

            // Check if there is an auth cookie
            var cookieIdentity = GetCookieIdentity(request);

            if (cookieIdentity != null)
            {
                SetPrincipal(cookieIdentity);

                return;
            }

            var authHeader = request.Headers["Authorization"];

            if (authHeader == null)
            {
                LogMessage("No authorization header found");

                return;
            }


            var authHeaderVal = AuthenticationHeaderValue.Parse(authHeader);

            // RFC 2617 sec 1.2, "scheme" name is case-insensitive
            if (authHeaderVal.Scheme.Equals("basic", StringComparison.OrdinalIgnoreCase) &&
                authHeaderVal.Parameter != null)
            {
                var authenticated = AuthenticateUser(context, authHeaderVal.Parameter);

                if (authenticated == false)
                {
                    LogMessage("Authentication failed");

                    context.Response.StatusCode = 401;
                    application.CompleteRequest();
                }
            }
        }

        private static void OnApplicationEndRequest(object sender, EventArgs e)
        {
            // If the request was unauthorized, add the WWW-Authenticate header 
            // to the response.
            var response = HttpContext.Current.Response;
            if (response.StatusCode == 401)
            {
                response.Headers.Add("WWW-Authenticate", string.Format("Basic realm=\"{0}\"", Realm));
            }
        }

        private static void OnAuthorizeRequest(object sender, EventArgs eventArgs)
        {
            var application = (HttpApplication)sender;
            var context = application.Context;
            var requestUri = context.Request.Url.ToString();

            if (Thread.CurrentPrincipal.Identity.IsAuthenticated)
            {
                LogMessage("User authenticated, request is allowed");

                return;
            }

            if (requestUri.IndexOf("/api/", StringComparison.OrdinalIgnoreCase) > -1)
            {
                // If this is a post, allow it as it should have an API key
                if (context.Request.HttpMethod == "PUT")
                {
                    return;
                }

                LogMessage("Request denied for /api/");

                context.Response.StatusCode = 401;
                application.CompleteRequest();
            }
            else if (requestUri.IndexOf("/nuget/", StringComparison.OrdinalIgnoreCase) > -1)
            {
                LogMessage("Request denied for /nuget/");

                context.Response.StatusCode = 401;
                application.CompleteRequest();
            }
            else
            {
                LogMessage("Request allowed by default");
            }
        }

        private static void SetPrincipal(HttpContext context, string name)
        {
            var principal = CreatePrincipal(name);

            const int AuthExpiryInDays = 30;
            var ticket = new FormsAuthenticationTicket(
                name, 
                true, 
                (int)TimeSpan.FromDays(AuthExpiryInDays).TotalMinutes);
            var encyptedTicket = FormsAuthentication.Encrypt(ticket);
            var cookie = new HttpCookie(AuthCookieKey, encyptedTicket)
            {
                Secure = true, 
                HttpOnly = true, 
                Expires = DateTime.UtcNow.AddDays(AuthExpiryInDays)
            };

            LogMessage("Auth cookie set");
            context.Response.Cookies.Add(cookie);

            SetPrincipal(principal);
        }

        private static void SetPrincipal(IPrincipal principal)
        {
            Thread.CurrentPrincipal = principal;
            if (HttpContext.Current != null)
            {
                HttpContext.Current.User = principal;
            }
        }
    }
}
@yishaigalatzer

Thanks, I was missing the part where you run in vso. That makes sense now.

@bhuvak
bhuvak commented May 3, 2015

@roryprimrose , I did a setup locally with a nuget.server which has the basic auth module that you have shared and the issue doesn't repro with nuget.exe 2.8.6. However, when proper permissions are not set on the "packages" folder ( for the nuget.server website user account), I hit this issue. In your case does trying to download the package via browser [https://nuget.myserver.com/api/v2/package/mysystem.core/1.0.1] work ?

@roryprimrose

Ah, interesting. I have pointed the packagesPath config to ~/App_Data/Packages to ensure that the packages are not publicly exposed.

I am surprised that this is a security issue because I expected that the server code would have read access to the packages folder on behalf of the user. What permissions are required outside of any IIS/ASP.Net defaults?

I'll so some more testing around this when I get a little more time.

@roryprimrose

Just tested this locally using IISExpress. I authenticated via the browser and the auth cookie was set (configured to use ~/App_Data/Packages as well).

I then listed packages and directly downloaded one of the packages using the browser. I got the package and my trace logs also indicated what happened.

3/05/2015 10:12:05 AM +00:00: https://localhost:44390/api/v2/package/othersystem.client/2.0.1 - Request for https://localhost:44390/api/v2/package/othersystem.client/2.0.1 - GET
3/05/2015 10:12:05 AM +00:00: https://localhost:44390/api/v2/package/othersystem.client/2.0.1 - Auth cookie found
3/05/2015 10:12:05 AM +00:00: https://localhost:44390/api/v2/package/othersystem.client/2.0.1 - User authenticated, request is allowed

I then created a restore scenario that used the same setup in my build process against the same IISExpress instance. This worked sucessfully.

3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget - Request for https://localhost:44390/nuget - GET
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget - No authorization header found
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget - Request allowed by default
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - Request for https://localhost:44390/nuget/ - GET
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - No authorization header found
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - Request denied for /nuget/
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget - Request for https://localhost:44390/nuget - GET
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget - No authorization header found
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget - Request allowed by default
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - Request for https://localhost:44390/nuget/ - GET
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - No authorization header found
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - Request denied for /nuget/
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - Request for https://localhost:44390/nuget/ - GET
3/05/2015 10:27:27 AM +00:00: https://localhost:44390/nuget/ - Password valid - setting the principal
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/ - Auth cookie set
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/ - User authenticated, request is allowed
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Request for https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - GET
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - No authorization header found
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Request denied for /nuget/
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Request for https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - GET
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Password valid - setting the principal
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - Auth cookie set
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/nuget/Packages(Id='OtherSystem.Client',Version='2.0.2') - User authenticated, request is allowed
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - Request for https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - GET
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - No authorization header found
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - Request denied for /api/
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - Request for https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - GET
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - Password valid - setting the principal
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - Auth cookie set
3/05/2015 10:27:28 AM +00:00: https://localhost:44390/api/v2/package/OtherSystem.client/2.0.2 - User authenticated, request is allowed

Why am I not seeing the same behavior when VSO build hits my custom NuGet server?

@yishaigalatzer

How did you get nugetm.exe to vso? Is it exactly the same binary?

@roryprimrose

I got it from the CommandLine NuGet package and put it in my repo. It is version 2.8.60318.667

@roryprimrose

I also tested with nuget 2.8.50926.602 and found the same. With both these versions, I consistently get the problem locally. I don't know what I missed when I tested this locally the other day.

@bhuvak
bhuvak commented May 5, 2015

If you try downloading a package from your custom nuget server ( not locally hosted in IISExpress) via browser ( using /api/v2// ) does it work ? If there is a permissions related issue, it should fail.
Also when you try to hit your nuget.server from nuget.exe from your local machine, can you capture the fiddler trace and share it with us ?

@bhuvak
bhuvak commented May 19, 2015

@roryprimrose, I am closing the issue as we couldn't repro it and it seems like a permission issue.

@bhuvak bhuvak closed this May 19, 2015
@roryprimrose

Sorry, I missed your last comment. I'll try to get a fiddler trace today.

I don't think it is a permissions issue. I have a workaround where I cache the IP address of a previously successful authentication attempt. I then allow that IP to make any call within a few minutes. This workaround (bad as it is for security) works fine both locally and hosted in Azure. Without the IP cache I get the same lack of authentication provision as described above.

@yishaigalatzer

This looks a lot like a dup of #599

CC @feiling

@yishaigalatzer yishaigalatzer assigned feiling and unassigned bhuvak May 19, 2015
@feiling
Contributor
feiling commented May 20, 2015

@roryprimrose I checked our code and didn't find anything wrong. It does send out basic authentication headers.

Please use fiddler to capture all requests and responses. Then search for requests of "GET https://nuget.myserver.com/api/v2/package/othersystem.client/2.0.2".

There should be at least two requests. The response of the first request should look like this:

HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/8.5
WWW-Authenticate: Basic realm="xxx"
X-Powered-By: ASP.NET
Date: Wed, 20 May 2015 20:04:10 GMT
Connection: close
Content-Length: 0

Note the line WWW-Authenticate: Basic realm="xxx". This tells NuGet that the server needs basic authentication. So NuGet will send the same request, with this header

Authorization: Basic xxxx

where xxxx is the password.

Could you verify that the right response and request headers are there? If no, then it's a bug in NuGet and we'll fix this bug. If yes, then it's a setup issue with the server (most likely the username specified in your nuget.config does not have the permission to access the packages folder on the server).

@roryprimrose

I spent a couple of hours working on this yesterday. I can't reliably reproduce this now. I looked at a restore scenario that uses both local and Azure based NuGet server. Looks like it is working on the command line ATM. Interestingly, a restore via the VS IDE fails though.

I still don't think it is a permissions issue because it works sometimes in some circumstances and works reliably with the IP cache. I just can't narrow down where this is going wrong. I'll close this for now unless I can reliably reproduce the issue.

@yishaigalatzer

@roryprimrose thanks for pushing on this, if you get a fiddler trace of what is going on, we will gladly take another look.

When you say restore from Visual Studio fails, can you share the version and a fiddler trace?

@roryprimrose

Sorry, I tried to get a fiddler trace, but nothing was recorded.

@johncmckim

@roryprimrose @feiling I seem to be having the same issue. This issue is that an authenticated download request is never sent to the server.

I am using nuget restore via the command line and my credentials are stored in the Nuget.config in AppData.

nuget restore

Note: The first line states 'Using credentials from config', followed by 'Please provide credentials for'

The initial search requests do the two step process of no Authorization Header and a 401 response followed by the same request with a Authorization header.

fiddler

However, when it comes to downloading the package, the Authorized request is never sent. This an example of the final request that is sent.

GET https://nuget.careerhub.com.au/api/v2/package/visualeyes.apphelper.web.migrator/1.0.45 HTTP/1.0
NuGet-Operation: Restore
User-Agent: NuGet Command Line/2.8.60318.667 (Microsoft Windows NT 6.2.9200.0)
Host: nuget.careerhub.com.au

HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/8.0
WWW-Authenticate: Basic realm="CareerHub.NuGet"
X-Powered-By: ASP.NET
Date: Thu, 16 Jul 2015 07:29:21 GMT
Connection: close
Content-Length: 0

It begs the question why is a request without the Authorization header never sent? In what situation would the user not want an Authorized request to be sent when credentials are set?

@yishaigalatzer

You never send a request with the headers upfront, that's not how basic auth works. This new input is interesting im reactivating and we can see if we can repro it this time around. thanks!

@deepakaravindr deepakaravindr assigned pranavkm and feiling and unassigned feiling and pranavkm Jul 23, 2015
@rossipedia

I'm running into the same issue on VS2015 RTM and v3.1.1 of the Nuget extension. This was not a problem before the v3.1.1 release.

I wonder if this commit has something to do with it.

@yishaigalatzer

@rossipedia this issue is about nuget.exe command line. Can you please open a separate issue for the 3.1.1 extension with exact repro steps?

@rossipedia

Sure. I'm also experiencing this issue with the command line as well.

@stevewilliamsuk

I am experiencing this issue as well. It is because the initial request for the package goes to http://nugetserverhost/nuget/[...](with the auth header using the information in the config file) which then tells the nuget client to visit http://nugetserverhost/api/v2/[...](which we do not have auth credentials for).

This is my example config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageRestore>
    <add key="enabled" value="True" />
    <add key="automatic" value="True" />
  </packageRestore>
  <activePackageSource>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
  </activePackageSource>
  <packageSources>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
    <add key="Obfuscated" value="http://nuget.obfuscated.com/nuget/" />
  </packageSources>
  <disabledPackageSources />
  <packageSourceCredentials>
    <Obfuscated>
      <add key="Username" value="removed" />
      <add key="ClearTextPassword" value="also-removed" />
    </Obfuscated>
  </packageSourceCredentials>
</configuration>

You'll notice even if you drop this into %APPDATA%\Nuget\Nuget.config you'll be able to access the package list fine, but as soon as you try to install one (through the package manager) you'll be prompted for the credentials.

This is because for some reason the private nuget server wants to listen on ~/nuget/ and not ~/api/v2/ for the initial package listing. If it were to listen on the latter address, we would be setting the URL to http://nuget.obfuscated.com/api/v2/ instead of /nuget/ and then we would be able to store credentials.

@stevewilliamsuk

Potential workaround

Further to this above post I have taken a trace with Charles. This shows behavior to support what I've said above...

nuget_charles_trace

You would have thought that a hack like this would work...

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageRestore>
    <add key="enabled" value="True" />
    <add key="automatic" value="True" />
  </packageRestore>
  <activePackageSource>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
  </activePackageSource>
  <packageSources>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
    <add key="Removed" value="http://nuget.removed.com/api/v2/" />
    <add key="Removed2" value="http://nuget.removed.com/nuget/" />
  </packageSources>
  <disabledPackageSources />
  <packageSourceCredentials>
    <Removed>
      <add key="Username" value="comeon" />
      <add key="ClearTextPassword" value="really?" />
    </Removed>
    <Removed>
      <add key="Username" value="comeon" />
      <add key="ClearTextPassword" value="really?" />
    </Removed>
  </packageSourceCredentials>
</configuration>

The command line tool still prompts for the credentials which is useless for CI.

So the only choice we are left with is another workaround/hack to make our private nuget server behave a little more like the public one - listen on /api/v2 for the package queries as well as the actual package downloads. Then the client will only worry about one set of credentials.

So stuff this in your web.config under system.webServer:

        <rewrite>
            <rules>
                <rule name="Rewrite" patternSyntax="ECMAScript">
                    <match url="api/v2(/)?$" />
                    <action type="Rewrite" url="/nuget/" />
                </rule>
                <rule name="Rewrite2" patternSyntax="ECMAScript">
                    <match url="api/v2/Packages\((.*)$" />
                    <action type="Rewrite" url="/nuget/Packages({R:1}" logRewrittenUrl="true" />
                </rule>
            </rules>
        </rewrite>

Now if we change the config file like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageRestore>
    <add key="enabled" value="True" />
    <add key="automatic" value="True" />
  </packageRestore>
  <activePackageSource>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
  </activePackageSource>
  <packageSources>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
    <add key="Removed" value="http://nuget.removed.com/api/v2/" />
  </packageSources>
  <disabledPackageSources />
  <packageSourceCredentials>
    <Removed>
      <add key="Username" value="comeon" />
      <add key="ClearTextPassword" value="really?" />
    </Removed>
  </packageSourceCredentials>
</configuration>

nuget_charles_trace

Then we see a much healthier trace in charles (after a few trial-and-errors).

The fix is simply to have the nuget private server code, going forward, listen on /api/v2 for it's queries. Can this be done in a future release please??

@yishaigalatzer

We could take a pr for that or see if we can improve the configuration mechanism to understand how to pick the username/password correctly.

Long term we plan to drop basic auth as the default authentication mechanism and move to something a long the lines of oauth with expiring tokens.

We of course wont deprecate it from the client, this is a move forward on the server side.

@martinsuchan

Hi, we're using custom NuGet repository secured by basic authentication and we've also started developing UWP Windows 10 apps that use the new project.json NuGet config file. The problem is that **NuGet 3 is unable to successfully restore

packages because the restore process is ignoring provided basic authentication credentials**.

I've created simple repro available here: http://1drv.ms/1hvdfFT
How to use it:
run the 0_InitCustomSource.ps1 file to add reference to test NuGet server located on
http://nugetbug.azurewebsites.net/nuget -UserName "nugetbug" -Password "bugbug"
The server contains two NuGet pacakges, LibCore and LibMain that depends on LibCore, both for Windows 8.1 apps.
The sample also contains UWP project referencing LibMain.

Expected behavior: when I open the project without project.lock.json, click Rebuild - all packages are properly restored, the build works on first try.
Current behavior: The build fails with this error message:

Restoring NuGet packages...
To prevent NuGet from restoring packages during build, open the Visual Studio Options dialog, click on the Package Manager node and uncheck 'Allow NuGet to download missing packages during build.'
Failed to retrieve information from remote source 'http://nugetbug.azurewebsites.net/'.
Response status code does not indicate success: 401 (Unauthorized).
Failed to retrieve information from remote source 'http://nugetbug.azurewebsites.net/'.
Response status code does not indicate success: 401 (Unauthorized).
1>------ Rebuild All started: Project: NuGetTest, Configuration: Debug ARM ------
1>  NuGetTest -> C:\Users\martin.suchan\Desktop\NuGet.bug\NuGetTest\bin\ARM\Debug\NuGetTest.exe

This currently breaks our build environment and it also makes it hard to impossible to do clean checkout and just build our project, this is basically show-stopper right now.
Possible workaround could be disabling the basic authentication on our NuGet server or caching the project.json.lock in source control, none of which is a good solution.

Tested on Windows 10, VS2015 with Win10 SDK, all RTM, and NuGet package manager 3.1.60724.766.

@Nicholi
Nicholi commented Aug 11, 2015

Bless your little dev heart @stevewilliamsuk . Your workaround works for both CLI and the new automatic package restore in VS.

I REALLY WISH nuget devs would seriously consider the fact that people run private repos with basic auth more. This has been a random issue off/on for the last few years with updates... it's getting tiresome.

@yishaigalatzer

@martinsuchan this seems to be unrelated to this issue, I opened another one to follow up for an immediate fix.

@Nicholi a private server still needs to behave like a public server (which is what @stevewilliamsuk did). Unless this is a server that we created and broke (which we should fix at the point), we are definitely not trying to ignore basic auth as a scenario, we do test it automatically on every build.

We will figure this out if it is at all possible to do it securely

@Nicholi
Nicholi commented Aug 12, 2015

@yishaigalatzer I'd agree with everything you say except the nuget clients almost never seem to work correctly when a repo requires basic auth. Especially in terms of package restore. Then you find some hack/fix, and wait for the next nuget release to break it. And find some new hack/fix. I'm not saying the servers are broken (since the basic auth is handled entirely by the webserver, not nuget at all), clearly this is a nuget client issue. Both CLI and VS extension.

@yishaigalatzer

I don't 100% agree. From what I have seen so far, I see servers bumping to different URL for package download (not ok) or not following the basic auth protocol causing basic auth to break. See the comment above about making the server behave like the public server. It's expected for basic auth not follow different hosts since it breaks security.

We will love to fix any issue but that requires reasonably solid repros, which we will turn into functional tests to guarantee the scenario is no longer broken.

I'm sorry that is your past experience, and it is my goal to push and make it better, I invite you to help by providing repros and trying out nightly builds (of course if you want to) and provide early feedback before we actually break a scenario we are not aware of.

@Nicholi
Nicholi commented Aug 14, 2015

Well I'm just using Nuget.Server package with no changes from package install, in IIS with HTTPS and Basic Authentication enabled from IIS side. So are we saying IIS doesn't properly support basic auth? Also when I use the word "private" I only mean in reference to who I give the credentials to, so it's not available to the public. It's private, in that its only usable by me and the people who have the credentials (my server is actually publicly routable on the internet). As opposed to "public" being a repository on the internet with no password, so is technically usable by anyone who can point to it...public. The whole point of requiring basic auth is so that the server IS NOT publicly usable.

I'm not sure what you are saying "It's expected for basic auth not follow different hosts since it breaks security". I'm only expecting the credentials setup for nuget repository "myRepo" with hostpath "https://blahblah.com/someprefix/nuget/" to use the credentials I specify in Nuget.config (in my roaming profile, not at sln/project level). And if I had a second repository "myRepo2" (again with its own hostpath), it would use its own credentials as specified in Nuget.config. Is that wrong?

Nuget.Server has been using the default prefix of /nuget/ for approximately the last 5 years though, so..... Did the clients just make breaking changes that made all (as of now) versions of Nuget.Server somewhat unusable with basic auth? Or is the Nuget.Server just simply no longer fully supported, even though the packages are still available and apparently being updated?

I'm just as confused as you I guess. I really wish I had the time to make wonderful detailed reproductions of the problem, but I don't. And hacking around the problem is much simpler. I'm guessing anyone who has also expected it to "just work" likely has given up in frustration. I think if you do some rudimentary searches on stackoverflow/your own issue tracker (and the older codeplex) for things involving basic authentication you'll find a wonderful rainbow of people reporting the same problems and all their hacks/instructions that sometimes work and sometimes don't. If you find this acceptable, by all means don't look any further into it. I don't really see anything I'm doing as some super customized non-standard scenario (maybe you can inform me otherwise).

Perhaps there could be "working instructions" from someone on the Nuget side as to how we are supposed to correctly use the Nuget.Server package with basic authentication, and would preferably work seamlessly with nuget clients without constantly prompting us for the passwords. I think if someone were to start walking down that path and attempt to show how everything "just works"... they'll actually find it doesn't.

@yishaigalatzer

Thanks for the clarification in what is it that you are doing. The fact that you are using nuget.server publically in IIS with basic credentials was missing. I'm not sure if that was an intended scenario for it or not, going to find out. Keep in mind that it was probably never intended to do that.

For anything advanced we really recommend going with the nuget gallery rather than the nuget.server.

However your comment puts us on the right path to figure out what is it that you are trying to achieve and an ability to test it out.

@yishaigalatzer

@Nicholi One more question. Are you restoring/installing into a project with packages.config or project.json?

@martinsuchan 's follow up bug - #1158

@Nicholi
Nicholi commented Aug 14, 2015

Still using packages.config. I haven't migrated to all the new nuget3/vs2k15 features just yet, but slowly moving towards them. I have migrated to the new automatic package restore method.

I have a feeling also just removing the sources and re-adding them with credentials again may have fixed more things then anything else. As I'm now ok even without @stevewilliamsuk's fix.

@martinsuchan

@yishaigalatzer I'm not sure if that was an intended scenario for it or not, going to find out.
The "nuget.server" package is intended for running as ASP.NET app on IIS. There is no other scenario where it should be used, maybe on Mono or Azure, but that's basically the same.

I've even created simple tutorial how to deploy your NuGet Server on Azure and secure it with Basic Authentication. The repro for my bug mentioned earlier is based on this guide.

@johncmckim

@Nicholi if you lucky enough that re-adding it fixes it for you that's great. However, it definitely has not fixed the issue for our setup (can't tell you the number of times I tried removing and re-adding config). I have a very similar setup, Nuget.Server package and Basic Auth.

@yishaigalatzer do you have enough information based on all the examples and comments to reproduce the problem and look into it? If not what more do we need to provide?

It seems likely that bug is related to the difference between having https://somehost.com/nuget/ in the configuration and which isn't being picked up for requests going to https://somehost.com/api/v2/. Either it's a client issue, i.e. it should get the username / password per host or Nuget.Server should be using /api/v2/ instead of /nuget/.

Are you able to outline some kind of plan to resolve this issue? It's obviously a very frustrating problem for those people experiencing it.

As a follow up to the Nuget Gallery comment, if that's what we are supposed to be using, why is there no mention of it in the nuget documentation http://docs.nuget.org/create/hosting-your-own-nuget-feeds. I assume that many many people wanting to host their own feeds are following that guide and using the Nuget.Server package. I would be happy to use the Gallery project instead if it supports an Authentication system that I can hook into my LDAP server, but there's no mention of it in the docs as an option at all.

@yishaigalatzer

@johncmckim I'm missing one bit of feedback.

The report from @martinsuchan talks about project.json scenarios. It is understood and we are tracking a fix.

I manage to connect to the feed and download the packages through VS2015 UI (though they are not applicable to NET452 projects they download fine).

So I still don't know if there is any other scenario broken here other than the one tracked at - #1158

@martinsuchan - Yes I wasn't aware it was (sorry relatively new to the team, I checked last night and yet it is supposed to work).

@johncmckim - Here is the difference between NuGet Gallery and nuget.server. Both support authentication, the former is more complicated to install, but provides a lot more features.

NuGet.Server
Works off the list of files on disk, becomes very inefficient to startup and find packages when the number of packages grows (above a few hundreds its going to start crawling)

Intended for small projects with a few packages. Mostly allows decoupling from using a fileshare, locking files and access over Http where fileshare access is not available.

NuGet Gallery

More of a mirror of the nuget.org V2 site, with a database behind the scenes and a lot more available features.

Here is one of the blogs I found on how to set it up with authentication. This might not be the most up to date one.
http://blogs.msmvps.com/p3net/2013/01/06/setting-up-a-private-nuget-repository/

You are right that the docs do not point to the nuget gallery solution, or go into the details above. I'll make sure that happens.

@Nicholi
Nicholi commented Aug 14, 2015

No I was wrong, I started randomly getting prompted for passwords when manually installing/upgrading packages through the UI interface from the private repo. From what I've found you need to have multiple credentials entered for different paths, otherwise the client doesn't know what to do.

One for plain /nuget/ (I'm assuming this is for using the repo through the UI), one for /api/v2/ (for package restore to actually work), and I also have one for /api/v2/package/ (used when I push packages with api key). I have the last one disabled.

@johncmckim I'd make sure you have the latest nuget.exe and Nuget.Server package. Also try and assure the nuget.org repos are first in your list, and the protocolVersion=3 one is present. Otherwise yeah...fun and frustrating.

Ultra P.S. Also transition to the new automatic package restore method.

@jmyersmsft

We're seeing some strange auth issues here too. Using nuget.exe 3.1.0-beta downloaded from the link in the blog post, with an authenticated v3 source with credentials stored in nuget.config, running

nuget list -source MyAuthenticatedSource

gives the following behavior:

  1. GET (index.json) (without credentials) -> 401 with auth challenge
  2. GET (index.json) (with Basic credentials) -> 200
  3. GET (SearchQueryService URL) (without credentials) -> 401 with auth challenge
  4. NuGet.exe gives up with the message "Failed to retrieve metadata from source: '[query URL]'.". No further HTTP requests are issued,

index.json and the SearchQueryService are on the same host, if that's interesting

@yishaigalatzer

How did you get an authenticated v3 feed?

@emgarten
Contributor

We believe this is fixed in the latest version of nuget.exe. We have verified the following scenarios for basic auth:

  • nuget.server hosted on IIS with basic auth
  • authenticated v3 feed from myget.org
  • project.json scenarios and packages.config scenarios
  • proxy scenarios

I'm going to close this bug for now, use the link below to verify that this fixes your scenarios:
https://www.myget.org/F/nugetbuild/api/v2/package/NuGet.CommandLine/3.2-beta-10471

If you are still seeing issues with the above nuget.exe please open an issue with exact repro steps and a server that we can hit to test nuget.exe similar to how @martinsuchan set up the repro.

@emgarten emgarten closed this Aug 28, 2015
@RanjiniM RanjiniM added the 2 - Working label Sep 1, 2015
@RanjiniM RanjiniM assigned zhili1208 and unassigned feiling Sep 1, 2015
@zhili1208 zhili1208 added 3 - Done and removed 2 - Working labels Sep 1, 2015
@martinsuchan

This just happened on my machine - Windows 10, Visual Studio 2015 Pro with UWP Tools v1.1, NuGet 3.2.60914. When rebuilding UWP app sample -> about 100 Basic Authentication dialogs popped up even though I got our custom feed with saved basic authentication credentials properly added in feed sources. Note this sample used only default UWP NuGet package Microsoft.NETCore.UniversalWindowsPlatform, no package from our custom feed.
Bottom right shows the NuGet sources settings. This is a serious issue needs to be handled.
Note this issue is probably related to #1158

nugetbug

@yishaigalatzer

Do you get a consistent repro? As in does this happen every time or just once?

We just did an overhaul for auth in 3.3 so it would be interesting to test it.

Last it doesn't matter if packages are in the authenticated source or not because the code cannot know that until they get queried

@martinsuchan

It looks like I got consistent repro (Windows 10, Visual Studio 2015 Pro with UWP Tools v1.1, NuGet 3.2.60914):

  1. download the repo of Windows 10 UWP samples
  2. clean all already downloaded NuGet packages in C:\Users[user].nuget\packages to start with clean slate
  3. make sure the custom NuGet feed with basic authentication has been added, see my post before
  4. open this solution: Windows-universal-samples\Samples\CortanaVoiceCommand\cs\AdventureWorks.sln
  5. start solution Rebuild -> lot of invalid Basic Authentication dialogs appear.
@martinsuchan

I can confirm this is no longer happening with NuGet 3.3.0.

@yishaigalatzer

@martinsuchan thanks for getting back to us! Much appreciated

@TKonov
TKonov commented Jul 15, 2016

@stevewilliamsuk thank you for the insight and the suggestion. Works like a charm!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment