Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NullReferenceException during list package in NuGet.CommandLine.XPlat #13397

Open
carstencodes opened this issue Apr 17, 2024 · 10 comments
Open
Labels
Type:Bug WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually.

Comments

@carstencodes
Copy link

NuGet Product Used

dotnet.exe

Product Version

8.0.204

Worked before?

No response

Impact

It's more difficult to complete my work

Repro Steps & Context

This issue has already been created in .NET SDK, but was closed There with the hint that It's more likely to be a NuGet issue.

Describe the bug

When I run dotnet list package for a solution where

a) Non-project MsBuild-Properties are used, e.g. SolutionDir
b) Directories are not defaulted, e.g. BaseIntermediateOutputPath

the command exits with an error, saying that

  • A file/directory cannot be found files, as $(SolutionDir) resolves to an empty string and hence is rooted erroneously
  • The project.assets.json cannot be found, as it is not stored in $(MSBuildProjectDirectory)obj\

In our solution we have > 200 projects. These projects use a common base intermediate output path set to $(SolutionDir)obj\$(MSBuildProjectName)\ as it makes it easier to reset the files.

To Reproduce

An example can be found at this gist

Expected output

$ dotnet list package --outdated

The following sources were used:
   https://api.nuget.org/v3/index.json
   C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

Project `sample` has the following updates to its packages
   [net8.0]:
   Top-level Package      Requested   Resolved   Latest
   > Grpc.Tools           [2.51.0]    2.51.0     2.62.0

Actual output

$ dotnet list package --outdated
 dotnet list package --outdated

The following sources were used:
   https://api.nuget.org/v3/index.json
   C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

No assets file was found for `C:\myprojects\temp\dotnet-sdk-tests\test-projects\sample\sample.csproj`. Please run restore before running this command.

Diagnostic output is not helpful:

$ dotnet list package --outdated -v diag 

The following sources were used:
   https://api.nuget.org/v3/index.json
   C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

No assets file was found for `C:\myprojects\temp\dotnet-sdk-tests\test-projects\sample\sample.csproj`. Please run restore before running this command.

Workarounds

None applicable. From my point of view, it would be applicable to pass MsBuildVarialbles to dotnet list package, e.g.

$ dotnet list package --outdated -p:SolutionDir=${PWD} -p:BaseIntermediateOutputPath='$(SolutionDir)obj\$(MSBuildProjectName)'

Exceptions (if any)

No exception to be found.

Further technical details

$ dotnet --info
.NET SDK:
 Version:           8.0.204
 Commit:            c338c7548c
 Workload version:  8.0.200-manifests.7d36c14f

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.204\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.4
  Architecture: x64
  Commit:       2d7eea2529

.NET SDKs installed:
  6.0.129 [C:\Program Files\dotnet\sdk]
  6.0.203 [C:\Program Files\dotnet\sdk]
  6.0.321 [C:\Program Files\dotnet\sdk]
  6.0.421 [C:\Program Files\dotnet\sdk]
  7.0.408 [C:\Program Files\dotnet\sdk]
  8.0.104 [C:\Program Files\dotnet\sdk]
  8.0.204 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
  • VS Code 1.88.1

Verbose Logs

$ dotnet list package --outdated -v diag 

The following sources were used:
   https://api.nuget.org/v3/index.json
   C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

No assets file was found for `C:\myprojects\temp\dotnet-sdk-tests\test-projects\sample\sample.csproj`. Please run restore before running this command.
@jebriede
Copy link
Contributor

jebriede commented Apr 19, 2024

@carstencodes Thanks for reaching out. I was able to reproduce the problem with the example scenario. The SolutionDir is a macro variable that, according to the documentation, is only defined when building a solution in the IDE. Therefore, it won't be set when running the dotnet list package command.

I understand what you're wanting to accomplish, and we handle a similar scenario in our NuGet codebase. Rather than using SolutionDir to define the path to the solution, we define a variable called RepositoryRootDirectory and reference that in the BaseIntermediateOutputPath.

I applied the same technique to the sample you provided and got this to work by making the BaseDir.props as follows:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
        <RepositoryRootDirectory>$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), 'test-project.sln'))\</RepositoryRootDirectory>
        <BaseIntermediateOutputPath>$(RepositoryRootDirectory)obj\$(MSBuildProjectName)\</BaseIntermediateOutputPath>
    </PropertyGroup>

</Project>

With that in place, the dotnet list package command works as expected on my side. Please try this and let us know if that works for you.

@jebriede jebriede added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed Triage:Untriaged labels Apr 19, 2024
@carstencodes
Copy link
Author

Hi @jebriede

thanks for coming back to me.

I replaced all occurrences of SolutionDir with a matching RepositoryRootDir Variable, that we were using before. In my example, it works.

If I apply it to my real solution, the issue remains. In my binary log, I see, that the variable has the correct value.

If I run dotnet 'C:\Program Files\dotnet\sdk\8.0.204\NuGet.CommandLine.XPlat.dll' package list C:\path\to\my\real.csproj -v diag, the output remains unclear:

Unable to read a package reference from the project `C:\path\to\my\real.csproj`. Please make sure that your project file and project.assets.json file are in sync by running restore.

Is there any switch or anything else that I could try to increase verbosity? Or to find the place that actually bugs our config?

@dotnet-policy-service dotnet-policy-service bot added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Apr 22, 2024
@carstencodes
Copy link
Author

If it is more helpful:

I was able to narrow it down to this, where the exception is not passed to the logger and hence get lost / swallowed.

I am currently not able to build a version of XPlat with this fix locally, as I cannot get a version of MsBuild (which seems to be loaded dynamically) to be loaded.

@jgonz120
Copy link

Looks like you got a new error message there, did you run a rebuild or restore before trying dotnet command?

@dotnet-policy-service dotnet-policy-service bot added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. labels Apr 24, 2024
@carstencodes
Copy link
Author

@jgonz120 Yes, I run the restore first. Then I run the list package command.

@dotnet-policy-service dotnet-policy-service bot added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Apr 25, 2024
@carstencodes
Copy link
Author

I was actually able to get more output from XPLat:

System.NullReferenceException: Object reference not set to an instance of an object.
   at NuGet.CommandLine.XPlat.MSBuildAPIUtility.GetResolvedVersions(Project project, IEnumerable`1 userInputFrameworks, LockFile assetsFile, Boolean transitive)

I think, the faulting lines might be these

                                    ProjectItemElement packageInCPM = directoryBuildPropsRootElement.Items.Where(i => (i.ItemType == PACKAGE_VERSION_TYPE_TAG || i.ItemType.Equals("GlobalPackageReference")) && i.Include.Equals(topLevelPackage.Name, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

                                    installedPackage = new InstalledPackageReference(topLevelPackage.Name)
                                    {
                                        OriginalRequestedVersion = packageInCPM.Metadata.FirstOrDefault(i => i.Name.Equals("Version", StringComparison.OrdinalIgnoreCase)).Value,
                                    };

From my side it should be:

                                    ProjectItemElement packageInCPM = directoryBuildPropsRootElement.Items.Where(i => (i.ItemType == PACKAGE_VERSION_TYPE_TAG || i.ItemType.Equals("GlobalPackageReference")) && i.Include.Equals(topLevelPackage.Name, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

                                   if (packageInCPM != null) {
                                    installedPackage = new InstalledPackageReference(topLevelPackage.Name)
                                    {
                                        OriginalRequestedVersion = packageInCPM.Metadata.FirstOrDefault(i => i.Name.Equals("Version", StringComparison.OrdinalIgnoreCase)).Value,
                                    };
                                   }
                                  else {
                                        installedPackage = projectPackage;
                                  }

@carstencodes carstencodes changed the title dotnet list packages should consider msbuild variables NullReferenceException during list package in NuGet.CommandLine.XPlat Apr 25, 2024
@carstencodes
Copy link
Author

carstencodes commented Apr 25, 2024

Or - TBH

ProjectItemElement packageInCPM = directoryBuildPropsRootElement.Items.Where(i => (i.ItemType == PACKAGE_VERSION_TYPE_TAG || i.ItemType.Equals("GlobalPackageReference")) && i.Include.Equals(topLevelPackage.Name, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

                                    installedPackage = new InstalledPackageReference(topLevelPackage.Name)
                                    {
                                        OriginalRequestedVersion = packageInCPM.Metadata?.FirstOrDefault(i => i.Name.Equals("Version", StringComparison.OrdinalIgnoreCase))?.Value,
                                    };

(I don´t know what will happen after this excpetion has been avoided, but I think that using the projectPackage instead of an ´OriginalRequestedVersion´ with a ´null´ value is preferable)

/cc @martinrrm

@jgonz120
Copy link

Is there a way you can modify the example project so we can reproduce this error?

@dotnet-policy-service dotnet-policy-service bot added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. labels Apr 26, 2024
@carstencodes
Copy link
Author

I am already trying, but it's tricky.

Would you accept a Stripped Version of the project.assets.json alongside with the props and Targets File? With Stripped I mean, that the parts of the File would be redacted that Show Login named etc.

In this case I would ASK my Management to publish These Files in a new gist.

@dotnet-policy-service dotnet-policy-service bot added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Apr 27, 2024
@carstencodes
Copy link
Author

Ok, by comparing some folders after deleting Directory.Build.props and .targets I was able to create a version, where this issue does not occur.

The reason is pretty simple and yet silly:

Some of our projects still require to run in .NET Fx based applications. Hence they become strong name signed. The strong name is brought to us as a nuget-package, but without a props file, hence the path to the key is constructed within the build folder.

In this case, the one who wrote the build recipe simply added the PackageReference item within that props file, hence it was never part of Directory.Packages.Props.

Moving this to GlobalPackageReferences now worked, but it was hell of work to make it work, so I ask you to

  • Increase Verbosity, so that all packages not found are listed, when verbosity is set to detailed (or to issue a warning, when this is not found), in order to help people find the issue better.

Here's how this might work:

                bool packagesInSync = true;     // added by carstencodes
                foreach (var library in target.Libraries)
                {
                    var matchingPackages = frameworkDependencies.Where(d =>
                        d.Name.Equals(library.Name, StringComparison.OrdinalIgnoreCase)).ToList();

                    var resolvedVersion = library.Version.ToString();

                    //In case we found a matching package in requestedVersions, the package will be
                    //top level.
                    if (matchingPackages.Any())
                    {
                        var topLevelPackage = matchingPackages.Single();
                        InstalledPackageReference installedPackage = default;

                        //If the package is not auto-referenced, get the version from the project file. Otherwise fall back on the assets file
                        if (!topLevelPackage.AutoReferenced)
                        {
                            try
                            { // In case proj and assets file are not in sync and some refs were deleted
                                var projectPackage = projectPackages.Where(p => p.Name.Equals(topLevelPackage.Name, StringComparison.Ordinal)).First();

                                // If the project is using CPM and it's not using VersionOverride, get the version from Directory.Package.props file
                                if (assetsFile.PackageSpec.RestoreMetadata.CentralPackageVersionsEnabled && !projectPackage.IsVersionOverride)
                                {
                                    ProjectRootElement directoryBuildPropsRootElement = GetDirectoryBuildPropsRootElement(project);
                                    ProjectItemElement packageInCPM = directoryBuildPropsRootElement.Items.Where(i => (i.ItemType == PACKAGE_VERSION_TYPE_TAG || i.ItemType.Equals("GlobalPackageReference")) && i.Include.Equals(topLevelPackage.Name, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

                                    installedPackage = new InstalledPackageReference(topLevelPackage.Name)
                                    {
                                        OriginalRequestedVersion = packageInCPM.Metadata.FirstOrDefault(i => i.Name.Equals("Version", StringComparison.OrdinalIgnoreCase)).Value,
                                    };
                                }
                                else
                                {
                                    installedPackage = projectPackage;
                                }
                            }
                            catch (Exception)
                            {
                                /* changed by carstencodes */
                                packagesInSync = false;
                                logger.Log(new LogMessage(LogLevel.Warning, string.Format(CultureInfo.CurrentCulture, Strings.ListPkg_PackageNotFound, topLevelPackage.Name)));
                                continue;
                               /* end change */
                            }
                        }
                        else
                        {
                            var projectFileVersion = topLevelPackage.LibraryRange.VersionRange.ToString();
                            installedPackage = new InstalledPackageReference(library.Name)
                            {
                                OriginalRequestedVersion = projectFileVersion
                            };
                        }

                        installedPackage.ResolvedPackageMetadata = PackageSearchMetadataBuilder
                            .FromIdentity(new PackageIdentity(library.Name, library.Version))
                            .Build();

                        installedPackage.AutoReference = topLevelPackage.AutoReferenced;

                        if (library.Type != "project")
                        {
                            topLevelPackages.Add(installedPackage);
                        }
                    }
                    // If no matching packages were found, then the package is transitive,
                    // and include-transitive must be used to add the package
                    else if (transitive) // be sure to exclude "project" references here as these are irrelevant
                    {
                        var installedPackage = new InstalledPackageReference(library.Name)
                        {
                            ResolvedPackageMetadata = PackageSearchMetadataBuilder
                                .FromIdentity(new PackageIdentity(library.Name, library.Version))
                                .Build()
                        };

                        if (library.Type != "project")
                        {
                            transitivePackages.Add(installedPackage);
                        }
                    }
                }

                var frameworkPackages = new FrameworkPackages(
                    target.TargetFramework.GetShortFolderName(),
                    topLevelPackages,
                    transitivePackages);

                resultPackages.Add(frameworkPackages);
            }

            // added by carstencodes
            if (!packagesInSync) {
                 throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, Strings.ListPkg_ErrorReadingReferenceFromProject, projectPath));
            }

            return resultPackages;

This would be very helpful.

I you like to, I would submit a PullRequest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type:Bug WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually.
Projects
None yet
Development

No branches or pull requests

3 participants