Cross-platform glob pattern and path issues

[dtivel]

NuGet/NuGet.Client#909 fixes a few issues, but there are more.

1. PathResolver.IsWildcardSearch(...) only checks for presence of a star character (*), but PathResolver.WildcardToRegex(...) supports question mark character (?).

2. NuSpec examples show glob patterns with both forward slash (/) and backslash (\), but it is not clear if NuGet intends to support both interchangeably on both Windows and Unix-flavored operating systems.

3. Code to handle glob patterns/paths with forward- or backslashes is scattered around the code base (example), which makes bug fixing and testing more difficult.

4. • Glob results differ when using forward vs back slashes #3959 glob results differing when using forward slash vs backward slashes
5. • NuGet Pack issues with forwash slack/backward slash handling. [Mono Issue] NuGet.exe pack will add duplicate files to nuget package #4073, nuget pack fails if path has double backslash #4729 and Spurious dotnet pack warnings #4596
6. • NuGet push xplat path issues.  #5992 NuGet push xplat path issues, related nuget.exe push .nupkg with multiple slashes in path fails #3544

[nkolev92]

Adding another issue to the list:

#3959

Here's the description from there:

Globbing patterns in nuspec using forward slashes do not match files. The same pattern with backslashes works.

Details

Using NuGet.exe 3.5.0.1938 on Windows 10

Repro

1. Create folder layout

TestPackage.nuspec
folder/
     test.txt

1. Use this nuspec

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2012/06/nuspec.xsd">
  <metadata>
    <id>TestPackage</id>
    <version>1.0.0</version>
    <authors>name</authors>
    <description>description</description>
  </metadata>
  <files>
    <file src="folder/*.txt" target="dir1/" />
    <file src="folder\*.txt" target="dir2/" />
  </files>
</package>

2. nuget.exe pack TestPackage.nuspec

Expected result

NuGet package contains:

<nupkg>/
   TestPackage.nuspec
   dir1/
       test.txt
   dir2/
        test.txt

Actual result

The globbing pattern for folder/*.txt does not match files, but folder\*.txt does.

<nupkg>/
   TestPackage.nuspec
   dir2/
       test.txt

[nkolev92]

Threw in a couple of pack issues with forward/backward slashes.

[nkolev92]

Added all related issues I could find that deal with paths cross-plat both pack and push related ones.
Assigning @rohit21agrawal and @zhili1208 to this mega-issue as they are the ones that had multiple of these assigned to them.

[RehanSaeed]

The glob syntax doesn't seem to be documented anywhere. Would be nice to add this to the TODO list. In particular I'd love to know how to negate a match e.g. *!(preview)*.nupkg (find all .nupkg files without preview in the file name).

[jzabroski]

@RehanSaeed Let me know if you ever discovered the inner secrets of how globbing works.

Running into an issue with dotnet.exe pack with a custom package.nuspec file.

One possible solution seems to be to disable auto globbing altogether

<PropertyGroup>
  <EnableDefaultContentItems>false</EnableDefaultContentItems>
</PropertyGroup>

But that doesn't quite work for my scenario.

[beatcracker]

NuGet v6.0.0 broke forward slash handling.

In v5.11.0 this correctly puts contents of the dir in the root of the package

<file src="repro/dir/**" target="/" />

In v6.0.0 the repro dir and it's entire contents are put into the package.

Here is repro script (PowerShell). It'll download NuGet v5/6 binaries, create nuspec, package directory structure and 2 packages from the same nuspec using different NuGet versions.

$repro_path = "$PSScriptRoot/repro/dir"
$nuspec_path = "$PSScriptRoot/repro.nuspec"

if (
  -not (Test-Path -LiteralPath $repro_path -PathType Container)
) {
  New-Item -Path $repro_path -ItemType Directory > $null
}

'repro' | Out-File -LiteralPath "$repro_path/repro.txt" -Encoding ascii
@'
<?xml version="1.0"?>
<package >
  <metadata>
    <id>Repro</id>
    <description>repro</description>
    <version>0.0.0</version>
    <authors>repro</authors>
  </metadata>
  <files>
    <file src="repro/dir/**" target="/" />
  </files>
</package>
'@ | Out-File -LiteralPath $nuspec_path -Encoding ascii

'5.11.0', '6.0.0' | ForEach-Object {
  $nuget_binary = "$PSScriptRoot/nuget.$_.exe"

  if (
    -not (Test-Path -LiteralPath $nuget_binary -PathType Leaf)
  ) {
    Invoke-WebRequest -UseBasicParsing -Uri (
      "https://dist.nuget.org/win-x86-commandline/v$_/nuget.exe"
    ) -OutFile $nuget_binary
  }

  & $nuget_binary @(
    'pack'
    '-Version'
    $_
    $nuspec_path
  )
}

[jzabroski]

@RehanSaeed Let me know if you ever discovered the inner secrets of how globbing works.

Running into an issue with dotnet.exe pack with a custom package.nuspec file.

One possible solution seems to be to disable auto globbing altogether

<PropertyGroup>
  <EnableDefaultContentItems>false</EnableDefaultContentItems>
</PropertyGroup>

But that doesn't quite work for my scenario.

Just to add on to my previous reply, I figured out what my problem was.

Up until a recent version of NuGet, it was possible for a nuspec file to have duplicate or overlapping mapping rules. NU5050 was added to prevent this. While Zip files allow two files with the same name to exist in the same zip file, the behavior when unzipping those files is undefined. Thus, you could unpack a nuget package and whichever one of those two files could "win". In most cases, these files MIGHT even be identical values IF there upstream source was identical.

While created with good intentions, this Nu5050 "feature" created a situation where, if you wanted to pack a Web site project using a custom nuspec, you needed some really verbose nuspec rules:

  <files>
    <file src="bin\**\*" target="bin" />
    <!-- We need to exclude the bin directory contents from each of the following includes to avoid NU5050 duplicate error. -->
    <file src="**\*.ascx" exclude="bin\**\*" />
    <file src="**\*.asmx" exclude="bin\**\*" />
    <file src="**\*.aspx" exclude="bin\**\*" />
    <file src="**\*.config" exclude="bin\**\*" />
    <file src="**\*.cshtml" exclude="bin\**\*" />
    <file src="**\*.css" exclude="bin\**\*" />
    <file src="**\*.eot" exclude="bin\**\*" />
    <file src="**\*.gif" exclude="bin\**\*" />
    <file src="**\*.jpg" exclude="bin\**\*" />
    <file src="**\*.js" exclude="bin\**\*" />
    <file src="**\*.map" exclude="bin\**\*" />
    <file src="**\*.master" exclude="bin\**\*" />
    <file src="**\*.png" exclude="bin\**\*" />
    <file src="**\*.sass" exclude="bin\**\*" />
    <file src="**\*.scss" exclude="bin\**\*" />
    <file src="**\*.skin" exclude="bin\**\*" />
    <file src="**\*.svg" exclude="bin\**\*" />
    <file src="**\*.ttf" exclude="bin\**\*" />
    <file src="**\*.woff" exclude="bin\**\*" />
    <file src="**\*.woff2" exclude="bin\**\*" />
    <file src="*.asax" exclude="bin\**\*" />
    <file src="*.htm" exclude="bin\**\*" />
    <file src="*.xml" exclude="bin\**\*" />
    <file src="*.sitemap" exclude="bin\**\*" />
  </files>

Unrelated, it also created a situation where you can't pack a .NET SDK csproj if the .NET SDK csproj contains no packable items. In other words, you can't use the special MSBuild csproj SDK types, like NoBuild, to create packaging of arbitrary content defined outside the csproj. Also, as far as I can tell, there is no way to inject fakes for the packable content so that MSBuild just passes that along to the nuspec. Also, if the MSBuild packable content also is covered by a nuspec wild card, the two will collide but the error reported by NU5050 isnt great.

[Stefan137]

File separator handling in globbing expressions should really be documented and fixed. Bugs like the one reported by beatcracker cause issues in other projects, while the root cause is within NuGet. A portable solution (e.g., support slashes on all platforms) would be welcome.

[FaustinCarter]

Just got bit by this today. Took ages to figure out that I needed to switch to backslash for Windows. Would love to have a cross-platform solution!

[jzabroski]

@JonDouglas I dont know if this is something your team is looking into, but the general lack of documentation on this is frustrating. More and more people will get bit by this as people use Nuget 6 (upgrade to VS 2022 for example).

[danielchalmers]

Ran into this problem as well (on Windows 10 21H2).

Forward slash in dotnet pack MyApp/MyApp.csproj -o publish works,
but dotnet nuget push ... publish/*.nupkg returns error: File does not exist (publish/*.nupkg)..

Changing from publish/*.nupkg to publish\*.nupkg makes it work. Not great.

[deruitda]

I also ran into this issue today doing a dotnet pack using a custom .nuspec file which used forward slashes in some paths (I dev on macOS), but my coworkers on windows couldn't pack the files due to them needing to use back slashes in those paths.