Windows (NTFS): Inconsistent support for the long-path prefix (\\?\) in filesystem paths

[mklement0]

Note: This is a generalization of #4439. @SteveL-MSFT, if you agree, please close the latter.

Prefixing full, normalized, native filesystem paths with \\?\ allows targeting filesystem items whose path is longer than the legacy limit of 259 characters.

Update:

• Prefix \\?\ isn't needed in PS Core (in .NET Core altogether), so long paths just work as-is, even if not enabled system-wide.
• However, old code may still use it, and even new code may have to, when creating cross-edition scripts.
• Additionally, and separately, \\?\ is useful for targeting files or directories with irregular names, such as ones with trailing spaces, so you can remove them with Remove-Item, for instance.

As an aside: In Windows 10 you can now opt in system-wide to support long paths, but individual applications must also opt in (PowerShell indirectly does) - see https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-4-6-2-and-long-paths-on-windows-10/

As of PowerShell Core 7.3.0-preview.2, the inconsistent support for \\?\ is as follows:

• With the -Path parameter, \\?\ isn't supported at all , so wildcard expressions cannot be used.

  • Get-Item and Get-ChildItem output nothing, and Remove-Item is a quiet no-op (silent failure) too.

• With the -LiteralPath parameter in combination with \\?\:

  • Only file paths can be targeted.
  • Directory paths exhibit the following behavior:
    • Get-Item and Remove-Item complain about not finding the path.
      • Adding -Force makes Get-Item output a broken DirectoryInfo instance.
      • Using Remove-Item without -Recurse on a nonempty directory presents the usual confirmation prompt (implying the ability to recognize the path as existent), but then fails on confirmation.
    • Get-ChildItem reports the root directory's content instead(!)

Additionally, \\?\ paths do not work in the following cases:

• With > / >>, which implicitly behave like Out-File -Path - see Some scripts / files with wildcard metacharacter [ in the file name or path cannot be invoked or redirected to #4726

• In the same vein, invocation of an executable - whether with or without & - is broken (possibly related to Some scripts / files with wildcard metacharacter [ in the file name or path cannot be invoked or redirected to #4726 as well). (no longer a problem as of PowerShell Core 7.3.0-preview.2)

• Start-Process (no longer a problem as of PowerShell Core 7.3.0-preview.2)

• Set-Location (not even with -LiteralPath)

Note:

• Some of these problems are regressions from Windows PowerShell, where only the invocation / Start-Process tests fail and > only with a new file.

• I haven't looked into whether invoking an executable with an overly long path is supported in principle by the underlying APIs.

Steps to reproduce

Run the following from a Pester test script (*.Tests.ps1) on Windows:

Describe "Support for long paths, via \\?\" {
  BeforeAll {
    Push-Location (Convert-Path TestDrive:\)
    $dirPath = $PWD.ProviderPath
    $PrefixedDir = "\\?\$dirPath"
    $fileName = ('a' * 248 + '.cmd')
    $fileNameAlt = ('b' * 248)
    $PrefixedFullName = "$PrefixedDir\$fileName"
    $PrefixedFullNameAlt = "$PrefixedDir\$fileNameAlt"
    # Create the file with the overly long path using .NET,
    # to avoid issues with New-Item
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "Get-ChildItem -LiteralPath" {
    Get-ChildItem -LiteralPath $PrefixedFullName | % FullName | Should -Be $PrefixedFullName
  }
  It "Get-ChildItem -Path" {
    Get-ChildItem -Path $PrefixedFullName | % FullName | Should -Be $PrefixedFullName
  }
  It "Remove-Item -LiteralPath" {
    { Remove-Item -LiteralPath $PrefixedFullName } | Should -Not -Throw
    # Recreate the file.
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "Remove-Item -Path" {
    { Remove-Item -Path $PrefixedFullName } | Should -Not -Throw
    # Recreate the file, if necessary
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "> with new file" {
    { '' >  $PrefixedFullNameAlt } | Should -Not -Throw
  }
  It "> / >> with existing file." {
    { '@echo Hi.' >  $PrefixedFullName } | Should -Not -Throw
    { 'REM ' >> $PrefixedFullName } | Should -Not -Throw
  }
  It "Invocation / &" {
    & $PrefixedFullName | Should -Be 'Hi.'
  }
  It "Start-Process" {
    Start-Process -FilePath $PrefixedFullName
  }
  It "New-Item" {
    { New-Item -Force -Type File $PrefixedFullName } | Should -Not -Throw
  }
  It "Set-Location -LiteralPath" {
    { Set-Location -EA Stop -LiteralPath $PrefixedDir } | Should -Not -Throw
  }
  It "Set-Location -Path" {
    { Set-Location -EA Stop -Path $PrefixedDir } | Should -Not -Throw
  }
  AfterAll {
    # Use .NET to remove the overly long path, so that Pester itself doesn't 
    # fail on trying to remove the dir. underlying TestDrive:
    [IO.File]::Delete($PrefixedFullName)
    [IO.File]::Delete($PrefixedFullNameAlt)
    Pop-Location
  }
}

Expected behavior

All tests should pass.

Actual behavior

All tests but the first one fail, with various error messages.

Environment data

PowerShell Core 7.0.0-preview.4

[iSazonov]

LocalGlobbing code is very sensitive and it seems it is not covered by tests explicitly - I think it is main stopper of a progress here.

[mklement0]

As of PowerShell Core 7.0.0-rc.2 Get-Item -LiteralPath \\?\C:\Windows is now broken too (complains about not finding the path; Get-ChildItem -LiteralPath \\?\C:\Windows still works).

I've noticed that the \\?\ prefix is never needed in PS Core. Still, people may use it in cross-edition scripts.

[iSazonov]

I've noticed that the \?\ prefix is never needed in PS Core.

Long paths is Windows feature, not PS Core. It seems we have a test for long path support.

[mklement0]

Yes, it only applies on Windows, as hopefully clearly implied by the issue title. PS Core should support the long-path prefix there too, and support has degraded from Windows PowerShell, and even further since v7.0.0-preview.4

What I was trying to say: In PowerShell Core on Windows, you can seemingly access paths longer than 259 chars. even without the \\?\ prefix, irrespective of whether long-path support is turned on system-wide.

[mklement0]

It seems we have a test for long path support.

You can run the tests from the initial post; for me, 7 out of the 11 tests fail as of v7.0.0-rc.2

[iSazonov]

What I was trying to say: In PowerShell Core on Windows, you can seemingly access paths longer than 259 chars. even without the \?\ prefix, irrespective of whether long-path support is turned on system-wide.

I don't understand how PowerShell Core can work with long paths if the feature is not enabled on Windows 10.

[mklement0]

That is a fair question, @iSazonov - I haven't looked into the why and how, but you can try it yourself:

First, open an elevated session and run the following, to turn system-wide support for long paths off:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 0

Then, open a regular session and run the following:

Set-Location; 'hi' > ('x'*(259 - "$pwd".length)); (Get-Item x*).FullName.Length; Remove-Item x*

You'll get no error message and 260 as the output, which is the longer-than-259-chars. path of the temporarily created file.

Running the same code in Windows PowerShell fails.

Perhaps \\?\ is being applied behind the scenes?

[iSazonov]

@mklement0 Please share your Windows 10 version. I guess latest versions can have the feature enabled by default.

[mklement0]

It's Microsoft Windows 10 Pro (64-bit; Version 1909, OS Build: 18363.592), but I don't think that is the issue, because the very same code also works on Windows 7 (where system-wide support for long paths didn't even exist).

[iSazonov]

@mklement0 If you replace 259 in your test with more large value (what about 400?) you get an error.

[mklement0]

That is an unrelated constraint: even in long paths the individual components must not be longer than 255 chars., as far as I know. You'd have to design the test differently to get longer paths.

[iSazonov]

Ah :-) my bad.
Get-Item really doesn't process long paths - it split path and works with leaf (the individual component < 256) but returns FileInfo object with FullName which is > 255 (266 chars). Perhaps just the fact confuses us.
Then I tried to Set-Location and create subdirectory but get error on Set-Location. So long paths do not works.

[mklement0]

I don't see the behavior you're seeing; try the following code, which creates a directory path that is itself longer than 259 chars., then a file in it, targets the directory with both Get-Item and Get-ChildItem and the file with Get-Item, changes to the directory with the overly long path with Set-Location, removes the file with the overly long path, then cleans up - it all seems to work fine, both on Windows 7 and Windows 10.

Push-Location ($dir = mkdir -force ($env:TEMP + "\$pid\" + 'x' * 250)); $file = "$pwd\test.txt"; 'hi' > $file; "dir path length: $($dir.FullName.Length)`nfile path length: $($file.length)"; Get-Item $dir; '###'; Get-ChildItem $dir; '###'; Get-Item $file; '###'; Set-Location $dir -passthru; Pop-Location; Remove-Item $file; Remove-Item (Split-Path $dir) -Recurse

Output on my Windows 10 machine:

dir path length: 288
file path length: 297


    Directory: C:\Users\jdoe\AppData\Local\Temp\5816

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           1/24/2020  4:59 PM                xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                                  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
###

    Directory: C:\Users\jdoe\AppData\Local\Temp\5816\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           1/24/2020  4:59 PM              4 test.txt
###
-a---           1/24/2020  4:59 PM              4 test.txt
###

Drive        : C
Provider     : Microsoft.PowerShell.Core\FileSystem
ProviderPath : C:\Users\jdoe\AppData\Local\Temp\5816\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Path         : C:\Users\jdoe\AppData\Local\Temp\5816\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[iSazonov]

@mklement0 I found why I don't understand you :-)
https://github.com/PowerShell/PowerShell/pull/3960/files#diff-52b1a915619c71b288b3f92f944924c4R1156-R1162

GitHub

Allow use of long paths by jeffbi · Pull Request #3960 · PowerShell/PowerShell

Fixes #3891 When calling Windows native API to determine if an item exists, ensure the path is prepended with \\?\ to allow for paths > 260 characters.

[mklement0]

😁 - thanks for digging deeper.

[SteveL-MSFT]

.NET Core itself supports long file paths implicitly, while .NET Framework requires app opt-in. I get that users may want to use the \\?\ legacy syntax, but it's not clear to me how impactful that is to motivate spending the time to fix this.

[mklement0]

Good to know that .NET Core supports long paths by default - seemingly even without the system-wide opt-in.

The code that @iSazonov linked to seems to no longer be part of the repo (except for the tests, which don't use the \\?\ prefix).

I get that users may want to use the \?\ legacy syntax

• They may have done so in the past, which means that old code with \\?\ prefixes that would otherwise be compatible breaks.

• They may have to going forward, if they want to write cross-edition code.

Given that the current behavior is clearly a regression from Windows PowerShell, I suggest either:

• fixing the behavior (which you seem disinclined to do)

• documenting the breaking change.

[iSazonov]

I am not sure we can fix this now. I think it must necessarily be in FileSystem provider V2.

[mklement0]

Understood, @iSazonov.

With the intent to fix this eventually, the categorization shifts from breaking change to (known) bug, so please label this issue accordingly.

[iSazonov]

Meta bug label :-)

[pjaclark]

This should be listed as a Known Issue causing breaking changes until resolved.

[musm]

Is this related to why I can't create very long paths?

> mkdir ('x' * 290)
New-Item: The filename, directory name, or volume label syntax is incorrect ....

[mklement0]

@musm, no, you're seeing a fundamental NTFS limitation that applies even when long-path support is enabled: individual file/directory names must not be longer than 255 chars (as opposed to the length of a path as a whole).

While you could argue that the error message technically applies, it is certainly vague.

[Liturgist]

What is the state of this issue? I see that no one is assigned. Will this be a will-not-fix?

[iSazonov]

What is the state of this issue?

Nice to have :-)

[beermattuk]

What is the state of this issue?

Nice to have :-)

How can this just be a nice to have? For file server administration it's very common for users to save things in ridiculously long paths, then when you come to doing maintenance and migrations you're not able to use powershell to work with them. I know there are workarounds for an individual file, but when you've got a server with millions of files on them and thousands of long paths, how are you supposed to automate anything?

[iSazonov]

but when you've got a server with millions of files on them and thousands of long paths, how are you supposed to automate anything?

It is my headache :-( If supporting \\?\ is not so critical now since Windows 10 API now can support long paths OOB, PowerShell can not be used for processing large file volumes because of very bad performance and now it's even worse than before. See #14469 - the MSFT team does not acknowledge this and I have no explanation for it - only bewilderment.

[ghost]

TL;DR :
Regression has happened because Windows PowerShell embedded the manifest file whereas PowerShell does not. Please embed the manifest in pwsh.exe

Just because .NET Core / .NET 5 / .NET 6 / .NET 7 those runtimes themselves support longPathAware , doesn't mean all .NET applications which are built on top of those runtimes will support it automagically.

Be it .NET or Completely Native, each and every .exe executables has to separately embed the manifest file.

---

---

Long Path works on windows 10 v1607 and upwards with *per executables opt-in.*
Benefit : No need for \\? or UNC Path Handling Complexities Nonsense.

---

build the pwsh.exe with following embedded manifest file.

Special Note : This manifest file has to be embedded into each executables.
That means, If your App contains SomeApp1.exe and SomeApp2.exe , the manifest file has to be embedded into each executables.

File name must be app.manifest

<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>
</assembly>

---

Users who wish to use long path must Set Long Path Aware On system wide through Registry : Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem there make a DWORD key LongPathsEnabled and set the value to 1

Why System Wide Long Path is not on by default (?) according to a MS engineer : microsoft/WindowsAppSDK#875 (comment)

---

Following APIs change behaviors after opt-in :

CopyFileW, CopyFile2, CopyFileExW, CreateFileW, CreateFile2, CreateDirectoryW, CreateDirectoryExW GetCurrentDirectoryW, RemoveDirectoryW, SetCurrentDirectoryW, CreateHardLinkW, CreateSymbolicLinkW, DeleteFileW, FindFirstFileW, FindFirstFileExW, FindNextFileW, GetFileAttributesW, GetFileAttributesExW, SetFileAttributesW, GetFullPathNameW, GetLongPathNameW, MoveFileW, MoveFileExW, MoveFileWithProgressW, ReplaceFileW, SearchPathW, FindFirstFileNameW, FindNextFileNameW, FindFirstStreamW, FindNextStreamW, GetCompressedFileSizeW, GetFinalPathNameByHandleW.

---

Ref:

1. https://docs.microsoft.com/en-us/archive/blogs/jeremykuhne/net-4-6-2-and-long-paths-on-windows-10#enabling-win32-long-path-support

2. https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd

.NET 4.6.2 and long paths on Windows 10

Maximum Path Length Limitation - Win32 apps

Maximum path length limitation.

[iSazonov]

@ghost The issue is about consistent support of \\?\, not long paths in whole. As for manifest, it force to turn on the feature per application. Now users can turn on the feature on OS level. I am not sure we want to force to turn on the feature since we don't cover code by tests and community doesn't vote for this actively. You could ask about adding the manifest in new issue.

[mkht]

I have a simple script that deletes all files in a particular folder. I have added the \\?\ prefix to the path so that it will work correctly even if the full path of the file is longer than 260 characters. This script worked fine in Windows PowerShell 5.1.

$TargetFolder = '\\?\C:\MyTemp\'
Get-ChildItem -LiteralPath $TargetFolder -Force | Remove-Item -Force -Recurse

As a result of running this script in PowerShell 7.2.1, PowerShell attempted to delete all files in the C drive and the system was destroyed.

Despite the fact that PowerShell 7 is advertised as being highly compatible with Windows PoweShell 5.1, this kind of behavior is ridiculous.
I don't know how hard it is, but hopefully it will be fixed to work consistently with Windows PowerShell 5.1.

[iSazonov]

@mkht Really PowerShell FileSystem Provider was never designed to support \\?\ prefix. I hope we will support this but not as particular case, but as generic PowerShell Direct path.

[JazBInKC]

I just submitted a bug report for Set-ACL. I'm using paths like:
in the creation of new folders and the subsequent change of owner using Set-ACL.
I'm having to wait 3 minutes for Set-ACL to timeout with each new folder. No error message.
Thinking the lack of support for \\?\ might be the issue.

[mkht]

PR #15873 that has been merged into v7.3.0-preview.7 seems to have partially resolved this issue.

In the case of the Get-ChidItem command, we can use the long-path prefix with the -LiteralPath parameter, but cannot be used with the -Path parameter.

[maysara]

I have worked around this issue by running test-path -literalPath "\?\C:\test" -ErrorAction:Stop and handling the exception. but now with a .Net framework update (4.8) this behaviour has changed again!

I'm unable to find a reliable way to detect if it is supported (the error shows up with out-file -literalpath even if it stopped with test-path)

[God-damnit-all]

I have a simple script that deletes all files in a particular folder. I have added the \\?\ prefix to the path so that it will work correctly even if the full path of the file is longer than 260 characters. This script worked fine in Windows PowerShell 5.1.

$TargetFolder = '\\?\C:\MyTemp\'
Get-ChildItem -LiteralPath $TargetFolder -Force | Remove-Item -Force -Recurse

As a result of running this script in PowerShell 7.2.1, PowerShell attempted to delete all files in the C drive and the system was destroyed.

Despite the fact that PowerShell 7 is advertised as being highly compatible with Windows PoweShell 5.1, this kind of behavior is ridiculous. I don't know how hard it is, but hopefully it will be fixed to work consistently with Windows PowerShell 5.1.

For some god-forsaken reason, piping IO.FileInfo and IO.DirectoryInfo objects (i.e. what Get-ChildItem outputs) to functions like Remove-Item don't have their paths interpreted literally. That's why I never pipe directly to Remove-Item and instead iterate collections to Remove-Item -LiteralPath – it's much safer.

I can understand interpreting strings in the classic way to prevent breakage for any script that pipes a collection of wildcard strings to such functions (though I can't imagine why someone would), but it's extremely strange and unintuitive for IO.FileInfo and IO.DirectoryInfo input objects to be handled that way.

[alanwilter]

So, is it going to be solved or not? PS7 does need to work with legacy "\?\UNC".
I'm still having problems with PS 7.4.1.

[Akarinnnnn]

Copy-Item doesn't support long path, too. These command works well in temp folder, but the anime folder with super-long path not working.

As it executed, the command fails silently.

This is a meaningful long name example: 线上游戏的老婆不可能是女生 [VCB-Studio] Netoge no Yome wa Onna no Ko Janai to Omotta [Ma10p_1080p]